Department of Communication and Electronic EngineeringUniversity of Plymouth, U.K.

Lingfen SunEmmanuel Ifeachor

New Methods for Voice Quality Evaluation for IP Networks

4 September, 2003 ITC-18, Berlin, Germany 2

Outline

Introduction Subjective MOS test from traditional to

Internet based Intrusive voice quality measurement from

listening-only to conversational Non-intrusive measurement from subjective

based to objective based Conclusions

4 September, 2003 ITC-18, Berlin, Germany 3

Introduction

Aims: to investigate new subjective and objective measurement methods for VoIP applications

Subjective tests MOS (Mean Opinion Score), user perceived quality Benchmarking objective methods Slow, time consuming and expensive

Objective measurements Intrusive methods (e.g. PESQ, only listening quality) Non-intrusive methods (e.g. E-model, only for limited

codec/applications)

4 September, 2003 ITC-18, Berlin, Germany 4

Traditional MOS test

Strict test requirement (e.g. sound proof room)

Guarantee consistent testing environment Slow, time-consuming and expensive Test environment far to reality Originated from codec quality assessment

(the quality difference is subtle).

4 September, 2003 ITC-18, Berlin, Germany 5

Internet-based MOS test

Based on Internet, carried out at office, project room or classroom, …, close to reality

Easier access to large number of subjects (e.g. 39 tests at the same time)

Save time and money Lack of a controlled testing environment

(e.g. background noise).

4 September, 2003 ITC-18, Berlin, Germany 6

Online MOS test

4 September, 2003 ITC-18, Berlin, Germany 7

Online MOS test

4 September, 2003 ITC-18, Berlin, Germany 8

Online MOS test results

4 September, 2003 ITC-18, Berlin, Germany 9

Voice Quality Evaluation

Referencespeech

encoder Trace dataJitterbuffer

Degraded speech

PESQ MOS

(PESQ)

E-model MOS (E-model)

decoder

Subjective test MOS

(Web or Room)

4 September, 2003 ITC-18, Berlin, Germany 10

Performance Comparison

MOS Comparison

0

1

2

3

4

5

0 5 10 15 20 25 30

Packet Loss (%)

MO

S

Room_MOS PESQ_MOS Web_MOS E-model_MOS

4 September, 2003 ITC-18, Berlin, Germany 11

Performance Comparison

Results show that Internet-based MOS test compares well with traditional MOS test.

Two objective test methods (PESQ and E-model) can both predict subjective MOS score well.

Correlation coefficients for MOS comparison

Name PESQ vs.

RMOS

PESQ vs.

WMOS

Emodel vs.

RMOS

Emodel vs.

WMOS

WMOS vs.

RMOS

Emodel vs.

PESQ

Corr. Coeff.

0.933 0.984 0.935 0.964 0.952 0.975

4 September, 2003 ITC-18, Berlin, Germany 12

Intrusive conversational quality measurement

PESQ is the latest ITU standard for intrusive voice quality measurement.

It can only predict one-way listening voice quality.

How to extend it to predict conversational quality? One solution: to combine with E-model

4 September, 2003 ITC-18, Berlin, Germany 13

Procedures: Obtain MOS from PESQ based on a comparison of reference

and degraded speech Convert it back to R-factor, then to Ie Obtain Id from end-to-end delay Calculate MOSc from Id and Ie from E-model

Intrusive conversationalmeasurement

PESQ

Delay model

MOS R Ie

Ie

End-to-end delay

E-modelMOSc

Id

Reference speech

Degraded speech

MOS (PESQ)

4 September, 2003 ITC-18, Berlin, Germany 14

R-factor with MOS

0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

20

40

60

80

100

MOS

R-v

alu

e

R-value vs. MOS

from G.1073rd order polynomial fitting

100 5.4

1000 107)100)(60(035.01

0 16

RforMOS

RforRRRRMOS

RforMOS(1)

336.57060.872314.253026.3 MOSMOSMOSR (2)

( From R to MOS, from G.107)

( From MOS to R, by 3rd order polynomial)

4 September, 2003 ITC-18, Berlin, Germany 15

Combined with E-model

RIe 2.93

0 if 1)(

0 if 0)(

)3.177()3.177(11.0024.0

xxH

xxHwhere

THTTI aaad

(3)

(4)

ed IIR 2.93 (5)

Convert R back to MOS to obtain conversational voice quality

4 September, 2003 ITC-18, Berlin, Germany 16

Non-intrusiveobjective measurement

E-model is the latest ITU standard for non-intrusive voice quality prediction.

Its parameters/equations (e.g. Ie) are still based on subjective tests.

Difficult to apply to new codecs/applications How to improve it ?

Based on objective methods (e.g. PESQ) to derive parameters/equations for new codecs and applications.

4 September, 2003 ITC-18, Berlin, Germany 17

Procedure 1:obtain MOS (PESQ)

Obtain MOS (PESQ) vs. packet loss rate (random) for the codec (e.g. AMR 12.2 Kb/s)

MOS (PESQ) value was obtained by averaging over different speech samples (male and female) in ITU data base and different loss locations/patterns.

0 5 10 15 20 25 301.5

2

2.5

3

3.5

4

4.5MOS vs. Packet loss rate for AMR (12.2Kb/s)

Packet Loss (random loss, %)

MO

S(P

ES

Q)

4 September, 2003 ITC-18, Berlin, Germany 18

Procedure 2:Obtain Ie equation

Convert MOS vs. packet loss to Ie vs. packet loss using Equations (2) and (3).

Curve fitting to obtain equation of Ie vs. loss.

0 5 10 15 20 25 3010

15

20

25

30

35

40

45

50

55Ie vs. Packet loss rate for AMR (12.2Kb/s)

Packet Loss (random loss, %)

Ie

from PESQfrom fitting

)*38.01ln(84.152.13 lossIe (6)

4 September, 2003 ITC-18, Berlin, Germany 19

Procedure 3: Obtain MOS (loss, delay)

From Ie and Id, R can be derived, then MOS can be obtained directly from packet loss and end-to-end delay.

This can be used for non-intrusive quality monitoring and perceptual buffer optimization and perceived QoS control.

0

200400

0 5 10 15 20 25 30

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Delay (ms)

MOS vs. packet loss and delay (for AMR,12.2Kb/s)

Packet loss (%)

MO

S

4 September, 2003 ITC-18, Berlin, Germany 20

Conclusions/Future work

Investigated novel subjective and objective voice quality evaluation methods.

Future work More extensive Internet-based MOS test

and comparison with P.800 MOS test New applications for predicted perceived

voice quality.

4 September, 2003 ITC-18, Berlin, Germany 21

Contact

MOS Test Website:http://www.tech.plymouth.ac.uk/spmc/people/lfsun/mos/ Contact

Lingfen Sun: L.Sun@plymouth.ac.uk Emmanuel Ifeachor: E.Ifeachor@plymouth.ac.uk

Thank you!