quality comparison of wideband coders including … · quality comparison of wideband coders...

research & development

Quality comparison of wideband coders including tandeming and transcoding

Catherine Quinquis ([email protected])France Telecom Research & Development, France

ETSI Workshop on Speech and Noise In Wideband Communication,

22nd and 23rd May 2007 - Sophia Antipolis, France

Quality comparison of wideband coders including tandeming and transcoding – p 2 research & development France Telecom Group

Wideband codecs

Last year, wideband extensions of narrowband codecs have been standardised

G.729.1• providing high packetized wideband voice quality with scalability and

interoperability with existing G.729 based VoIP networks and terminalsEVRC-WB

• providing wideband in 3GPP2 networks using the same rate set as the current EVRC.

Other ITU-T wideband codecsG.722

• Mainly used in conference call, but introduced also in VoIP networks G.722.2

• providing wideband in 3GPP networks (also called AMR-WB)


Comparison of Wideband codecsG.729.1, G.722 & G.722.2EVRC-WB, AMR-WB & VMR mode0G.722 PLC

Impact of transcoding and tandemmingSelf tandemingtranscoding

Comparison of wideband codecs with narrowband codecs

1

2

3

summary


Comparison of Wideband codecs


Compared Subjective WB quality of G.729.1, G.722 & G.722.2

Extract from the characterisation phase (step 2)Experiment 1

• narrowband

Experiment 2• Purpose : evaluate the performance of G729.1 algorithm with respect to well

known references, in wide band clean speech (free of background noise) conditions with a variety of input levels and frame error rates.

• Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW).

• Languages : French (Canada) & English (US)• Subjects : 32 naïve listeners

Experiment 3• Wideband Music


Compared Subjective WB quality of G.729.1, G.722 & G.722.2 (no FER)

Experiment 2 - clean speech

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Direct G.72264k

G.729.132k

G.729.124k

G.729.114k

AMR-WB

23.85k

AMR-WB

12.65k

MO

S LQ

SW

Dynastat

VoiceAge


Compared Subjective WB quality of G.729.1, G.722 & G.722.2 (FER)

Exp 2 clean speech + FER - Lab E

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

FER = 0% FER = 3% FER = 6% FER = 10%

G.729.1 14k

G.729.1 24k

G.729.1 32k

AMR-WB 12.65k

AMR-WB 23.85k

Exp 2 clean speech + FER - Lab F

1

1.5

2

2.5

3

3.5

4

4.5

5

FER = 0% FER = 3% FER = 6% FER = 10%

G.729.1 14k

G.729.1 24k

G.729.1 32k

AMR-WB 12.65k

AMR-WB 23.85k


Compared Subjective EVRC-WB quality with AMR-WB and VMR Mode-0

Extract from EVRC-WB Characterization test Experiment 1

• Purpose : evaluate the performance of EVRC-WB algorithm with respect to well known references, in wide band clean speech (free of background noise) conditions with a variety of input levels and frame error rates.


• Languages : English (US)• Subjects : 32 naïve listeners

Experiment 2• Purpose : evaluate the performance of EVRC-WB algorithm with respect to

well known references, in wide band noisy speech, and VAD/DTX scheme• Methodology : P.835

Experiment 3 & 4• Narrowband


Compared Subjective EVRC-WB quality with AMR-WB and VMR Mode-0

CT1 Results

1

2

3

4

5

-22 dB level -32 dB level -12 dB level 1% FER withHRPD Rev A

mask

2% FER withHRPD Rev A

3% FER 6% FER 1% D&B+1%packet levelsignaling*

MO

S

EVRC-WB EVRC-WB with DTX AMR-WB @ 12.65 kbps with DTX VMR Mode 0


Performance of G.722 with packet loss concealment

Extract from G.722 PLC Selection test Experiment 1a &1b

• Purpose : evaluate the performance of PLC algorithm with respect to well known references, in wide band clean speech (free of background noise) conditions with a variety of frame error rates (random for exp1a, burst for exp 1b).


• Languages : Japanese, French & English (US)• Subjects : 32 naïve listeners

Experiment 2a &2b• Purpose : evaluate the performance of PLC algorithm with respect to well known

references, in noisy speech, (random for exp2a, burst for exp 2b).


Performance of G.722 with packet loss concealment

1

2

3

4

5

0% 1% 3% 6% 3%+0.1%RBER

PLC A PLC C PLC0 G.729.1 -32k

1

2

3

4

5

0% 1% 3% 6% 3%+0.1%RBER

PLC A PLC C PLC0 G.729.1 -32k

Clean peech, Random FER Clean speech, Bursty FER


Conclusion on Wideband Quality

All these codecs provide high wideband quality that can be roughly divided into 2 categories:

Maximum wideband quality for the most recent codecs at their maximum bit rates: very close to "direct" quality in the test conditions

Slightly lower quality for these codecs when operating at reduced bit rates around 12-14 kbit/s and for G.722 at 64 kbit/s but for much reduced complexity


Impact of transcoding and tandemming


Impact of transcoding/tandeming G.722.2 & G.722

Extract from the characterisation phase of AMR-WBExperiment 1

• Purpose : evaluate the performance of AMRWB algorithm, in wide band clean speech (free of background noise) tandeming conditions with a variety of input levels.


• Languages : Finnish & English • Subjects : 32 naïve listeners

Experiment 2• Purpose : evaluate the performance of AMRWB algorithm, in wide band

clean speech (free of background noise) conditions in transcoding with other wideband standards


• Languages : French & English (US) • Subjects : 32 naïve listeners


Compared Subjective WB quality of G.722 & G.722.2 in self tandeming

1

1.5

2

2.5

3

3.5

4

4.5

5

- singlecoding

self-tandem singlecoding

self-tandem singlecoding

self-tandem

Direct G.722-64kbit/s

G.722-64kbit/s

Mode 2(12.65kbit/s)

Mode 2(12.65kbit/s)

Mode 8(23.85kbit/s)

Mode 8(23.85kbit/s)

Lab = NokiaLab = BT


Compared Subjective WB quality of G.722 & G.722.2 in transcoding

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

- - - - G.722-64 - G.722-64

Direct G.722-64 G.722-48kbit/s

Mode 2(12.65kbit/s)

Mode 2(12.65kbit/s)

Mode 8(23.85kbit/s)

Mode 8(23.85kbit/s)

Lab = FTLab = LMGT


Conclusion on tandemming and transcoding

Codecs self tandemings produce quite limited quality degradations of around 0.2 MOS-LQSW.

Transcodings between different wideband formats produce more significant degradation :

G722↔AMR-WB transcoding quality score 0.2 to 0.4 MOS-LQSW below G.722 64 k quality.


Comparison od wideband codecs with narrowband codecs


Comparison of wideband codecs with narrowband codecs (1)

Extract from the G.729.1 characterisation phase (step1)Experiment 1a

• NarrowbandExperiment 1b

• Purpose : evaluate the performance of G.729.1 algorithm, in wide band clean speech (free of background noise) with a variety of input levels.


• Languages : French & English (US)• Subjects : 32 naïve listeners


Comparison of wideband codecs with narrowband codecs (2)

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

DirectNB

DirectWB

G.729A-8k (NB)

G.722 –48k

(WB)

G.722 –56k

(WB)

G.729.1-14 k(WB)

G.729.1-24 k(WB)

G.729.1-32 k(WB)

MO

S-LQ

SM

Lab: VoiceAgeLab : FT


Conclusion WB versus NB

Results show that wideband voice, even coded at the lowest bit rates of G.722 (48 kbit/s), gets better score than direct narrow band quality with a gap up to +0.5 MOS-LQSM

MOS-LQSM difference between narrow band and wideband direct speech is greater than 1 MOS-LQSM and remain between 0.5 MOS-LQSM and 1 MOS-LQSM between direct narrow band and high quality wideband coded speech.


ReferencesG.729.1 Characterization step 2 references

ITU-T-SG12-TD42rev3(WP1/12), " G.729EV Characterization phase step 2 Quality Assessment Test Plan ", Source: Rapporteurfor Question 7/12, Geneva, 5-13 June 2006ITU-T-SG16-TD258(GEN/16), "LS on testing issues", Source: Rapporteurs Q7/12, Geneva, 14 - 24 November 2006ITU-T-SG16-TD258(GEN/16)-Attachment 2, " Executive summary of G729.1 Characterisation step 2– Experiments 1, 2 & 3. ", Source: France Télécom, Geneva, 14 - 24 November 2006

G.729.1 Characterization step 2 referencesITU-T-SG12-TD22rev2(WP1/12), " G729EV Characterisation/Optimisation step1 Test plan ", Source: Rapporteur for Question 7/12, Geneva, 17-21 October 2005ITU-T-SG16-TD202(GEN/16), " LS on audio issues ", Source: Rapporteurs Q7/12, Geneva, 3 - 13 April 2006ITU-T-SG16-TD202(GEN/16)-Attachment 1, " G729EV Characterisation/Optimisation step1: Summary of results ", Source: France Télécom, Geneva, 3 - 13 April 2006

AMR-WB Characterization referencesETSI TR 126 976 V6.0.0 (2004-12), " Performance characterization of the Adaptive Multi-Rate Wideband (AMR-WB) speech codec", Tdoc S4 (01)0351R1, " AMR Wideband Characterisation Phase 1 Listening Tests, Experiment 1 - BT Results", Source: BT, June 4-8, 2001, Naantali, FinlandTdoc. S4 (01)0326, " Executive Summary from France Télécom R&D for the ETSI AMR-WB Characterisation Phase – Results for Experiments 2 & 5", Source: France Telecom, June 4-8, 2001, Naantali, FinlandTdoc S4 (01)0321, " AMR WB Characterization Experiments 2A and 6A - LMGT Results ", Source: LMGT, June 4-8, 2001, Naantali, FinlandTdoc S4-010353, " Nokia report for AMR-WB characterisation experiments 1A & 6B ", Source: Nokia, June 4-8, 2001, Naantali, Finland

EVRC-WB Characterization referencesITU-T-SG16-TD291(GEN/16) "Updated LS reply on follow-up on embedded extension to G.722.2 and media coding summary database " Source: Chairman SG 16 (on behalf of 3GPP2 TSG-C) Geneva, 14 - 24 November 2006

G.722 Packet Loss Concealment referencesITU-T-SG16-TD217(WP3/16), " Report of Question 10/16 “Software tools for signal processing standardization activities and maintenance and extension of existing voice coding standards”", Source: Rapporteur for Question 10/16, Geneva, 14 - 24 November 2006Extract from the G.729.1 characterisation phase (step1)

quality comparison of wideband coders including … · quality comparison of wideband coders...

Documents