Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

ON LOSSES, PAUSES AND JUMPS AND THE

WIDEBAND E-MODEL

Adil Raja Anna Jagodzinska Vincent Barriac

France Telecom R&D,TECH/OPERA/MOV

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

OUTLINE

1 BACKGROUND AND MOTIVATION

2 VOIP SIMULATION

3 METHODOLOGY

4 PREPARATION OF THE TEST MATERIAL

5 INTRODUCTION TO GP

6 EXPERIMENTAL SETUP

7 RESULTS AND ANALYSIS

Comparison With Multiple Linear RegressionComparison With E-ModelPerformance Evaluation Against Data From Auditory Tests

8 CONCLUSIONS

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

OUTLINE

1 BACKGROUND AND MOTIVATION

2 VOIP SIMULATION

3 METHODOLOGY

4 PREPARATION OF THE TEST MATERIAL

5 INTRODUCTION TO GP

6 EXPERIMENTAL SETUP

7 RESULTS AND ANALYSIS

Comparison With Multiple Linear RegressionComparison With E-ModelPerformance Evaluation Against Data From Auditory Tests

8 CONCLUSIONS

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

OUTLINE

1 BACKGROUND AND MOTIVATION

2 VOIP SIMULATION

3 METHODOLOGY

4 PREPARATION OF THE TEST MATERIAL

5 INTRODUCTION TO GP

6 EXPERIMENTAL SETUP

7 RESULTS AND ANALYSIS

Comparison With Multiple Linear RegressionComparison With E-ModelPerformance Evaluation Against Data From Auditory Tests

8 CONCLUSIONS

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

OUTLINE

1 BACKGROUND AND MOTIVATION

2 VOIP SIMULATION

3 METHODOLOGY

4 PREPARATION OF THE TEST MATERIAL

5 INTRODUCTION TO GP

6 EXPERIMENTAL SETUP

7 RESULTS AND ANALYSIS

Comparison With Multiple Linear RegressionComparison With E-ModelPerformance Evaluation Against Data From Auditory Tests

8 CONCLUSIONS

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

OUTLINE

1 BACKGROUND AND MOTIVATION

2 VOIP SIMULATION

3 METHODOLOGY

4 PREPARATION OF THE TEST MATERIAL

5 INTRODUCTION TO GP

6 EXPERIMENTAL SETUP

7 RESULTS AND ANALYSIS

Comparison With Multiple Linear RegressionComparison With E-ModelPerformance Evaluation Against Data From Auditory Tests

8 CONCLUSIONS

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

OUTLINE

1 BACKGROUND AND MOTIVATION

2 VOIP SIMULATION

3 METHODOLOGY

4 PREPARATION OF THE TEST MATERIAL

5 INTRODUCTION TO GP

6 EXPERIMENTAL SETUP

7 RESULTS AND ANALYSIS

Comparison With Multiple Linear RegressionComparison With E-ModelPerformance Evaluation Against Data From Auditory Tests

8 CONCLUSIONS

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

OUTLINE

1 BACKGROUND AND MOTIVATION

2 VOIP SIMULATION

3 METHODOLOGY

4 PREPARATION OF THE TEST MATERIAL

5 INTRODUCTION TO GP

6 EXPERIMENTAL SETUP

7 RESULTS AND ANALYSIS

Comparison With Multiple Linear RegressionComparison With E-ModelPerformance Evaluation Against Data From Auditory Tests

8 CONCLUSIONS

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

OUTLINE

1 BACKGROUND AND MOTIVATION

2 VOIP SIMULATION

3 METHODOLOGY

4 PREPARATION OF THE TEST MATERIAL

5 INTRODUCTION TO GP

6 EXPERIMENTAL SETUP

7 RESULTS AND ANALYSIS

Comparison With Multiple Linear RegressionComparison With E-ModelPerformance Evaluation Against Data From Auditory Tests

8 CONCLUSIONS

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

BACKGROUND

VoIP listening quality is not only distorted by packet lossand codec related impairments only.

Temporal discontinuities such as pauses and jumps(packet discards) also play a role. (S. Voran, 03)

Packet loss happens due to network congestion.

Jumps and Pauses happen due to the jitter/jitter bufferinteraction.

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

BACKGROUNDLOSS, PAUSE AND JUMPS

0 100 200 300 400 500−6

−4

−2

0

2

4

6

8

10x 10

−3

1 32

FIGURE: A sequence of 3 frames

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

BACKGROUNDLOSS, PAUSE AND JUMPS

0 100 200 300 400 500−6

−4

−2

0

2

4

6

8

10x 10

−3

1 E 3

FIGURE: Loss

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

BACKGROUNDLOSS, PAUSE AND JUMPS

0 100 200 300 400 500−6

−4

−2

0

2

4

6

8

10x 10

−3

1 E 2

FIGURE: Pause

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

BACKGROUNDLOSS, PAUSE AND JUMPS

0 100 200 300 400 500−6

−4

−2

0

2

4

6

8

10x 10

−3

1 3 4

FIGURE: Jump

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

VOIP SIMULATION

Sender ReceiverJitter Loss

FIGURE: VoIP Simulation

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

WEIBULL DISTRIBUTION

VoIP jitter is a self-similar phenomenon that can be modeled bya heavy tailed distribution.Notable distributions are:

Weibull (✓), Pareto, Exponential.

Weibull distribution is characterized by: A shape parameter(A), a scale (B) parameter, and a location parameter.

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

WEIBULL DISTRIBUTION

0 50 100 150 200 250 300 3500

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2x 10

4

FIGURE: A=1

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

WEIBULL DISTRIBUTION

0 10 20 30 40 50 60 70 80 900

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

FIGURE: A=2

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

WEIBULL DISTRIBUTION

0 10 20 30 40 50 60 70 800

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

FIGURE: A=2.5

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

PACKET LOSS

1 (NO LOSS)

0 (LOSS)

p 1-q

q

1-p

FIGURE:

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

LOSS, PAUSE AND JUMP STATE MODEL

A loss, pause jump state model can be learned from anetwork trace analysis or the network emulation.

with state transition probabilities.

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

LOSS, PAUSE AND JUMP STATE MODEL

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

LOSS, PAUSE AND JUMP STATE MODEL

Conversely the state model can be used to generaterealistic loss, pause and jump patterns given realisticvalues for mean loss, pause and jump rates.

For instance:

n2l = l2n ×(mlr)

1−mlr .

l2n = 1 − clp.

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

VOIP SIMULATION SYSTEM

FIGURE: Simulation system for derivation of Ie,WB,eff

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

NETWORK TRAFFIC PARAMETERS

TABLE: Various Network Traffic Parameters

No. Variable Abbreviation1 mean loss rate mlr2 mean burst length – loss mbl_loss3 mean pause rate mpr4 mean burst length – pause mbl_pause5 mean jump rate mjr6 mean burst length – jump mbl_jump7 mean impairment rate mir=mlr+mpr+mjr8 mean burst length impairments mbl_impairment9 equipment impairment factor Ie,WB

10 gradient of the Ie,WB,eff grad

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

Ie,WB,eff VS mir

0 5 10 15 20 250

10

20

30

40

50

60

70

80

MIR

I e,W

B,e

ff

G.722G.729G.711

FIGURE: Ie,WB,eff vs mir for ITU-T G.711, G.729 and G.722

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

VARIOUS TEMPORAL DISCONTINUITY RATES AND THE

RESPECTIVE BURST LENGTHS

TABLE: Various Temporal Discontinuity Rates and The RespectiveBurst Lengths

Temporal Discontinuity Rate Burst Length0 0

0.005 1, 20.01 1, 2, 40.015 1, 2, 30.02 1, 2, 40.025 1, 2, 50.03 1, 2, 30.035 1, 2, 70.04 1, 4, 8

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

INTRODUCTION TO GENETIC PROGRAMMING (GP)

Genetic Programming is a coarse emulation of DarwinianEvolution.

The search space is composed of all the possiblecomputer programs.GP Life Cycle:

1 Create an initial population of computer programs.2 Evaluation.3 Selection.4 Reproduction.5 Evaluation.6 Replacement.7 Continue from 3.

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

A TYPICAL GP BREEDING CYCLE

FIGURE:

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

TABLE: Common GP Parameters among all experiments

Parameter ValueInitial Population Size 300Initial Tree Depth 6Selection LPPTournament Size 2Genetic Operators Crossover and Subtree MutationOperators Probability Type AdaptiveInitial Operator probabilities 0.5 eachSurvival Half ElitismGeneration Gap 1Function Set plus, minus, multiply, divide,sin,

cos, log2, log10, loge, sqrt,power, if

Terminal Set Random real-valued numbersbetween 0.0 and 1.0. Integers (2–10) andNetwork traffic parameters from Table 1.

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

RESULTS AND ANALYSIS

TABLE: Statistical analysis of the GP experiments and derived models

(a) RMSE Statistics for Best Individuals of 50 Runs for Experiments 1, 2 & 3Experiment 1 Experiment 2 Experiment 3

Stats RMSEtr RMSEte Size RMSEtr RMSEte Size RMSEtr RMSEte SizeMean 5.5482 98.9506 26.2800 5.55 18.94 29.34 5.34 13.90 28.18Std.Dev. 0.3514 152.59 11.4661 0.39 70.27 12.3612 0.4612 47.19 9.73Max. 5.97 500 68 6.0084 494.52 73 5.89 333.60 59Min. 4.5409 4.82 11 4.41 4.37 6 4.40 4.38 14

(b) Results of Mann-Whitney-Wilcoxon Significance TestExperiment 1 Experiment 2 Experiment 3

RMSEtr RMSEte Size RMSEtr RMSEte Size RMSEtr RMSEte SizeExperiment 1 0 0 0 0 1 0 1 1 0Experiment 2 0 1 0 0 0 0 1 0 1Experiment 3 1 1 0 1 0 1 0 0 0

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

RESULTS AND ANALYSIS

TABLE: Performance Statistics of the Proposed Models

Training TestModel RMSEsMOS RMSEs Ie,WB,eff σ Ie,WB,eff RMSEs MOS RMSEs Ie,WB,eff σ Ie,WB,effEquation (1) 0.1763 4.8185 0.8840 0.1759 4.8182 0.8805Equation (2) 0.1602 4.4108 0.9038 0.1596 4.3708 0.9028Equation (3) 0.1619 4.4021 0.9042 0.1611 4.3808 0.9023Equation (4) 0.1692 4.6026 0.8948 0.1679 4.5460 0.8944Equation (5) 0.1764 4.8644 0.8816 0.1948 5.4231 0.8781

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

THE PROPOSED MODELS

Ie,WB,eff = (1){

mir × cos(Ie,WB) +

√

mbl_imp

Ie,WB− mir 1/4 − mir

}

×(−163.87)− 9.35

Ie,WB,eff = (2)

√

√

log10(grad)

mir + 9+

(sin(mir ) + mir )log10(grad)

4 −

√ √mpr

mbl_loss+9

√

log10(grad)−

√

log10(grad)mbl_jump+9

×(−0.0933) + 87.1174

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

THE PROPOSED MODELS

Ie,WB,eff = (3)

log10

(

0.54grad + 3 × mir

)

+log10

(

0.74grad +2×mir

)

3

7 × log10

(

0.54grad + 2 × mir + 6.56 −

√

mbl_imp)

+ mir

×(270.37) + 102.40

Ie,WB,eff = (4)

(sin(grad × mir ))

√40×mbl_imp

Ie,WB × 107.43 − 5.94

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

SCATTER PLOTS

0 10 20 30 40 50 60 70 80 900

10

20

30

40

50

60

70

80

90

Ie,WB,eff

−− GP

I e,W

B,e

ff −−

WB

−P

ES

Q

(a)

0 10 20 30 40 50 60 70 80 900

10

20

30

40

50

60

70

80

90

Ie,WB,eff

−− GP

I e,W

B,e

ff −−

WB

−P

ES

Q(b)

FIGURE: Ie,WB,eff predicted by equation (3) vs target Ie,WB,eff for: (a)training data (b) test data

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

PARAMETER SIGNIFICANCE ANALYSIS

Ie,wb grad mlr mbl_loss mjr mbl_jump mpr mbl_pause mir mbl_impairment0

10

20

30

40

50

60

70

80

90

100

Experiment 1Experiment 2

FIGURE: Percentage of the best individuals employing various inputparameters in acceptable runs of each of the two experiments.

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

COMPARISON WITH MULTIPLE LINEAR REGRESSION

Ie,WB,eff = (5)

0.35 × Ie,WB − 0.006 × grad + 383.62 × mir − 1.18 ×

mbl_imp + 34.65

Has inferior performance as opposed to proposed models.

Results are reported in Table 5

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

COMPARISON WITH E-MODEL

Ie,WB,eff = Ie,WB + (129 − Ie,WB)×Ppl

PplBurstR + Bpl

(6)

TABLE: Comparison between the Prediction Accuracies of theE-Model and the Proposed Model

E-Model Equation (3)Codec RMSE RMSE RMSE RMSE(kbps) Bpl train test train testG.711 (64) 22.39 6.7971 6.7003 4.6748 4.5626G.729 (8) 30.50 4.0824 3.8701 3.0513 3.1362G.722 (64) 19.8053 8.1087 8.1510 5.6865 5.6093

Average – 6.3294 6.2405 4.4709 4.4360% PG – – – 29.36 28.92

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

PERFORMANCE EVALUATION AGAINST DATA FROM

AUDITORY TESTS

TABLE: Target Network Impairment Conditions for the Auditory Tests

Condition mlr mbl_loss mjr mbl_jump mpr mbl_pause mir mbl_impairment1 0 0 0 0 0 0 0 02 0.03 1.0 0 0 0 0 0.03 1.03 0.03 4.0 0 0 0 0 0.03 4.04 0 0 0.03 1.0 0 0 0.03 1.05 0 0 0.03 4.0 0 0 0.03 4.06 0 0 0 0 0.03 1.0 0.03 1.07 0 0 0 0 0.03 4.0 0.03 4.08 0.06 1.0 0 0 0 0 0.06 1.09 0.06 4.0 0 0 0 0 0.06 4.010 0 0 0.06 1.0 0 0 0.06 1.011 0 0 0.06 4.0 0 0 0.06 4.012 0 0 0 0 0.06 1.0 0.06 1.013 0 0 0 0 0.06 4.0 0.06 4.014 0.09 1.0 0 0 0 0 0.09 1.015 0.09 4.0 0 0 0 0 0.09 4.016 0 0 0.09 1.0 0 0 0.09 1.017 0 0 0.09 4.0 0 0 0.09 4.018 0 0 0 0 0.09 1.0 0.09 1.019 0 0 0 0 0.09 4.0 0.09 4.020 0.04 4.0 0.04 4.0 0.04 4.0 0.12 12.0

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

PERFORMANCE EVALUATION AGAINST DATA FROM

AUDITORY TESTS

TABLE: Comparison Between the Results of Auditory Tests andWB-PESQ

RMSE MOS σ MOS0.4475 0.8399

TABLE: Comparison between the Prediction Accuracies of theE-Model and the Proposed Model Against Data From Auditory Tests

E-Model Equation (3)Codec (kbps) Bpl Ie,WB RMSE σ RMSE σ

G.711 (64) 25.1 36 18.2549 0.7827 11.8532 0.8182G.729 (8) 19.0 47 34.9341 0.8249 9.4212 0.9309G.722 (64) 7.1 13 46.6618 0.8124 11.8840 0.8966

Overall – – 24.8994 0.3852 11.1129 0.8758

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

PERFORMANCE EVALUATION AGAINST DATA FROM

AUDITORY TESTS

0 0.06 0.12 0.06 0.12 0.06 0.12−50

0

50

100

150

200

MIR

I e,W

B,e

ff

AuditoryE−ModelProposedG.711

G.722

G.729

FIGURE: Ie,WB,eff vs mir derived from auditory tests, E-Model andequation (3) are plotted for G.711, G.722 and G.729.

Background and Motivation VoIP Simulation Methodology Preparation of The Test Material Introduction to GP Experimental

CONCLUSIONS

We have proposed a new methodology that employs GP toderive novel equipment impairment factors.

The poposed models outperform the existing E-Modelformulation.

We have taken into account additional sources ofimpairments; pauses and jumps.

We have also proposed a 4-state loss, pause and jumpMarkov model to characterize VoIP traffic.

The methodology is general and may be augmented withthe results of auditory tests.