cmpt 855module 7.0 1 network traffic self-similarity carey williamson department of computer science...

CMPT 855 Module 7.01

Network Traffic Self-SimilarityNetwork Traffic Self-Similarity

Carey WilliamsonCarey Williamson

Department of Computer ScienceUniversity of Saskatchewan


IntroductionIntroduction

A recent measurement study has shown that A recent measurement study has shown that aggregate Ethernet LAN traffic is aggregate Ethernet LAN traffic is self-similarself-similar [Leland et al 1993] [Leland et al 1993]

A statistical property that is very different A statistical property that is very different from the traditional Poisson-based modelsfrom the traditional Poisson-based models

This presentation: definition of network This presentation: definition of network traffic self-similarity, Bellcore Ethernet traffic self-similarity, Bellcore Ethernet LAN data, implications of self-similarityLAN data, implications of self-similarity


Measurement MethodologyMeasurement Methodology

Collected lengthy traces of Ethernet LAN Collected lengthy traces of Ethernet LAN traffic on Ethernet LAN(s) at Bellcoretraffic on Ethernet LAN(s) at Bellcore

High resolution time stampsHigh resolution time stamps Analyzed statistical properties of the resulting Analyzed statistical properties of the resulting

time series datatime series data Each observation represents the number of Each observation represents the number of

packets (or bytes) observed per time interval packets (or bytes) observed per time interval (e.g., 10 4 8 12 7 2 0 5 17 9 8 8 2...)(e.g., 10 4 8 12 7 2 0 5 17 9 8 8 2...)


Self-Similarity: The IntuitionSelf-Similarity: The Intuition

If you plot the number of packets observed If you plot the number of packets observed per time interval as a function of time, then per time interval as a function of time, then the plot looks ‘‘the same’’ regardless of the plot looks ‘‘the same’’ regardless of what interval size you choose what interval size you choose

E.g., 10 msec, 100 msec, 1 sec, 10 sec,...E.g., 10 msec, 100 msec, 1 sec, 10 sec,... Same applies if you plot number of bytes Same applies if you plot number of bytes

observed per interval of timeobserved per interval of time


Self-Similarity: The IntuitionSelf-Similarity: The Intuition In other words, self-similarity implies a In other words, self-similarity implies a

‘‘fractal-like’’ behaviour: no matter what ‘‘fractal-like’’ behaviour: no matter what time scale you use to examine the data, you time scale you use to examine the data, you see similar patternssee similar patterns

Implications:Implications: Burstiness exists across many time scalesBurstiness exists across many time scales No natural length of a burstNo natural length of a burst Traffic does not necessarilty get ‘‘smoother” Traffic does not necessarilty get ‘‘smoother”

when you aggregate it (unlike Poisson traffic)when you aggregate it (unlike Poisson traffic)


Self-Similarity: The MathematicsSelf-Similarity: The Mathematics Self-similarity is a rigourous statistical Self-similarity is a rigourous statistical

property (i.e., a lot more to it than just the property (i.e., a lot more to it than just the pretty ‘‘fractal-like’’ pictures)pretty ‘‘fractal-like’’ pictures)

Assumes you have time series data with Assumes you have time series data with finite mean and variance (i.e., covariance finite mean and variance (i.e., covariance stationary stochastic process)stationary stochastic process)

Must be a Must be a very long very long time series time series (infinite is best!)(infinite is best!)

Can test for presence of self-similarityCan test for presence of self-similarity


Self-Similarity: The MathematicsSelf-Similarity: The Mathematics Self-similarity manifests itself in several Self-similarity manifests itself in several

equivalent fashions:equivalent fashions: Slowly decaying varianceSlowly decaying variance Long range dependenceLong range dependence Non-degenerate autocorrelationsNon-degenerate autocorrelations Hurst effectHurst effect


Slowly Decaying VarianceSlowly Decaying Variance

The variance of the sample decreases more The variance of the sample decreases more slowly than the reciprocal of the sample sizeslowly than the reciprocal of the sample size

For most processes, the variance of a For most processes, the variance of a sample diminishes quite rapidly as the sample diminishes quite rapidly as the sample size is increased, and stabilizes soonsample size is increased, and stabilizes soon

For self-similar processes, the variance For self-similar processes, the variance decreases decreases very slowlyvery slowly, even when the , even when the sample size grows quite largesample size grows quite large


Variance-Time PlotVariance-Time Plot

The ‘‘variance-time plot” is one means The ‘‘variance-time plot” is one means to test for the slowly decaying to test for the slowly decaying variance propertyvariance property

Plots the variance of the sample versus the Plots the variance of the sample versus the sample size, on a log-log plotsample size, on a log-log plot

For most processes, the result is a straight For most processes, the result is a straight line with slope -1line with slope -1

For self-similar, the line is much flatterFor self-similar, the line is much flatter

Variance-Time PlotVariance-Time PlotV

aria

nce

m


aria

nce

m

Variance of sampleon a logarithmic scale

0.0001

0.001

10.0

0.01

100.0


aria

nce

m

Sample size mon a logarithmic scale

1 10 100 10 10 10 104 5 6 7


aria

nce

m


aria

nce

m

Slope = -1for most processes


aria

nce

m


aria

nce

m

Slope flatter than -1for self-similar process


Long Range DependenceLong Range Dependence

Correlation is a statistical measure of the Correlation is a statistical measure of the relationship, if any, between two random relationship, if any, between two random variablesvariables

Positive correlation: both behave similarlyPositive correlation: both behave similarly Negative correlation: behave as oppositesNegative correlation: behave as opposites No correlation: behaviour of one is No correlation: behaviour of one is

unrelated to behaviour of otherunrelated to behaviour of other


Long Range Dependence (Cont’d)Long Range Dependence (Cont’d)

Autocorrelation is a statistical measure of the Autocorrelation is a statistical measure of the relationship, if any, between a random relationship, if any, between a random variable and itself, at different time lagsvariable and itself, at different time lags

Positive correlation: big observation usually Positive correlation: big observation usually followed by another big, or small by smallfollowed by another big, or small by small

Negative correlation: big observation usually Negative correlation: big observation usually followed by small, or small by bigfollowed by small, or small by big

No correlation: observations unrelated No correlation: observations unrelated


Long Range Dependence (Cont’d)Long Range Dependence (Cont’d)

Autocorrelation coefficient can range Autocorrelation coefficient can range between +1 (very high positive correlation) between +1 (very high positive correlation) and -1 (very high negative correlation)and -1 (very high negative correlation)

Zero means no correlationZero means no correlation Autocorrelation function shows the value of Autocorrelation function shows the value of

the autocorrelation coefficient for different the autocorrelation coefficient for different time lags ktime lags k

Autocorrelation FunctionAutocorrelation Function

+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient

Maximum possible positive correlation


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient

Maximum possiblenegative correlation


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient

No observedcorrelation at all


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient

Significant positivecorrelation at short lags


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient

No statistically significantcorrelation beyond this lag


Long Range Dependence (Cont’d)Long Range Dependence (Cont’d) For most processes (e.g., Poisson, or For most processes (e.g., Poisson, or

compound Poisson), the autocorrelation compound Poisson), the autocorrelation function drops to zero very quickly function drops to zero very quickly (usually immediately, or exponentially fast)(usually immediately, or exponentially fast)

For self-similar processes, the For self-similar processes, the autocorrelation function drops very slowly autocorrelation function drops very slowly (i.e., hyperbolically) toward zero, but may (i.e., hyperbolically) toward zero, but may never reach zeronever reach zero

Non-summable autocorrelation functionNon-summable autocorrelation function


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient

Typical short-rangedependent process


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient

Typical long-range dependent process


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient

Typical long-range dependent process

Typical short-rangedependent process


Non-Degenerate AutocorrelationsNon-Degenerate Autocorrelations For self-similar processes, the For self-similar processes, the

autocorrelation function for the aggregated autocorrelation function for the aggregated process is indistinguishable from that of the process is indistinguishable from that of the original processoriginal process

If autocorrelation coefficients match for all If autocorrelation coefficients match for all lags k, then called lags k, then called exactlyexactly self-similar self-similar

If autocorrelation coefficients match only If autocorrelation coefficients match only for large lags k, then called for large lags k, then called asymptoticallasymptotically y self-similarself-similar


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient

Original self-similarprocess


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient


Aggregated self-similarprocess


AggregationAggregation

Aggregation of a time series X(t) means Aggregation of a time series X(t) means smoothing the time series by averaging the smoothing the time series by averaging the observations over non-overlapping blocks observations over non-overlapping blocks of size m to get a new time series X (t)of size m to get a new time series X (t)m


Aggregation: An ExampleAggregation: An Example

Suppose the original time series X(t) Suppose the original time series X(t) contains the following (made up) values:contains the following (made up) values:

2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1... Then the aggregated series for Then the aggregated series for m = 2 m = 2 is:is:




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1... Then the aggregated series for m = 2 is:Then the aggregated series for m = 2 is:





4.54.5





4.5 8.04.5 8.0





4.5 8.0 2.54.5 8.0 2.5





4.5 8.0 2.5 5.04.5 8.0 2.5 5.0





4.5 8.0 2.5 5.0 6.0 7.5 7.0 4.0 4.5 5.0...4.5 8.0 2.5 5.0 6.0 7.5 7.0 4.0 4.5 5.0...




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...

Then the aggregated time series for Then the aggregated time series for m = 5 m = 5 is:is:




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...

Then the aggregated time series for m = 5 is:Then the aggregated time series for m = 5 is:




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...


6.06.0




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...


6.0 4.46.0 4.4




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...


6.0 4.4 6.4 4.8 ...6.0 4.4 6.4 4.8 ...




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...

Then the aggregated time series for Then the aggregated time series for m = 10 m = 10 is:is:




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...


5.25.2




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...


5.2 5.65.2 5.6




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1...


5.2 5.6 ...5.2 5.6 ...


+1

-1

0

lag k0 100Aut

ocor

rela

tion

Coe

ffic

ient


Aggregated self-similarprocess


The Hurst EffectThe Hurst Effect For almost all naturally occurring time For almost all naturally occurring time

series, the rescaled adjusted range statistic series, the rescaled adjusted range statistic (also called the (also called the R/S statisticR/S statistic) for sample size ) for sample size n obeys the relationshipn obeys the relationship

E[R(n)/S(n)] = c n E[R(n)/S(n)] = c n

where:where:

R(n) = max(0, W , ... W ) - min(0, W , ... W )R(n) = max(0, W , ... W ) - min(0, W , ... W )

S (n) is the sample variance, andS (n) is the sample variance, and

W = W = X - k X for k = 1, 2, ... nX - k X for k = 1, 2, ... n

H

11 nn

2

k i ni =1

n


The Hurst Effect (Cont’d)The Hurst Effect (Cont’d)

For models with only short range For models with only short range dependence, H is almost always 0.5dependence, H is almost always 0.5

For self-similar processes, 0.5 < H < 1.0For self-similar processes, 0.5 < H < 1.0 This discrepancy is called the This discrepancy is called the Hurst EffectHurst Effect, ,

and H is called the and H is called the Hurst parameterHurst parameter Single parameter to characterize Single parameter to characterize

self-similar processesself-similar processes


R/S Statistic: An ExampleR/S Statistic: An Example


2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 12 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1 There are 20 data points in this exampleThere are 20 data points in this example




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 12 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1 There are 20 data points in this exampleThere are 20 data points in this example For R/S analysis with For R/S analysis with n = 1n = 1, you get 20 , you get 20

samples, each of size 1:samples, each of size 1:




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 12 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1 There are 20 data points in this exampleThere are 20 data points in this example For R/S analysis with n = 1, you get 20 For R/S analysis with n = 1, you get 20


Block 1: X = 2, W = 0, R(n) = 0, S(n) = 0Block 1: X = 2, W = 0, R(n) = 0, S(n) = 0n 1




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 12 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1 There are 20 data points in this exampleThere are 20 data points in this example For R/S analysis with n = 1, you get 20 For R/S analysis with n = 1, you get 20


Block 2: X = 7, W = 0, R(n) = 0, S(n) = 0Block 2: X = 7, W = 0, R(n) = 0, S(n) = 0n 1




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 12 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1 For R/S analysis with For R/S analysis with n = 2n = 2, you get 10 , you get 10




Suppose the original time series X(t) contains Suppose the original time series X(t) contains the following (made up) values:the following (made up) values:

2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 12 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1 For R/S analysis with n = 2, you get 10 For R/S analysis with n = 2, you get 10


Block 1: X = 4.5, W = -2.5, W = 0, Block 1: X = 4.5, W = -2.5, W = 0,

R(n) = 0 - (-2.5) = 2.5, S(n) = 2.5,R(n) = 0 - (-2.5) = 2.5, S(n) = 2.5,

R(n)/S(n) = 1.0R(n)/S(n) = 1.0

n 1 2




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 12 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1 For R/S analysis with n = 2, you get 10 For R/S analysis with n = 2, you get 10


Block 2: X = 8.0, W = -4.0, W = 0, Block 2: X = 8.0, W = -4.0, W = 0,

R(n) = 0 - (-4.0) = 4.0, S(n) = 4.0,R(n) = 0 - (-4.0) = 4.0, S(n) = 4.0,

R(n)/S(n) = 1.0R(n)/S(n) = 1.0

n 1 2




2 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 12 7 4 12 5 0 8 2 8 4 6 9 11 3 3 5 7 2 9 1 For R/S analysis with n = 3, you get 6 samples, For R/S analysis with n = 3, you get 6 samples,

each of size 3:each of size 3:

Block 1: X = 4.3, W = -2.3, W = 0.3, W = 0 Block 1: X = 4.3, W = -2.3, W = 0.3, W = 0

R(n) = 0.3 - (-2.3) = 2.6, S(n) = 2.05,R(n) = 0.3 - (-2.3) = 2.6, S(n) = 2.05,

R(n)/S(n) = 1.30R(n)/S(n) = 1.30

n 1 2 3






Block 2: X = 5.7, W = 6.3, W = 5.7, W = 0 Block 2: X = 5.7, W = 6.3, W = 5.7, W = 0

R(n) = 6.3 - (0) = 6.3, S(n) = 4.92,R(n) = 6.3 - (0) = 6.3, S(n) = 4.92,

R(n)/S(n) = 1.28R(n)/S(n) = 1.28

n 1 2 3






Block 1: X = 6.0, W = -4.0, W = -3.0,Block 1: X = 6.0, W = -4.0, W = -3.0,

W = -5.0 , W = 1.0 , W = 0, S(n) = 3.41,W = -5.0 , W = 1.0 , W = 0, S(n) = 3.41,

R(n) = 1.0 - (-5.0) = 6.0, R(n)/S(n) = 1.76R(n) = 1.0 - (-5.0) = 6.0, R(n)/S(n) = 1.76

n 1 2

3 4 5






Block 2: X = 4.4, W = -4.4, W = -0.8,Block 2: X = 4.4, W = -4.4, W = -0.8,

W = -3.2 , W = 0.4 , W = 0, S(n) = 3.2,W = -3.2 , W = 0.4 , W = 0, S(n) = 3.2,

R(n) = 0.4 - (-4.4) = 4.8, R(n)/S(n) = 1.5R(n) = 0.4 - (-4.4) = 4.8, R(n)/S(n) = 1.5

n 1 2

3 4 5





sample of size 20:sample of size 20:


R/S PlotR/S Plot

Another way of testing for self-similarity, Another way of testing for self-similarity, and estimating the Hurst parameterand estimating the Hurst parameter

Plot the R/S statistic for different values of n, Plot the R/S statistic for different values of n, with a log scale on each axiswith a log scale on each axis

If time series is self-similar, the resulting plot If time series is self-similar, the resulting plot will have a straight line shape with a slope H will have a straight line shape with a slope H that is greater than 0.5that is greater than 0.5

Called an R/S plot, or R/S pox diagramCalled an R/S plot, or R/S pox diagram

R/S Pox DiagramR/S Pox DiagramR

/S S

tati

stic

Block Size n


/S S

tati

stic

Block Size n

R/S statistic R(n)/S(n)on a logarithmic scale


/S S

tati

stic

Block Size n

Sample size non a logarithmic scale


/S S

tati

stic

Block Size n


/S S

tati

stic

Block Size n

Slope 0.5


/S S

tati

stic

Block Size n

Slope 0.5

Slope 1.0


/S S

tati

stic

Block Size n

Slope 0.5

Slope 1.0Self-similarprocess


/S S

tati

stic

Block Size n

Slope 0.5

Slope 1.0Slope H (0.5 < H < 1.0)(Hurst parameter)


SummarySummary Self-similarity is an important mathematical Self-similarity is an important mathematical

property that has recently been identified as property that has recently been identified as present in network traffic measurementspresent in network traffic measurements

Important property: burstiness across many Important property: burstiness across many time scales, traffic does not aggregate welltime scales, traffic does not aggregate well

There exist several mathematical methods There exist several mathematical methods to test for the presence of self-similarity, to test for the presence of self-similarity, and to estimate the Hurst parameter Hand to estimate the Hurst parameter H

There exist models for self-similar trafficThere exist models for self-similar traffic

cmpt 855module 7.0 1 network traffic self-similarity carey williamson department of computer science...

Documents

variancetime plot n

variancetime plot variance

intuition n

introduction n

pictures n

sample size n

variance of sample

mathematics n selfsimilarity