dept. of mobile systems engineering junghoon kim

Metrics for Evaluating Video Streaming Qualityin Lossy IEEE 802.11 Wireless Networks

Dept. of Mobile Systems EngineeringJunghoon Kim

OutlinePaper InfoIntroductionBackgroundMotivationIdeaExperimentsEvaluationContribution

Paper InfoIEEE INFOCOM 2010

The 29th conference on computer communica-tions sponsored by IEEE communications society

March 15-19, 2010, San Diego, CA, USAAuthors

An (Jack) Chan, Kai Zeng, Prasant Mohapatra Dept. of Computer Science University of California, Davis

Sung-Ju Lee, Sujata Banerjee Multimedia Communication & Networking Lab Hewlett-Packard Labs

IntroductionImportant issue

Multimedia streaming is becoming one of the most popular applications recently

Video streaming over WLANs in very commonVideo quality can be measured objectively and

automatically by a computer program It is important to government and industries For specification of system performance require-

ments Comparison of competing service offerings

IntroductionPeak Signal-to-Noise Ratio (PSNR)

simplest and the most widely used video quality evaluation methodology

Problem of traditional PSNRFail to capture the packet loss characteristics of

wireless networksNon-linearity of the human visual system

MPSNR (Modification of PSNR)Retaining the simplicity of PSNR calculationHandles video frame losses

IntroductionDeriving two specific objective video quality

metricsPSNR-based Objective MOS (POMOS)Rates-based Objective MOS (ROMOS)Demonstrate high correlation with MOS

Our metrics evaluate video streaming quality in wireless networks with a much higher accu-racy

BackgroundMean Opinion Score (MOS)

Measured through each viewers giving a score ranging from one to five

Arithmetic mean of all these individual scoresPros

MOS is subjective metricCons

Expensive process Needs a large number of viewers Controlled evaluation environments

BackgroundPeak Signal-to-Noise Ratio (PSNR)

Most widely used objective video quality metric

MSE : Mean Squared Error

BackgroundPeak Signal-to-Noise Ratio (PSNR)

Problem A missing frame results in the latter frames in

shifted positions when compared with the reference video

MotivationInaccuracy in the existing PSNR calculation

Average PSNR value of the reference video : 100dB

Video streaming A : 38dBVideo streaming B : 40dB

(a) Reference video (b) Video streaming A

(c) Video streaming B

IdeaMPSNR

Modification of PSNRAdd matching process in the correct PSNR cal-

culationTwo ways

An optimized algorithm for matching corresponding frames

A heuristic algorithm for matching corresponding frames

IdeaAn optimized algorithm

Assumption The sum of PSNR of all frames in a streamed video is

the maximum when all the frames are correctly matched with the corresponding frames in the refer-ence video

Each frame in a streamed video must have a matched frame in the reference video

We consider a global maximization of the sum of PSNR

IdeaAn optimized algorithm (Cont’d)

Define Maximum total PSNR value achieved when a

streamed video with j frames is matched to the ref-erence video with i frames

Define PSNR value of frame x and frame y

If no match can be found for a frame in the ref-erence video, we ignore the frame in the calcu-lation of the total PSNR value

IdeaAn optimized algorithm (Cont’d)

Three possible cases for the last match in two videos But, Case 3 would never happen

Recurrence equation

IdeaAn optimized algorithm (Cont’d)

Use dynamic programming! Time complexity :

g : the total number of frames lost during streaming n : the number of frames in the streamed video

Given a streamed video of 40 seconds (1000 frames) with 20 frames lost (about 2% frame loss rate), a personal computer with 2.8GHz CPU and 1GB RAM Traditional PSNR : less than 2 seconds Optimized algorithm : about 20 seconds

We need a faster algorithm!!!

IdeaA heuristic algorithm

Define The PSNR value calculated for frame j in the

streamed video when it is compared with frame i in the reference video

Define The set containing the continuous frames in the ref-

erence video when frame j in the streamed video is processed

Define The PSNR value of the frame j in the streamed video

IdeaA heuristic algorithm (Cont’d)

A parameter called PSNR threshold, thresh To mitigate this problem

Frame j in the streamed video is distorted severely and has a larger similarity to a non-corresponding frame k than to the actual corresponding frame h

Take the maximum only if it is greater than thresh Otherwise, we will regard the first frame in as the

matched frame

IdeaA heuristic algorithm (Cont’d)

Time complexity : t : the number of different thresh tried w : window size n : the total number of frames in the streamed video

t and w are small constants. Therefore, time complexity is

Previous experiment Traditional PSNR : less than 2 seconds Heuristic algorithm : about 4 seconds

IdeaMeasuring other parameters

Distorted frame rate Averaged PSNR of distorted frames Frame loss rate

ExperimentsCollecting videos of dif-

ferent qualityA total of 40 streamed

videos with different qualities 30 video clips in the

training set 10 video clips in the vali-

dation set

(a) Streaming with intra-flow interfer-ence

(b) Streaming with inter-flow interfer-ence

(c) Streaming with background data flow

ExperimentsCollecting subjective evaluation for video

qualityEngaged 21 volunteers

Diversity was taken into account Age : from 20 to 45 Occupation : from university undergraduate students

to laboratory techniciansFor each video clip, average the quality scores

given by the subjects and obtain MOS

ExperimentsCollecting subjective evaluation for video

quality

MOS and 95% confidence intervals of videos in the train-ing set

ExperimentsDeriving metrics from subjective evaluation

and MPSNRPSNR-based Objective MOS (POMOS)

Define The average PSNR calculated from MPSNR

Define By setting the window size to one

ExperimentsDeriving metrics from subjective evaluation

and MPSNR (Cont’d)PSNR-based Objective MOS (POMOS)

Use the linear model package of the statistics tool R

ExperimentsDeriving metrics from subjective evaluation

and MPSNR (Cont’d)Rates-based Objective MOS (ROMOS)

To mitigate this problem Assigned a PSNR of 100dB for the perfect frames

ExperimentsDeriving metrics from subjective evaluation

and MPSNR (Cont’d)Rates-based Objective MOS (ROMOS)

Use the linear model package of the statistics tool R

EvaluationEvaluation of objective metrics

Pearson correlation (= correlation coefficient) A heuristic algorithm

: 0.8666 : 0.9346

An optimized algorithm : 0.8838 : 0.9509

EvaluationEvaluation of objective metrics

ContributionIdentify the detrimental impact of packet

losses during video streaming on video quality metric, such as PSNR

Propose a simple objective video quality eval-uation methodology, MPSNR, that alleviates the inaccuracy caused by packet loss

Derive two specific video quality metrics that provide a tool for evaluating video streaming over lossy wireless networks