1

Presented by Jari Korhonen

Centre for Quantifiable Quality of Service in Communication Systems (Q2S)Norwegian University of Science and Technology (NTNU)

Research Activities for Quality of Experience in

Networked Multimedia within Q2S

2

Outline

About Q2S Research topics and vision at Q2S

o Research directions and visiono Subjective / Objective assessment of Audio / Video /

Audiovisual content

Research highlights: comparing apples and orangeso Subjective comparison of source and channel distortion in

video streaming

Conclusions

3

• Centre of Excellence in Quantifiable Quality of Service in Communication Systems – Q2S

• Funded by Norwegian Research Council• Supported by Telenor R&I• Hosted by Norwegian University of

Science and Technology• 52 people (7 profs., post-docs, PhD

students and administration)• Basic research and laboratory

experimentation• Main Goals:

– Network Media Handling – QoS and QoE assessment and monitoring – QoS mechanisms for dynamic networks.

Q2S

4

Q2S Research

Networked Media Handlingo Technology to present, manipulate and evaluate multimedia contento Focus on media representation, error protection, perceived quality,

functional placement

Quality assessment and monitoringo Measuring methods and models for QoS and QoEo Measurement of perceived QoS and QoE, measurement methods and

architecture, and traffic performance

QoS mechanisms for dynamic networkso Mechanisms that affect QoS, emphasis on heterogeneous and dynamic

environmento Focus on dependability, security, resource allocation

5

Research Directions (QoE)

QoE

Applications

Virtual worldsSerious gamingBroadcastingMedicalTourism...

Content

User generatedInteractivityIdentity and security

Technology

Network Media Handling3D Media

Perception

Cognitive technologies

MarketsBusiness Models

Creativity

Implem

entation

Modeling and

assessment Su

bjec

tive

test

met

hods

6

Facilities

7

QoS and QoE modeling and assessment

Development of Subjective Test Methods and Objective Models- Multimodal perception of audio and video- Content dependency

- Perceived audio-visual quality assessment for future media (multimodal media with complex scenes (HDTV, UHDTV, NHX, 3D, etc.))- Design of quantifiable metrics for perceived quality (audio, video, audiovisual, …)

Use of Objective Models- Automated monitoring of end-user perceived quality- Using “Perceived QoS” to adapt and enhance system performance- Objective network measurements of end-to-end QoS

8

Research highlights: Comparing apples to oranges In multimedia communications applications (such as video

streaming), both source and channel distortion may appearo Source distortion is derived from lossy compressiono Channel distortion is caused by transmission errors (packet losses

and/or bit errors)

Encoding (compression)

Transmission channel

Decoding and error

concealment

Original quality video

Compressed video with

source distortion

Compressed video with source and

channel distortion

Reproduced video with source and

channel distortion

Video quality experienced by user

9

Motivation – comparison of source and channel distortion

Qualitative characteristics of source and channel distortion are differento Source distortion impacts the

overall quality (see the upper image)

o Channel distortion typically appear in spatially and temporally limited areas (see the lower image)

o Classical example of ”comparing apples and oranges” applies

Source distortion

Channel distortion

10

Motivation – limitations of known quality assessment methods It is challenging to compare the perceptual impact of

source and channel distortiono Objective metrics, such as PSNR, typically give reliable

results only when same type of distortions are comparedo Traditional subjective metrics and methodologies require

quite a lot of work and reliability may be questionable• Different persons may understand subjective scales differently

There is a need for new subjective quality assessment method!

11

Proposed test method Instead of rating the difference between test sequence

and anchor sequence (traditional double-stimulus impairment scaling), we use adjustable stimuluso Named as double-stimulus adjustable quality fixed anchor

(DSAQFA)o User adjusts one of the stimuli so that its quality is (as closely

as possible) similar to the non-adjustable anchor sequenceo Allows comparison between different types of distortionso Minimizes the need for training and eliminates the personal

differences how quality labels are understood

12

Functionality of the test program Anchor sequence and

adjustable sequence played in sync

Adjustable sequence: different source distortion levels the user can choose between

When ready with adjustment, user presses ’ok’, result is recorded and system moves to the next test case

Quality level 2

...

Quality level 1

Quality level 16

Adjustable video player

Anchor video player

Anchor video

Sync

Quality level adjustment

13

UI of the test program

Adjustable sequence

Anchor sequence

Slider for adjusting quality,

and ok-button

14

Generation of test sequences Adjustable sequences: original video clip is encoded

with H.264/AVC using 16 different quantization parameter (QP), varying from 24 (best quality) to 51 (worst quality)

Anchor sequences: bit errors inserted in encoded files using Gilbert-Elliot model, to create channel distortion

H.264/AVC encoder

Original video

Channel error

simulator

H.264/AVC decoder

Resulting video

Quantization Parameter (QP)

Gilbert-Elliot model parameters

15

Voted and anchor PSNR in selected cases

H M1M2 L H M1M2 L H M1M2 L H M1M2 L25

30

35

40

45

PS

NR

[d

B]

Test Case

Comparison of voted and anchor PSNR

Akiyo Harbour Ice Bus

O = Anchor PSNR

16

Discussion of results

PSNR seems to overestimate the quality, when source distortion in anchor is low and there is some channel distortion

However, this difference seems to disappear when the source distortion in anchor gets lower

Content matters: channel distortion seems to be less annoying (compared to PSNR) for sequences with high temporal and spatial activity

Also a lot of individual variance: some test subjects give constantly higher or lower ratings than average

Ongoing research: other objective metrics than PSNR included

17

Concluding summary

General introduction of NTNU-Q2S Research areas and vision

o Emphasis of presentation on quality metrics and assessment for networked video

Research highlightso Comparing apples and oranges: novel subjective quality

assessment method for comparing video sequences with different types of distortion (source and channel)

18

QoMEX ’10The Second International Workshop on Quality of

Multimedia Experience

June 21-23, 2010Trondheim, Norway

Topics of interest (not limited to):o User experience assessment and enhancemento Visual and auditory user experienceo QoE for virtual, augmented and mixed realitieso Link between QoS, QoE and acceptanceo Psychological and social dimensions of QoEo Standardization acitivities in multimedia quality evaluation

Important dates:o Submission deadline: January 31, 2010o Notification of acceptance: April 1, 2010

More information: qomex2010.org

QoMEX