state of the art of multimedia quality assessment methods · state of the art of multimedia quality...
TRANSCRIPT
International Telecommunication Union
ITU-T VICA Workshop22-23 July 2005, ITU Headquarters, Geneva
State of the Art ofState of the Art ofMultimedia Quality Multimedia Quality
Assessment MethodsAssessment Methods
Takanori HayashiNTT Service Integration Labs.
2ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Outline
o Present state of multimedia quality assessment methods
o Future trends of quality assessment research
• Multimodality• Multiparty• Wideband
o Examples of NTT’s studies on multimedia quality assessment for audiovisual communication services
3ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Relevant ITU-T/R Recommendationson Multimedia Quality Assessment
ObjectiveSubjective
J.148, P.931P.911, P.920Multimedia
J.144P.910, BT.500Video
P.563, P.862, BS.1387P.800, BS.562Audio
Quality assessment methodMedia
o Multimedia quality assessment is at a reasonably advanced stage.
4ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Existing Subjective Multimedia Quality Assessment Methods (P.911)
o Multimedia quality is assessed in a similar way to individual audio/video qualities.
• Absolute Category Rating (ACR)• Degradation Category Rating (DCR)• Pair Comparison Method (PC)• Single Stimulus Continuous Quality
Evaluation (SSCQE)o Assessment paying attention to the
cross-modal influences is important.• Interactions between differing quality
levels in different modalities
5ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-To Several category judgment scales are
used to evaluate multimedia quality.• Overall audiovisual quality• Individual audio/video qualities• Effort needed to interrupt• Communication difficulty• Acceptability of communication
o Communication quality depends on tasks used in conversational test.
o Assessment considering interactivity and usability is also important.
Existing Subjective Multimedia Quality Assessment Methods (P.920)
6ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Existing Concept Model of Objective Multimedia Quality Evaluation (J.148)
Auditory source
Visualsource
Differential delay
Visual quality
Auditory quality
Multimediaquality
integrationfunction
Task
Multimedia quality
Aq
Aq(Vq)
Vq(Aq)
Vq
o Basic components of an objective multimedia quality model are defined.
7ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of Multimedia Quality Evaluation Model
o ITU-T P.911 Annex• MOSMM=C1MOSAMOSV+C2
o Other multimedia quality evaluation models have been proposed. [1-7]
• MOSMM=C1MOSA+C2MOSV+C3MOSAMOSV+C4! Constants vary with tasks (audiovisual
contents, service environments, etc.).
MOSMM: MOS for multimediaMOSA: MOS for audioMOSV: MOS for videoC: Constant
8ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Framework for ConductingQuality Assessment Research [8]
Sing
le m
ediu
mSi
ngle
med
ium
Media
WidebandWideband
Bandwidth
N:N(Multiparty)(Multiparty)
Communication style
InterdestinationInterdestination
synchronizationsynchronization
InterdestinationInterdestination
quality balancequality balance
Quality Quality assessment for assessment for
••multimodalitymultimodality
••multiparty multiparty
••wideband wideband
IntermediaIntermediasynchronizationsynchronization
IntermediaIntermediaquality balancequality balance
(VoIP)
1:11:1
Conventional research field
Future research field
Mul
tim
odal
Mul
tim
odal
9ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Key Words for Future Trends of Multimedia Quality Assessment (1/3)
o Multimodality
• Combination of multiple media such as audio, video, text, graphics, fax, and telephony in the communication of information
• Key points of quality assessment! Intermedia synchronization! Intermedia quality balance
10ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Key Words for Future Trends of Multimedia Quality Assessment (2/3)
o Multiparty
• Communication style extending from 1:1 to N:N (e.g., instant messaging, teleconferencing, and distributed collaboration services)
• Key points of quality assessment! Interdestination synchronization! Interdestination quality balance
11ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Key Words for Future Trends of Multimedia Quality Assessment (3/3)
o Wideband
• Telecommunications applications having more bandwidth available for higher-quality multimodal services
• Key points of quality assessment! Psychological factors: We need to assess
the richness of high-quality services not only on a one-dimensional scale, like MOS, but also on a multi-dimensional scale.
12ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Multimodality” (1/3)
o Interactive multimodal quality model considering conversational delay [9]
• MOSOA=(C1MOSMM+C2)(C3MOSR+C4)• MOSR=C5exp(-D/C6)+C7
! Constants C5, C6, and C7 depend on conversational task.
1
2
3
4
5
1 2 3 4 5Subjective MOSOA
Esti
mat
ed M
OS O
A
R2: 0.94MOSOA: MOS for overall qualityMOSMM: MOS for multimediaMOSR: MOS for responseD: DelayC: Constant
13ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Multimodality” (2/3)
o Influence of differential delay is different when DA>DV or DA<DV.[10]
Acceptable quality region for response
One-way delay for audio: DA [ms]
One
-way
del
ay f
or v
ideo
: D
V[m
s]
800
600
400
200
2000 400 600 800
Quality can be improved by synchronizing
DA: 100 msDV: 300 ms
DA: 300 msDV: 300 ms
Subjects: 4 expertsTask: Free conversation
14ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Multimodality” (3/3)
o Opinion model for audiovisual communication services is now being discussed in ITU-T SG12. [11]
Multimediaquality
Opi
nion
mod
el f
or
audi
ovis
ual c
om.
serv
icesVideo codec
Video size/resolutionVideo coding bit rateVideo packet loss rateVideo delayVideo jitter
Audio codecAudio echoAudio packet loss rateAudio delayAudio jitter
Video qualityparameters
Audio qualityparameters
15ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Multiparty” (1/2)
o Quality imbalance is one of the multiparty quality degradation factors.
Point 1
Point 2 Point 3
Three-point audiovisual communication
Other-point audiovisual quality:
MOS1-2
Other-point audiovisual quality:
MOS1-3
Overall audiovisual quality: MOS1
Good No GoodQuality
16ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Multiparty” (2/2)
o Overall quality is strongly affected by the inferior quality at another point. [12]
o Other-point audiovisual quality depends on conversation task or roles.
MOS1-2
MO
S 1-3
Overall audiovisual quality: MOS1
Higher
qua
lity
Higher quality
Hig
her
qual
ity
17ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Wideband” (1/3)
o Interactive multimodal quality can be evaluated using a multi-dimensional scale of psychological factors. [11]
Multimediaquality
Video qualityparameters
Audio qualityparameters
Perceptual quality indices
Parameter A
Parameter B
Parameter C
Psychological factor indices
Factor X
Factor Y
Factor Z
Video codecVideo size/resolutionVideo coding bit rateVideo packet loss rateVideo delayVideo jitter, …
Audio codecAudio echoAudio packet loss rateAudio delayAudio jitter, …
18ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Wideband” (2/3)
o Psychological factors were extracted by using the semantic differential (SD) technique and factor analysis. [13]
• Subject’s impression of an audiovisual communication service was evaluated on the basis of 25 pairs of bipolar adjectives on a seven-grade comparison scale.
Slow
NeutralSlightlyRatherExtremely ExtremelyRatherSlightly
Fast
Example of scale
19ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Wideband” (3/3)
o Multimedia quality was formulated as a function of two psychological factors expressing an aesthetic feeling and a feeling of activity. [13, 14]
Mul
tim
edia
qua
lity
g(f 1
,f2)
Video qualityparameters
Audio qualityparameters
Perceptual quality indices
Psychological factor indices
Blurring
Jerkiness
Distortion
Aesthetic feeling
f1(BR,PLRV,PLRA)Video coding bit rate: BRVideo frame rate: FRVideo packet loss rate: PLRVVideo delay: D
Audio packet loss rate: PLRAAudio delay: D
Delay
Lack of sound
Feeling of activity
f2(BR,PLRV, FR,D)
20ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Conclusions
o Multimedia quality assessment is at an advanced stage.
o Perceptual quality assessment methodologies for multimedia communications systems of the next generation are being discussed.
o Three important characteristics of upcoming services have been revealed by recent studies on multimedia quality evaluation models:
• Multimodality, multiparty, and wideband.
21ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
References (1/2)[1] David S. Hands, “A Basic Multimedia Quality Model,” IEEE
Transactions on Multimedia, vol. 6, no. 6, pp. 806 - 816, Dec. 2004.
[2] C. Jones and D. J. Atkinson, “Development of opinion-based audiovisual quality models for desktop video-teleconferencing,”IEEE Sixth Int. Workshop on Quality of Service (IWQoS’98), pp. 196 - 203, May 1998.
[3] A. Joly, N. Montard, and M. Buttin, “Audio-visual quality and interactions between television audio and video,” Sixth International Symposium on Signal Processing and its Applications, Vol. 2, pp. 438 - 441, Aug. 2001.
[4] A. Watson, and M. A. Sasse, “Measuring perceived quality of speech and video in multimedia conferencing applications,”Proceedings of the sixth ACM international conference on Multimedia, Sept. 1998.
[5] ITU-T Contribution COM12-19-E, “Relations between audio, video, and audiovisual quality,” Dec. 1997.
[6] ITU-T Contribution COM12-61-E “Study of the influence of experimental context on the relationship between audio, video, and audiovisual subjective qualities,” Sept. 1998.
22ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
References (2/2)[7] ITU-T Contribution COM12-64-E “Results of an audiovisual
desktop video teleconferencing subjective experiment,” Sept. 1998.
[8] A. Takahashi, H. Yoshino, and N. Kitawaki, “Perceptual QoSAssessment Technologies for VoIP,” IEEE Com. Magazine, pp. 28 - 34, July 2004.
[9] ITU-T Contribution COM12-D14-E, “Example of multimedia quality integration function for videophones,” Jan. 2005.
[10] S. Iai, T. Kurita, and N. Kitawaki, “Quality requirements for multimedia communication services and terminals – interaction of speech and video delays,” Globecom'93, pp. 394 – 398, 1993.
[11] ITU-T Contribution COM12-D13-E, “Proposal on basic concepts of a multimedia quality assessment model,” Jan. 2005.
[12] T. Kurita, “Effects of Conversation Roles on Quality of Multiparty Audiovisual Communication Services,” IEICE Tec. Rep. CQ, July 2005 (in Japanese).
[13] ITU-T Contribution COM12-D15-E, “Example of multimedia quality assessment model for videophones,” Jan. 2005.
[14] K. Yamagishi, and T. Hayashi, “Analysis of psychological factors for quality assessment of interactive multimodal service,”Electronic Imaging 2005, 5666-15, pp. 130 - 138, Jan. 2005.