on evaluation of video quality metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · simple...
TRANSCRIPT
![Page 1: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/1.jpg)
On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík*, Tunç O. Aydın, Karol Myszkowski, and Hans-Peter Seidel
![Page 2: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/2.jpg)
January 26, 2011
� Quality Assessment– In Computer Graphics
� Proposed Dataset– Reference-test video pairs
– LDR-LDR, HDR-HDR, LDR-HDR
� Example Evaluation– 4 VQMs
� Conclusion and Future Work
Outline
On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 3: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/3.jpg)
FR Quality Assessment
Ratethe
Quality
Subjective Studies: + Reliable - Time Consuming
Reference Test
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 4: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/4.jpg)
Simple Metrics
MSE = 280 MSE = 280 MSE = 280 !
Random Noise Blur ~15% DecreasedLuminance
Reference
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 5: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/5.jpg)
Advanced (HVS Based) Metrics
Probability of Detection:
~15% DecreasedLuminance
Random NoiseDistortionMap
DistortionMap
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 6: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/6.jpg)
January 26, 2011
� Input data + Subjective responses = dataset
� Datasets
– Simpler evaluations
– Reproducible evaluations
– Should be publicly available
– Should comprise typical artifacts
Validation of Metrics
On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 7: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/7.jpg)
January 26, 2011
� VQEG FRTV Phase 1 [VQEG ‘00]
� LIVE video db [Seshadrinathan et al. ‘09]
� LDR videos only
� Focus on compression/transmission related artifacts
� Subjective responses: only overall quality rating
Available Video Datasets
On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 8: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/8.jpg)
VQA in Computer Graphics
� User studies take time & require expertise in psychophysics– Usually only informal studies
– Objective metrics help in this
– Objective metrics can be applied in algorithms
� CG– LDR, HDR, LDR/HDR
– Localized distortion maps
– Specific artifacts
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 9: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/9.jpg)
Evaluation of Rendering Methods
No temporal filteringWith temporal filtering[Herzog et al. 2010]
Predicted distortion map
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 10: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/10.jpg)
Evaluation of Rendering Qualities
High quality Low quality Predicted distortion map
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 11: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/11.jpg)
HDR Video
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
HDR video LDR videoTone mapping
![Page 12: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/12.jpg)
Evaluation of HDR Compression
Medium Compression High Compression
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 13: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/13.jpg)
Evaluation of Tone Mapping
[Fattal et al. 2002] Detail Amplification Detail Loss
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
No suitable dataset available
![Page 14: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/14.jpg)
Proposed Dataset� LDR-LDR, HDR-HDR, HDR-LDR
� Including subjective distortion maps
� “2.5D videos”
� Temporal noise, HDR video compression, tone mapping
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 15: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/15.jpg)
Proposed Dataset (cont.)� 9 stimuli (reference-test video pairs)
– 1 LDR-LDR, 2 HDR-LDR, 6 HDR-HDR
– 60 frames each, 24 fps
� Sub-, Near-, Supra-threshold distortions
� http://www.mpi-inf.mpg.de/resources/hdr/quality
� ~900MB
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 16: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/16.jpg)
Subjective Data Acquisition
(1) Show videos side-by-sideon a HDR Display
(2) Subjects mark regionswhere they detect differences
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
� 16 subjects, calibrated Brightside DR37-P display
![Page 17: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/17.jpg)
Example EvaluationSubj. Response
Average distortion
DRIVQM[Aydin et al.’10]
PDM[Winkler’05]
HDRVDP[Mantiuk al.‘05]
DRIVDP[Aydin et al.’08]
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 18: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/18.jpg)
Example Evaluation Results - CC
Stimulus DRIVQM PDM HDRVDP DRIVDP
1 0.765 -0.0147 0.591 0.488
2 0.883 0.686 0.673 0.859
3 0.843 0.886 0.0769 0.865
4 0.815 0.0205 0.211 -0.0654
5 0.844 0.565 0.803 0.689
6 0.761 -0.462 0.709 0.299
7 0.879 0.155 0.882 0.924
8 0.733 0.109 0.339 0.393
9 0.753 0.368 0.473 0.617
Average 0.809 0.257 0.528 0.563
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 19: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/19.jpg)
Comparison: Test Scene� HDR Scene tone
mapped with [Pattanaik 2000]
� Spatio-temporal distortion– Random pixel noise
filtered with a Gaussian.
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 20: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/20.jpg)
Metric Comparison LDR-LDR
DRIVQM PDM [Winkler 2005]
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 21: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/21.jpg)
Metric Comparison HDR-HDR
DRIVQM[Aydin et al. 2010]
PDM[Winkler 2005]
HDRVDP[Mantiuket al 2005]
DRIVDP[Aydin et al
2008]
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 22: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/22.jpg)
Metric Comparison HDR-LDR
DRIVQM[Aydin et al. 2010]
PDM[Winkler 2005]
HDRVDP[Mantiuket al 2005]
DRIVDP[Aydin et al
2008]
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 23: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/23.jpg)
DRIVQM Online� http://drivqm.mpi-inf.mpg.de
� Uploaded videos stored only during processing
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 24: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/24.jpg)
� Dataset for evaluation of VQMs
– LDR-LDR, HDR-HDR, HDR-LDR
– CG-related artifacts
– Locally-assessed distortions (distortion maps)
� Future work
– More “advanced” CG artifacts
– Real videos (how to get subjective distortion maps?)
Conclusions
January 26, 2011 On Evaluation of Video Quality Metrics: an HDR Dataset for Computer Graphics Applications
Martin Čadík, [email protected] HVEI’11, San Francisco
![Page 25: On Evaluation of Video Quality Metricsresources.mpi-inf.mpg.de/hdr/...presentation.pdf · Simple Metrics MSE = 280 MSE = 280 MSE =280 ! Random Noise Blur ~15% Decreased Luminance](https://reader036.vdocuments.site/reader036/viewer/2022081522/5f038fb57e708231d409a941/html5/thumbnails/25.jpg)
THANK YOU.
<http://www.mpi-inf.mpg.de/resources/hdr/quality>