super resolution для сжатого видео
DESCRIPTION
TRANSCRIPT
![Page 1: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/1.jpg)
Super Resolution для сжатого видео
Моисейцев Алексей
Video GroupCS MSU Graphics & Media Lab
![Page 2: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/2.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
222
Содержание
Введение
Example-based SR
SRME
HMRF SR
![Page 3: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/3.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Введение
3
![Page 4: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/4.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Введение
4
source
Bi-cubic
Super-res
Lossless 2Mbps 1Mbps 256Kbps
Super-Resolving Compressed Video with Large Artifacts, Wen-Yi Zhao, ICPR 2004
![Page 5: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/5.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Введение
Алгоритмы SR:
Iterative Backprojection (IBP)
Projection Onto Convex Sets (POCS)
Probabilistic Methods
Maximum a posteriori (MAP)
Model-based approach (MBSR)
Example-based
6
![Page 6: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/6.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Введение
IBP
7
C2
CC1
C3
g
P1g
P3g
P2g
f=Pg
POCS
![Page 7: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/7.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Введение
8Super resolution: an overview, C Papathanassiou and M Petrou, Geoscience and Remote Sensing Symposium, 2005
![Page 8: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/8.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Введение
Типичная скорость работы: 0.15-0.4 fps
9
![Page 9: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/9.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
101010
Содержание
Введение
Example-based SR
SRME
HMRF SR
![Page 10: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/10.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Example-based SR
В SR только по LR-кадрам существует теоретический предел качества
Иногда есть доступ и к отдельным кадрам в высоком разрешении
11Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
![Page 11: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/11.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Example-based SR
Registration
Для компенсации движения используется алгоритм Lucas-Kanade:
12Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
![Page 12: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/12.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Example-based SR
Registration
13
В 99% случаев ошибка не превышает 0.15 LR-пикселя и 0.45 HR-пикселя
Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
![Page 13: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/13.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Example-based SR
Построение пар HR-LR
14
Используются LR-блоки 8x8
Рассматриваются повороты блоков
Ключ —10 AC-коэффициентов LR-блока и 68 граничных пикселей HR-блока
Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
![Page 14: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/14.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Example-based SR
Восстановление
15Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
![Page 15: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/15.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Example-based SR
Восстановление
16Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
![Page 16: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/16.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Example-based SR
Результат
MJPEG, Q=50
Обучение на первом кадре
Восстановление двадцатого кадра
17
source, frame 0 compressed frame 20 SR, frame 20
Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
![Page 17: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/17.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Example-based SR
Результат
18
compressed frame 20 SR, frame 20 Spatial example-based SR, frame 20
Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
![Page 18: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/18.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Key-frame based SR
19Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008
Идея: использовать HR ключевые кадры
![Page 19: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/19.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Key-frame based SR
Восстановление
20Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008
![Page 20: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/20.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Key-frame based SR
Результаты
21Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008
![Page 21: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/21.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Key-frame based SR
Результаты
22Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008
![Page 22: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/22.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus Key-frame based SR
Результаты
23Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008
![Page 23: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/23.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Example-based SR
Слабая обоснованность метода
Требуется наличие специфичного видеопотока
Зависимость от обучающей выборки
24
![Page 24: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/24.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
252525
Содержание
Введение
Example-based SR
SRME
HMRF SR
![Page 25: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/25.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
26
SRME
Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
![Page 26: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/26.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
SRME
27Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
![Page 27: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/27.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
SRME
28
— интересующее
решениеSimultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
— апостериорная
вероятность
![Page 28: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/28.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus SRME
Задание вероятностей (1)
29Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
![Page 29: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/29.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus SRME
Задание вероятностей (2)
30Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
![Page 30: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/30.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus SRME
Задание вероятностей (3)
31Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
![Page 31: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/31.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus SRME
Решение (1)
32Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
![Page 32: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/32.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus SRME
Решение (2)
Две итерации:
Компенсация движения
Построение улучшенного кадра
33Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
![Page 33: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/33.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
SRME
34Y-PSNR = 20.44dB
Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
source
![Page 34: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/34.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
SRME
35Y-PSNR = 21.98dB
Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
bilinear
![Page 35: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/35.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
SRME
36Y-PSNR = 25.64dB
Simultaneous motion estimation and resolution enhancement of compressed low resolution video.Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
proposed SR
![Page 36: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/36.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
373737
Содержание
Введение
Example-based SR
SRME
HMRF SR
![Page 37: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/37.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
HMRF SR
38Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
Standard decompression Smoothed image Sharpened image
![Page 38: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/38.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
HMRF SR
39Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
![Page 39: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/39.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Шум квантования (1)
40Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
![Page 40: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/40.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Шум квантования (2)
41Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
![Page 41: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/41.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Шум компенсации (1)
42Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
![Page 42: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/42.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Шум компенсации (2)
43
— общее выражение
— независимость шума
Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
![Page 43: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/43.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Условие гладкости
44Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
![Page 44: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/44.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
HMRF SR
45
— гладкость
— квантование
— компенсация
Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
![Page 45: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/45.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Решение
46Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
Градиент:
![Page 46: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/46.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Решение
Упрощение:
Градиентный спуск:
47Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
![Page 47: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/47.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Результаты
48Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
![Page 48: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/48.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Результаты
49Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
![Page 49: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/49.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus HMRF SR
Результаты
50Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
![Page 50: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/50.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
Заключение
Рассмотрены Example-based и MAP методы
Многие методы используют информацию из потока или требуют дополнительные данные для работы
Прямые реализации алгоритмов сложны, но тем не менее многие из них возможно распараллелить (например, с использованием CUDA)
51
![Page 51: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/51.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
52
Литература
Simultaneous motion estimation and resolution enhancement of compressed low resolution video. Javier Mateos, A. K. Katsaggelos, Rafael Molina, ICIP, 2000
Restoration of Compressed Video using Temporal Information, Mark A. Robertson and Robert L. Stevenson, SPIE 2001
DCT Quantization Noise in Compressed Images, Mark A. Robertson and Robert L. Stevenson, 2004
Super-Resolving Compressed Video with Large Artifacts, Wen-Yi Zhao, ICPR 2004
Z. Lin and H-Y. Shum. Fundamental limits of reconstruction-based superresolution algorithms under local translation. PAMI, 2004
Resolution enhancement of low quality videos using a high-resolution frame, Tuan Phama, Lucas van Vlieta, Klamer Schutte, SPIE 2006
T.Q. Pham, M. Bezuijen, L.J. van Vliet, K. Schutte, and C.L. Luengo Hendriks. Performance of optimal registration estimators, SPIE 2005
Super resolution: an overview, C Papathanassiou and M Petrou, Geoscience and Remote Sensing Symposium, 2005
Super Resolution of Video Using Key Frames, Fernanda Brandi, Ricardo de Queiroz, Debargha Mukherjee, ISCAS 2008
![Page 52: Super Resolution для сжатого видео](https://reader030.vdocuments.site/reader030/viewer/2022020720/54bea1784a79590c528b4864/html5/thumbnails/52.jpg)
CS MSU Graphics & Media Lab (Video Group)
Only for Maxus
5353
Вопросы
?
53