robust face hallucination using quantization-adaptive ... · robust face hallucination using...
TRANSCRIPT
![Page 1: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/1.jpg)
Robust Face Hallucination using Quantization-Adaptive Dictionaries
Reuben FarrugiaChristine Guillemot
IEEE Int. Conf. on Image Processing, Arizona, USA 26th September 2016
![Page 2: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/2.jpg)
![Page 3: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/3.jpg)
Outline
• Limitations of existing Face Hallucination Methods• Analysis of Compression artefacts on faces• Proposed Robust Face Hallucination Methods• Experimental Results• Robust Face Hallucination in the Wild
![Page 4: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/4.jpg)
Limitations of existing Face Hallucination methods
![Page 5: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/5.jpg)
Face Hallucination
The low-quality face image 𝑿 isdivided into overlapping patchesෝ𝒙1,ෝ𝒙2,…ෝ𝒙𝑷.
These weights 𝒘𝑝are then used to
combine the corresponding high-
quality dictionary 𝑯𝑝 to restore 𝒚𝑝.
The optimal weights 𝒘𝑝 suitable to
reconstruct ෝ𝒙𝑝 are derived using the
low-quality dictionary 𝑳𝑝.
![Page 6: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/6.jpg)
Face Hallucination Dictionary Construction
Let X and Y denote the high- andlow- quality facial images,respectively. Face Hallucinationmethods formulate the acquisitionmodel as
𝑿 =↓ 𝑩𝒀 + 𝜖 where 𝑩 is the blurring function and ↓ is thedownscaling operator.
However, this formulation is valid for uncompressed images and is noteffective when the low-quality image is compressed.
![Page 7: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/7.jpg)
Non-Robust Face Hallucination
Comparing the original high quality face image (left), the downscaled and compressed low-quality face image (centre) and the super-resolved image using a state-of-the-art face hallucination method (right). The original image is down-scaled such that they have inter-eye distance of 10 pixels and compressed using H.264/AVC Intra with a quantization parameter of 20.
![Page 8: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/8.jpg)
Analysis of Compression artefacts on Faces
![Page 9: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/9.jpg)
Compression Artefacts on FacesH
.26
4/A
VC
Co
de
c d
x =
20
H.2
64
/AV
C C
od
ec d
x =
10
QP = 10 QP = 35 QP = 10 QP = 35
![Page 10: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/10.jpg)
Compression Artefacts on FacesJP
EG C
od
ec
dx
= 2
0
JPEG
Co
dec
dx
= 1
0
Quality = 70 Quality = 30 Quality = 70 Quality = 50
![Page 11: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/11.jpg)
Proposed Robust Face Hallucination Method
![Page 12: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/12.jpg)
Robust Face Hallucination Dictionary Const.
The acquisition model considered in this work to construct thecoupled dictionaries includes the codec following the blurring anddownscaling operators and can be formulated using𝑿 = 𝑻−1𝑄 𝑻 ↓ 𝑩𝒀 + 𝜖 where T is the transform and Q is thequantization function.
![Page 13: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/13.jpg)
Robust Position Patch (RPP)
The Robust Position Patch derives the combination weights byminimising the following optimization
𝑤𝑝 = 𝑎𝑟𝑔𝑚𝑖𝑛𝑤𝑝ො𝑥𝑝 − 𝐿𝑝𝑤𝑝 2
𝑠. 𝑡. 𝑤𝑝 2= 1
![Page 14: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/14.jpg)
Robust Position Patch (RPP)
The Robust Position Patch derives the combination weights byminimising the following optimization
𝑤𝑝 = 𝑎𝑟𝑔𝑚𝑖𝑛𝑤𝑝ො𝑥𝑝 − 𝐿𝑝𝑤𝑝 2
𝑠. 𝑡. 𝑤𝑝 2= 1
![Page 15: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/15.jpg)
Robust Sparse Position Patch (RSPP)
The Robust Sparse Position Patch derives the combination weights byminimising the following optimization
𝑤𝑝 = 𝑎𝑟𝑔𝑚𝑖𝑛𝑤𝑝𝑤𝑝 1
𝑠. 𝑡. ො𝑥𝑝 − 𝐿𝑝𝑤𝑝 2< 𝜖
![Page 16: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/16.jpg)
Experimental Results
![Page 17: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/17.jpg)
PSNR Analysis
20 25 30
Baseline 10 27.59 27.33 26.63
PP 10 25.74 25.73 25.53
RPP 10 30.19 29.73 28.65
SPP 10 25.52 25.43 25.20
RSPP 10 30.12 29.54 28.51
LINE 10 25.32 25.22 25.01
RLINE 10 29.95 29.39 28.30
Baseline 20 29.63 29.46 29.03
PP 20 34.13 33.29 31.70
RPP 20 34.11 33.36 32.05
SPP 20 33.91 32.89 31.22
RSPP 20 34.01 33.21 31.88
LINE 20 33.90 32.88 31.25
RLINE 20 33.94 33.09 31.64
Quantization ParameterMethod dx
H.264/AVC Intra Codec
50 60 75
Baseline 10 26.78 27.02 27.48
PP 10 27.36 27.71 28.50
RPP 10 28.34 28.58 29.18
SPP 10 26.82 27.16 27.83
RSPP 10 28.13 28.40 28.96
LINE 10 26.67 27.02 27.74
RLINE 10 27.99 28.23 28.83
Baseline 20 30.05 30.30 30.75
PP 20 31.07 31.39 32.23
RPP 20 31.83 32.09 32.75
SPP 20 30.54 30.86 31.59
RSPP 20 31.62 31.88 32.47
LINE 20 30.52 30.84 31.65
RLINE 20 31.51 31.81 32.47
Quality ParameterMethod dx
JPEG Codec
![Page 18: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/18.jpg)
Recognition Analysis
20 25 30
Baseline 10 0.538 0.498 0.354
PP 10 0.416 0.377 0.281
RPP 10 0.553 0.486 0.318
SPP 10 0.469 0.399 0.304
RSPP 10 0.619 0.561 0.385
LINE 10 0.485 0.424 0.293
RLINE 10 0.607 0.556 0.394
Baseline 20 0.722 0.693 0.625
PP 20 0.748 0.714 0.640
RPP 20 0.745 0.700 0.623
SPP 20 0.780 0.732 0.667
RSPP 20 0.761 0.723 0.634
LINE 20 0.779 0.744 0.681
RLINE 20 0.770 0.724 0.654
Quantization ParameterMethod dx
H.264/AVC Intra Codec
50 60 75
Baseline 10 0.362 0.380 0.450
PP 10 0.424 0.433 0.517
RPP 10 0.382 0.436 0.479
SPP 10 0.415 0.436 0.525
RSPP 10 0.428 0.461 0.526
LINE 10 0.444 0.481 0.532
RLINE 10 0.417 0.455 0.523
Baseline 20 0.663 0.685 0.699
PP 20 0.673 0.689 0.719
RPP 20 0.668 0.672 0.711
SPP 20 0.675 0.707 0.731
RSPP 20 0.687 0.705 0.727
LINE 20 0.683 0.719 0.726
RLINE 20 0.679 0.698 0.726
Quality ParameterMethod dx
JPEG Codec
![Page 19: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/19.jpg)
Subjective Evaluation
Comparing the original high quality face image (first column), the downscaled and compressed low-quality face image(second column), the super-resolved image using a state-of-the-art face hallucination method (third column) and thesuper-resolved face image using the proposed robust method, where the LM-CSS face SR method is used. The originalimage is down-scaled such that dx = 10 and compressed using H.264/AVC Intra with a quantization parameter of 20.
![Page 20: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/20.jpg)
Robust Face Hallucination in the Wild
![Page 21: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/21.jpg)
Robust Face Hallucination in the WildTraining Images
Low-quality image
Upscale
Warping
HallucinationDegradation
![Page 22: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/22.jpg)
Robust Face Hallucination in the Wild
HQ Image LQ Image RLM-CSS HQ Image LQ Image RLM-CSS
![Page 23: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/23.jpg)
Conclusion
• This work presents a new dictionary construction methodwhich employs the syntax available in the image file tomodel the distortions affecting the low-quality image.
• Experimental Results show a significant gain in terms ofquality with respect to the non-robust hallucinationmethods.
• The proposed method can be extended to super-resolvefacial images in the wild.
![Page 24: Robust Face Hallucination using Quantization-Adaptive ... · Robust Face Hallucination using Quantization-Adaptive Dictionaries Reuben Farrugia Christine Guillemot IEEE Int. Conf](https://reader031.vdocuments.site/reader031/viewer/2022021900/5b6564d57f8b9a6e1f8bb0a8/html5/thumbnails/24.jpg)
Thank You
Reuben Farrugiahttps://www.um.edu.mt/staff/[email protected]
More information about this project can be found onhttps://www.um.edu.mt/staff/reuben.farrugia/projects/face_hallucination