j. dong, i. frosio*, j. kautz ifrosio@nvidia · j. dong, i. frosio*, j. kautz [email protected]...
TRANSCRIPT
![Page 1: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/1.jpg)
J. Dong, I. Frosio*, J. Kautz
LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING
![Page 2: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/2.jpg)
2
MOTIVATION
![Page 3: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/3.jpg)
3
NON-ADAPTIVE VS. ADAPTIVE FILTERINGBox-filtering example
Ground truth Noisy, PSNR = 18.61dB
![Page 4: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/4.jpg)
4
NON-ADAPTIVE VS. ADAPTIVE FILTERINGBox-filtering example
Ground truth Noisy + 3x3 box filter, PSNR = 23.47dB
![Page 5: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/5.jpg)
5
NON-ADAPTIVE VS. ADAPTIVE FILTERINGBox-filtering example
Ground truth Noisy + 5x5 box filter, PSNR = 22.14dB
![Page 6: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/6.jpg)
6
NON-ADAPTIVE VS. ADAPTIVE FILTERINGBox-filtering example
Ground truth Noisy + 7x7 box filter, PSNR = 21.12dB
![Page 7: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/7.jpg)
7
NON-ADAPTIVE VS. ADAPTIVE FILTERINGBox-filtering example
Ground truth Noisy + 9x9 box filter, PSNR = 20.62dB
![Page 8: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/8.jpg)
8
NON-ADAPTIVE VS. ADAPTIVE FILTERINGBox-filtering example
Ground truth Noisy + 13x13 box filter, PSNR = 19.81dB
![Page 9: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/9.jpg)
9
NON-ADAPTIVE VS. ADAPTIVE FILTERINGBox-filtering example
Ground truth Noisy + 17x17 box filter, PSNR = 19.28dB
![Page 10: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/10.jpg)
10
NON-ADAPTIVE VS. ADAPTIVE FILTERINGBox-filtering example
Ground truth Noisy + 21x21 box filter, PSNR = 18.90dB
![Page 11: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/11.jpg)
11
NON-ADAPTIVE VS. ADAPTIVE FILTERINGBox-filtering example
Ground truth Noisy + 29x29 box filter, PSNR = 18.41dB
![Page 12: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/12.jpg)
12
NON-ADAPTIVE VS. ADAPTIVE FILTERINGBox-filtering example
Ground truth Best filter size for each pixel
![Page 13: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/13.jpg)
13
NON-ADAPTIVE VS. ADAPTIVE FILTERINGBox-filtering example
Ground truth Adaptive box filter, PSNR = 26.34dB
![Page 14: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/14.jpg)
14
NON-ADAPTIVE VS. ADAPTIVE FILTERINGBox-filtering example
Ground truthAdaptive box filter,
PSNR = 26.34dB
Noisy, PSNR = 18.61dB
Noisy + 3x3 box filter, PSNR = 23.47dB
Noisy + 29x29 box filter, PSNR = 18.41dB
Best filter size for each pixel
![Page 15: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/15.jpg)
15
A LIST OF PROS…
A generalization of an existing filtering technique:
∙ Explainability (vs. pure ML)
∙ Power / computationally efficient (vs. pure ML)
Open question:
∙ Learn parameter tuning (requires ML)
… For adaptive filters
![Page 16: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/16.jpg)
16
SUMMARY
![Page 17: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/17.jpg)
17
Features
SUMMARYLearning Adaptive Parameter Tuning for Image Processing
Adaptive filter
parameters
[1] I. Frosio, K. Egiazarian, K. Pulli, Machine Learning for Adaptive Bilateral Filtering, 2015.
![Page 18: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/18.jpg)
18
SUMMARY
1. Method:
1. Feature design
2. Training
2. Results:
1. NLM denoising
2. Demosaicing
Learning Adaptive Parameter Tuning for Image Processing
[1] I. Frosio, K. Egiazarian, K. Pulli, Machine Learning for Adaptive Bilateral Filtering, 2015.
![Page 19: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/19.jpg)
19
METHOD
![Page 20: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/20.jpg)
20
METHOD: HAND PICKED FEATURES
Computationally efficient features [1]:
Local (3x3, 5x5) variance
Local (3x3, 7x7) entropy
Local (3x3, 7x7) gradient entropy
Local entropy / gradient entropy (noise free) Local entropy / gradient entropy (noisy)
[1] I. Frosio, K. Egiazarian, K. Pulli, Machine Learning for Adaptive Bilateral Filtering, 2015.
![Page 21: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/21.jpg)
21
METHOD: HAND PICKED FEATURES
Computationally efficient features [1]:
Local (3x3, 5x5) variance: edge / texture / noise intensity
Local (3x3, 7x7) entropy: how many sets (homogeneous / textured / edge area)
Local (3x3, 7x7) gradient entropy: how many gradient sets (texture / edge area)
Local entropy / gradient entropy (noise free) Local entropy / gradient entropy (noisy)
[1] I. Frosio, K. Egiazarian, K. Pulli, Machine Learning for Adaptive Bilateral Filtering, 2015.
![Page 22: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/22.jpg)
22
METHOD: TRAININGCost function
Ground truthCorrupt
image
Image
quality
metric Dq
![Page 23: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/23.jpg)
23
METHOD: TRAININGDifferentiable / not differentiable cost function / filter
Ground truthCorrupt
image
RMSE,
PSNR,
MSSSIM,
FSIM, … Dq
N. Ponomarenko et. al, Color Image Database TID2013: Peculiarities and Preliminary Results, 2013.
Can be not differentiable
![Page 24: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/24.jpg)
24
METHOD: TRAININGDifferentiable / not differentiable cost function / filter
Ground truthCorrupt
image
RMSE,
PSNR,
MSSSIM,
FSIM, … Dq
Can be not differentiable
[1] A. Buades, B. Coll, and J. M. Morel, “A review of image denoising algorithms, with a new one”, 2005.
[2] I. Frosio, C. Olivieri, M. Lucchese, N. Borghese, and P. Boccacci, “Bayesian denoising in digital radiography: A comparison in the dental field”, 2013.
![Page 25: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/25.jpg)
25
METHOD: TRAININGDifferentiable / not differentiable cost function / filter
Ground truthCorrupt
image
RMSE,
PSNR,
MSSSIM,
FSIM, … Dq
Can be not differentiable
Nelder-Mead simplex (does not require derivatives)
![Page 26: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/26.jpg)
26
RESULTS
![Page 27: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/27.jpg)
27
RESULTS: NLM DENOISINGNon-adaptive filter
image from opencv.org
p0: patch size [optimal image scale]
p1: affinity measure [bias/variance tradeoff]
[1] A. Buades, B. Coll, and J. M. Morel, A review of image denoising algorithms, with a new one 2005.
![Page 28: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/28.jpg)
28
RESULTS: NLM DENOISING
Large patch size in homogeneous areas
Small patch size for high frequency details
Favor variance reduction in homogeneous areas
Small bias close to the edges, use similarity
Analytical results [1] suggesting a similar strategy
AWGN, s=20 - learned adaptive filter, maximize PSNR
Patch size Filtering par
[1] V. Duval, J.-F. Aujol, and Y. Gousseau,“On the parameter choice for the non-local means, 2011
![Page 29: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/29.jpg)
29
RESULTS: NLM DENOISINGAWGN, s=20 - learned adaptive filter, maximize MS-SSIM
Patch size Filtering par
[1] Z. Zwang, E. Simoncelli, A. Bovik, Multiscale Structural
Similarity for Image Quality Assessement, 2003.
![Page 30: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/30.jpg)
30
RESULTS: NLM DENOISINGPSNR vs. MS-SSIM
![Page 31: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/31.jpg)
31
RESULTS: NLM DENOISINGNumerical results
![Page 32: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/32.jpg)
32
RESULTS: DEMOSAICING
SOA algorithms for demosaicing:
ECC [1]
ARI [2]
CS [3]
Blending outputs:
p0 *ECC + p1*ARI + p2*CS
p0 + p1+ p2 = 1
Blending filters
[1] S. P. Jaiswal, O. C. Au, V. Jakhetiya, Y. Yuan, and H. Yang, “Exploitation of inter-color
correlation for color image demosaicking,” in ICIP, 2014.
[2] Y. Monno, D. Kiku, M. Tanaka, and M. Okutomi, “Adaptive residual interpolation for color
image demosaicking,” in ICIP, 2015.
[3] P. Getreuer, “Image demosaicking with contour stencils,” IPOL, 2012.
![Page 33: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/33.jpg)
33
RESULTS: DEMOSAICINGNumerical evaluation
![Page 34: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/34.jpg)
34
CONCLUSION
![Page 35: J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia · J. Dong, I. Frosio*, J. Kautz ifrosio@nvidia.com LEARNING ADAPTIVE PARAMETER TUNING FOR IMAGE PROCESSING. 2 MOTIVATION. 3 NON-ADAPTIVE](https://reader033.vdocuments.site/reader033/viewer/2022042711/5f6f59ef17e738082f3bb820/html5/thumbnails/35.jpg)
35
CONCLUSION
Learning Adaptive Parameter Tuning for Image Processing
J. Dong, I. Frosio*, J. Kautz
Learnable adaptive filters
- Explainability
- Computational / power effectiveness
- Tunability for different metrics
Possible future directions
- Feature learning
- Reinforcement Learning
- Other applications (beyond image processing)