low-complexity single-image super-resolution based … · 2018-11-24 · british machine vision...

British Machine Vision Conference (BMVC) 2012 September 6, 2012

LOW-COMPLEXITY SINGLE-IMAGE SUPER-RESOLUTIONBASED ON NONNEGATIVE NEIGHBOR EMBEDDING

Marco Bevilacqua1,2, Aline Roumy1, Christine Guillemot1,Marie-Line Alberi Morel2

1Inria Rennes - Bretagne Atlantique 2Alcatel-Lucent - Bell Labs France

Overview

I Single-image Super-ResolutionI What is it?I Methods usedI Neighbor embedding SR

I Proposed algorithmI [KP1] Feature representationI [KP2] Neighbor embedding methodI [KP3] Choice of the dictionary

I Results

I Conclusions

BMVC 2012 - Low-Complexity Single-Image SR September 6, 2012 - 2

Single-image Super-Resolution

What is it?Given a low resolution (LR) input image, the aim is to produce anenhanced upscaling (high resolution - HR).

Our targetI Design a low-complexity yet efficient single-image SR

algorithm


Methods

Single-image SR

I Inverse problem methods

I Machine Learning methodsI Example-based methods ( ⇒ correspondences of LR/HR

“patches” )I Direct local Learning (Support Vector Regression, Ridge

Regression...)I Nearest Neighbor (NN) estimation


Nearest Neighbor SR1. Dictionary learning Learning correspondences of LR/HR patches

I training images ⇒ external dictionary[Freeman et al., 2002, Yang et al., 2008, Chang et al., 2004]

I correspondences learnt exploiting image self-similarities[Glasner et al., 2009]

2. NN search Search for best matching patches for the LR inputpatches

3. Weight computation Compute the weights of the patchcombinations by using the selected LR candidates

4. HR patch reconstruction Combine the corresponding HRcandidates to reconstruct the HR output patches, according tothe weights computed

I The whole procedure is carried out in a feature spaceI One-pass or multi-pass approach


LLE-based Nearest Neighbor SR

In [Chang et al., 2004] an example-based SR algorithm, inspired byLLE (Locally Linear Embedding), is proposed:

I The weights for the linear combination (3. Weightcomputation), like in LLE, are the results of a LS problemthat minimizes the approximation error with a sum-to-1constraint (SUM1-LS)

wi = arg minw‖xi

t − X idw‖2 s.t. 1T w = 1 . (1)

I Gradient features are used for the LR patches;mean-subtracted luma values are used as features for the HRpatches.


Key points

Three key points

KP1 Features used to represent the LR and HR patches

KP2 Method used to compute the neighbor embedding (i.e. theweights of the patch combination)

KP3 Nature of the dictionary: external or “internal”?


[KP1] Representation by featuresEach patch is represented by a feature vector

Role of the featuresI To catch the most salient information in the LR patches in order to

predict the HR detailsI To enforce the hypothesis of manifold similarity

Various possibilitiesI Simple luminance valuesI Centered luminance values (with mean removal)I Gradient valuesI . . .


[KP1] Analysis of the featuresF1 1st order gradientF2 Centered luminance values

F1+F2 Concatenation of F1 and F2

I All curves present a fall (even dramatic in case of F2)I We decide to use F2: ”low-cost” and best performing overall → Can

we avoid the fall?


[KP1] Why the fall?Observation 1d : dimension of the LR vectors; K : number of neighbors⇒ The ith neighborhood matrix X i

d has the highest possible rank w.h.p., i.e.ri = min(d − 1,K ).

Observation 2For K = d the SUM1-LS problem is equivalent to a square linear system, as wehave d equations in K = d unknowns ⇒ unique solution in the LR domain.Here, experimentally we have a “critical point” in the performance.⇒ The fall is because of an overfitting problem!


[KP2] A nonnegative embedding?

I Idea: replace the sum-to-1 equality constraint by an inequalityconstraint to avoid the unique solution problem

I ⇒ Patches reconstructed only by additive combinationsaccording to the “intuitive notion of combining parts to forma whole”

I The LS problem (1) becomes a nonnegative least squares(NNLS) problem:

wi = arg minw‖xi

t − X idw‖2 s.t. w ≥ 0 . (2)


[KP2] Analysis of the weightsI Distribution of the weights

SUM1-LS NNLSI Distance between the actual LR weights and the “ideal” HR weights


[KP3] Choice of the dictionary

Two possibilitiesI build an external dictionary from a set of training images (from the

original HR images, generate the LR versions, and extract HR and LRpatches, respectively)

I learn the patch correspondences in a pyramid of recursively scaled images,starting from the LR input image, in the way of [Glasner et al., 2009]

Internal DB Ext DB “esa” Ext DB “wiki”Image Scale PSNR DB size PSNR DB size PSNR DB size

Head 4 28.62 525 30.22 56514 30.24 218466Baby 4 27.95 2179 30.62 56514 30.32 218466Eyetest 3 16.10 3827 18.38 100966 18.10 389232bird 3 27.72 2256 31.37 100966 31.42 389232Woman 2 27.29 15044 30.91 229096 30.44 880440

I In our case the dictionary derived from the pyramid is insufficient.


Algorithm: summaryKP1 Features used to represent the LR and HR patches

I Centered luminance features (F2)

KP2 Method used to compute the neighbor embedding (i.e. theweights of the patch combination)

I Nonnegative embedding (NNLS weights)

KP3 Nature of the dictionary: external or “internal”?I External


Experiments: algorithms considered

Name Procedure LR features Patch reconstructionmethod

Chang et al.(LLE)

single-step gradient (1st-2nd) NN embedding withSUM1-LS weights

Glasner et al.(Pyramid)

multi-pass luminance NN embedding with ex-ponential weights

Tang et al.(KRR)

single-step gradient (1st-2nd) kernel ridge regression

Our algo-rithm

single-step centered luminance NN embedding withNNLS weights


Results 1/3Our algorithm Chang et al. Glasner et al. Tang et al.

Image Scale PSNR Time PSNR Time PSNR Time PSNR Time

baby 2 34.64 58 33.42 339 34.66 4083 33.72 425bird 2 34.69 18 32.94 110 34.42 406 33.31 132butterfly 2 27.54 17 25.90 77 26.83 265 26.05 82head 2 32.88 18 32.34 145 32.68 367 32.43 151woman 2 30.91 15 29.43 114 30.61 410 29.64 128



I Higher PSNR for scale = 2; Glasner et al. outperforming for larger scalefactors.


Results 2/3Our algorithm Chang et al. Glasner et al. Tang et al.

Image Scale PSNR Time PSNR Time PSNR Time PSNR Time




I Computational time sensibly reduced, thanks to 1) one-pass procedure or2) shorter feature vectors.


Results 3/3

Our algorithm Chang et al. Glasner et al. Tang et al. Lin. Regr. + F2Image Scale PSNR Time PSNR Time PSNR Time PSNR Time PSNR Time

baby 2 34.64 58 33.42 339 34.66 4083 33.72 425 34.76 30bird 2 34.69 18 32.94 110 34.42 406 33.31 132 34.91 9butterfly 2 27.54 17 25.90 77 26.83 265 26.05 82 27.66 7head 2 32.88 18 32.34 145 32.68 367 32.43 151 32.88 8woman 2 30.91 15 29.43 114 30.61 410 29.64 128 31.01 8



I Linear Ridge Regression + centered luminance features: promisingmethod!


Visual results: bird 1/6

M. factor: x3

Input image



M. factor: x3

Bicubic interpolation



PSNR: 29.71 dBTime: 47 s.

Chang et al.




Glasner et al.




Tang et al.




Our algorithm


Visual results: baby 1/6

M. factor: x3

Input image



M. factor: x3

Bicubic interpolation




Chang et al.




Glasner et al.




Tang et al.




Our algorithm


Conclusions

I Our method present better results than other one-passalgorithms [Chang et al., 2004, Tang et al., 2011]

I Results comparable to the multi-pass algorithm of [Glasner etal., 2009] for scale factor of 2 and 3, but much lowercomputational time

Future workI Further study regression methods (already promising results)I Investigate other strategies for neighbor search


THANKSQuestions?

BMVC 2012University of Surrey

September 3-7, 2012

low-complexity single-image super-resolution based … · 2018-11-24 · british machine vision...

Documents