Image and Video Upscaling from Local Self Examples

Gilad FreedmanRaanan Fattal

Hebrew University of Jerusalem

Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults

Single image upscaling

1. large2. realistic3. faithful4. fast

Previous workparametric image model example based

Freeman et al. 2002

generic looking edges

Sun et al. 2008

Shan et al 2008

Fattal 2007

Glasner et al. 2009

noisyresult

New approach: locale example based

corner step edge

line step edge

non smooth shading

local self similarity small upscaling ratios

1/2

4/5

new non-dyadic filter bank

local self similarity

local search speed

locality too few examples

Increase exemplar quality and size

maintain search locality

novel components:

1. local self similarity2. non-dyadic filter bank

Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults

Local self-examples upscaling

low pass original image high pass

interpolated image

frequency content

low pass high pass

interpolated image

For each patch:

Search a local area for best example

Take corresponding

patch

Add to interpolated

image

frequency content

Local self-examples upscaling

Local self-examples upscaling

low pass high pass

interpolated image

Repeat for all patches, to fill the high frequenciesfrequency content

OverviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults

Local self similarity

cropped downscaled

Local self similarity

Patches in original image can matched locally with ones in downscaled version

Local examples are enough

full imageimage database

query db image local

4.0 2.9 3.55

1.6 1.05 1.05

2.7 2.05 2.05

3.3 2.96 3.06

6.5 5.61 5.61

best matches

Visual assessment – external, exact NN, local

Large externalexample database

Searching theentire image

Searching localregions in image

external database

global search

local search

Comparison of example search methods

Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults

Need for non-dyadic scalingslarge ratios mixed ratiossmall ratios

Dyadic filters

1:2

full frequency

content

higher half

lower half

dyadic filter bank

Non-dyadic filter bank

1:2

4:5

small scaling ratios

better examples

full frequency

content

higher part

lower part

non-dyadic filter bank

Non-dyadic filters: downscaling

1. convolve with 2 filters2. subsample each by 3

dyadic case:

example for the 2:3 ratio:

1. convolve with one filter2. subsample by 2

Non-dyadic filters: upscaling

1. zero upsample by 22. convolve with 2 filters3. sum

dyadic case:

example for the 2:3 ratio:

1. zero upsample by 12. convolve with 1 filter

Use of the filters in upscalingUpscaling using inverse scaling

filters

Smoothing by downscaling

and upscaling

Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults

When interpolating, smooth areas come from inputUniformly spaced grids should remain uniform

1. Uniform stretch

0

255

brig

htne

ss

grid coordinates

2. Consistency

upsample downsample

The interpolated image, if downscaled should be equal to the input. Formally,

Previous methods achieve consistency by solving large linear systems to achieve this property

3. PSF modeling

Large image - small camera point spread

function

Small image - large camera point spread

function

Difference between point spread functions

4. Low frequency spanfr

eque

ncy

When upsampling don’t add new frequencies

Upsampling filter should be low-pass

original interpolated

5. Singularities preservation

≈blurred Image interpolated image

similar amount of blur

Real time video upsampling on GPUmain GPU memory

GPU cores

NTSC to full HD @ 24 fps

Search and filter-banksare both local operations

Background and overviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResults

Bicubic x3(zoomed in)

Ours X3(zoomed in)

Bicubic x3 Ours X3

Genuine Fractals™ x4 Ours X4

Glasner et al. 2009 x4 Ours X4

Thank you!

Paper & additional results can be found at: www.cs.huji.ac.il/~giladfreedmn