pragmatic approaches to image resampling

Pragmatic approaches to image resampling

Tom McGlynnNASA/GSFC

Tom McGlynn IPAM WorkshopJan 27, 2004

Outline

Why do we resample? Isn’t resampling a solved problem? Some approaches to resampling.

A new algorithm for exact-area resampling using clipping.

How should astronomers choose resampling algorithms?


Why do we need to resample? Display – transform image into ‘standard’ form

Undo warps and distortions Transform to standard frame Resizing

Magnification and minification Rotation

Image comparison – transform one image to match another Similar operations as in display, but final frame may not be ‘standard’

Mosaicking Building sky region and all sky images

Image arithmetic Dither additions, image differencing, speckle analysis

Can have most extreme requirements on accuracy of reconstruction but often involves very similar images.

Different problems levy different requirements for accuracy and robustness. Many resampling problems involve substantial changes in geometry.


Resampling examples

SkyView transforms the EGRET all sky map in Galactic coordinates to Equatorial coordinates.


WMAP data must be transformed from HEALpix formats to human friendly ones.


On Resampling of Solar ImagesSolar Physics, 2003,C.E. DeForest

Transforming SOHO/EIT images to Cartesian coordinates. And that’s before handling differential rotation…


SkyView dynamically mosaics surveys data for display or image comparison


SkyMorph image addition.

Adding NEAT images for asteroid discovery using SkyMorph. Other survey datasets can also be searched for pre-discovery

Three co-added NEAT images taken at 20 minute intervals.

(1999 discovery images for asteroid 9460)

DSS image (1950)


Classical image resampling

1. Image reconstructionThe continuous image is regenerated by some

method, often by interpolating between the grid point by using an interpolation kernel.

2. SamplingThe new grid is constructed by sampling the

reconstructed function appropriately.

3. Filters and transformationsIntermediate filters and transformations may be

applied at various points in the process.


Ideal Reconstruction

A band-limited function can be exactly reconstructed by convolution with the appropriate kernel.

Shannon says…By hypothesis f…

…has a Fourier transform, F bandlimited to 1/2T.

-1/(2T) 1/(2T)Pixelate: Multiply by Comb function) with separation of T…

… means convolve the Fourier transform with the Fourier Transform of Comb, but this is just another Comb function with separation of 1/T which just relplicates the Fourier Transform.

5T

So mutiplying by a boxcar gets back original F …

… but that’s equivalent to convolving in space with Fourier transform of boxcar, sinc(). So we just need to use sinc interpolation to get back original image.


Solved problem?

So sinc function is the optimal filter. Just reconstruct the image using sinc interpolation and resample as needed!


But… real images are finite

Chips typically have 1-4K pixels with larger detectors using arrays of chips.

Edge effects need to be considered … Copying image, image reversal, constant

value, 0 beyond edge How do we handle missing pixels? Typical CD song has ~100K samples

before we begin hearing the music.


…Is the image well sampled?

Is sampling rate adequate? How do we tell?

What can cause high frequencies? Noise and photon statistics Features Processing

Filtering and compression – flat can mean high frequencies.

The sky 1 Gigapixel Nyquist-sampled image for

0.001” resolution covers only 16”. Do users prefilter images?


…The sky isn’t a plane Proper basis for sphere is spherical

harmonics. CMB studies use HEALPix pixels to make this

easier. Are HTM grids going to be used?

Derivation of sinc function kernel assumes rectangular grid. Other functions are appropriate for other samplings.

What are the effects of ignoring curvature?


…Point versus Area resampling Pixels have finite extent. In 2-d case the relationship between the

original and resampling pixels can be complex. 1-D and 2-D resampling share ‘calculus’ but 2-D

geometry is more complex. If input and output pixels are similar we can

presume the function being reconstructed is the flux convolved with the pixel mask.

How do we handle resampling pixels with complex shapes?


What makes resampling hard? Calculus

Undersampling Noise, features, missing pixels Constraints on the output range (avoidance of

negative values, integer valued functions) Geometry

Differing projections and coordinate systems and orientations.

Scale changes Non rectangular pixels. Variations over the image


Alternatives Image regeneration

Is the image a derived product? Pixels added? Models?

Retake the image Adapt requirements to minimize

projection issues Often not feasible or desirable.


Some reconstruction kernels

The background image is reconstructed by convolution with the given kernel.

The sinc function is the ‘optimal’ reconstruction kernel for a well sampled image.


Nearest Neighbor

Universally panned… Bad geometric distortions Suspect photometry Poor for magnification

but even the least elegant techniques may have their place.

Fast Histogram of values can be preserved. Sometimes exactly invertible. Integers stay integers

Classifications (does vegetation+water=city) Photon counting algorithms

Retention of discontinuities Man-made boundaries

Constellation maps


Bilinear Interpolation Fast Better at preserving astrometry and

photometryBut Smooths and blurs the image


‘Higher order’ interpolation Best job reproducing well sampled images Limits blurringBut… Sometimes slower Can be sensitive to ‘features’ or undersampling.

- negative values when resampling. Harder to handle missing pixels Gazillions of choices

Interpolating or approximating Polynomial approximations

Splines Local support polynomials

Truncated sinc Windowed sinc: Lanczos, Hamming, … Image support for calculation of kernel Image support for resampling calculations

Very high order methods (e.g., exact sinc) can be very slow


More choices: Interpolation versus Redistribution Interpolation

Output flux at a pixel is computed as a sum of weighted input pixels using reconstruction kernel.

Tends to assure differential properties of image (continuity, derivatives)

Redistribution (Drizzle, exact area sampling) Input pixel’s flux is distributed over output pixels

using redistribution kernel. Can ensure integral properties of image (flux or flat

field). Easy to handle missing pixels or other discontinuities Global integrals can be conserved to the limits of

arithmetic precision


Exact Area Resampling

Calculate the resampling pixels as the weighted averages of the pixels they cover (or weighted sums for extensive data) weighting


Approaches Delaunay Triangulation

Using suitable collection of points one finds a set of triangles where each triangle belongs to just one input and output pixel.

Hideously slow… Girard’s theorem (Montage: Berriman and Good)

Work on the celestial sphere and use Girard’s theorem to calculate area – a `practical’ example of using parallel transport of vectors.

Very, very slow. Clipping

How much of the resampling pixels can you view through the window of the input pixels?

Comparable speed to other high-order methods. Equivalent to algorithm used within Drizzle?


Exact area resampling using clipping.

For each resampling pixel…

First find all the pixels that may overlap the resampling pixel, by looking at the range of the resampling pixel corners. Now for each candidate original pixel…


Clip the resampling pixel on each edge.

Clip the resampling pixel by each of four clipping boundaries.

(Sutherland-Hodgman algorithm but the extensive clipping literature suggests more efficient approaches, e.g., Liang-Barsky).

4

1

3

2


Clipping by a infinite line

v1

v3

v2

v4

c2

c1

Inside Outside

For i = 1 to n

if vi-1 is outside

if vi is inside

emit crossing point (e.g., c2) and vi

else if vi is outside

emit crossing point (e.g., c1)

else

emit vi

Input (v1,v2,v3,v4)

Output

(v1,c1,c2,v3,v4)


Triangulating the overlap

A

E

D C

B

The area of the overlap polygon ABCDE is easily computed as the sum of the triangles ABC, ACD, ADE


Normalization

Intensive images Track total flux and overlapped area for

each pixel and use ratio for pixel value. Preserves flat fielding

Extensive images Just add flux. Preserves total flux


Features Convexity of pixels allows simplification of algorithm (since

convex regions clip convex regions to convex regions). Clipping on rectangular grid is especially easy but either grid

could be triangular or hexagonal or even discontiguous. Don’t have to make clipping window the same as pixel

boundaries. If clipping window is smaller we get Drizzle-like algorithm If clipping window is larger than pixels we have a variable box-car

along with resampling. Easy to accommodate ‘bad’ pixels or regions. Symmetry between resampling and original grid

Can resample in the `convenient’ direction if transformation is easier one way or the other.

Not the most ‘accurate’ algorithm, but it can be extremely robust. “Best worst-case resampler”


Possible clipping ‘kernels’

Drizzle redistribution

Exact area sampling

Boxcar smoothing with resampling

Step pyramid kernel with different fraction of flux in each box

Adjusting the size of the rectangle changes how the algorithm considers the flux in the pixel to be distributed.


How can astronomers decide which resampling algorithm to use? Accuracy

Does resampling affect astronomical measurements? Detection, Astrometry, Photometry, Morphology,

Resolution Cost (CPU, memory) Complexity

Coding and comprehension costs Robustness

Point versus area resampling How well does it work when in hard resampling

situations?


Users

Issues:UndersamplingPixel distortions

Spherical geometryEdges

PrefilteringBad pixel regions

Compression…

Goals: Mosaicking, Resizing,Displaying, Subtracting, Dithering,

Comparing, Undistorting, Projecting

Methods:Nearest neighbor

Linear interpolationSplines

Polynomial kernelsGaussian

SincHammingLanczos

Exact area…

Traits:Accuracy

SpeedComprehensibility

AvailabilityRobustness

…

Need for a roadmap


Analysis in other regimes

Quantitative Comparison of Sinc-Approximating Kernels for Medical Image InterpolationErik H. W. Meijering, Wiro J. Niessen, Josien P. W. Pluim, Max A. Viergever

Round-robin resampling – a sequence of resamplings leading back to the original image. Good for determining the ‘best’, but not for assessing the cost

• Truncated sinc resamplers are the worst.• Variety of higher order resamplers do pretty well.• But the images don’t look much like typical astronomical data)


…Image Reconstruction by Convolution withSymmetrical Piecewise nth-OrderPolynomial Kernels

Erik H. W. Meijering, Karel J. Zuiderveld, Max A. ViergeverIEEE Transactions on Image Processing, vol. 8, no. 2, February 1999, pp. 192, 201.

•Going beyond third order kernels doesn’t seem to buy one anything.

Comparing image resamplers via a model of the human vision system

Richard Harvey, Stephen King, Richard Aldridgeand J. Andrew Banghamh

A big driver in commercial applications is rendering of text.


SWarp v2:0 User's guideE. Bertin

NN LI Lcz2 Lcz3 Lcz4

Moire patterns in linearly interpolated resampled images

Features in Lanczos resampling.

Even astronomical references often compare samplers in qualitative terms.


Measuring accuracy quantitatively Single resampling Astronomicalish images How does noise affect resampling? Two resampling scenarios:

Small Pixel: Rotation, final pixel size/original pixel size=1.1 Big Pixel: Rotation, final pixel size/original pixel size=2.5

Use Sextractor to estimate parameters of original and resampled objects.

Model image with 100 gaussian objects and variable noise

Use DSS image for reality check

Test Images 0: 0 (Number: Noise) 5: 0.006

15: 0.2

10: 0.03

20: 1.0


North Pole image

DSS image


Samplers Tested Interpolation algorithms

NN – Nearest Neighbor LI – Linear Interpolation LCZn – A Lanczos n-lobe interpolator (n=3,4) SPn – An n’th order spline (n=3,4)

Redistribution algorithms CL – Clipping exact area CL0.5 – Clipping using a window of half the size

of the pixel (similar to Drizzle) MN –Montage exact area algorithm, resampling

done on celestial sphere.


Small Pixel Detection

Did we detect all of the objects detected in the unresampled data?

Boxes give the results for the real image looking at the first six samplers listed. The dashed line gives the measurement in the original unresampled image.

0

20

40

60

80

100

120

140

160

180

200

0 5 10 15 20 25

Sample

Sou

rces

Det

ecte

dCL

CL0.5

LCZ3

LI

MN

NN

Orig

Real

Lcz4

Sp3

Sp4


Small Pixel Astrometry

What was the average offset of the resampled image from the measurement in the original image?

0.0001

0.001

0.01

0.1

1

0 5 10 15 20 25

Noise Sample

Off

set

in p

ixel

s

CLCL0.5LCZ3LIMNNNOrigRealLcz4Sp3Sp4


Small Pixel Photometry

How much did the flux change when we resampled?

0.0001

0.001

0.01

0.1

1

0 5 10 15 20 25

Noise Sample

Off

set

in p

ixel

s

CLCL0.5LCZ3LIMNNNOrigRealLcz4Sp3Sp4


Small Pixel Morphology

The image modeled circular gaussians. What is the average axis ratio (a-b)/a measured in the resampled data?

Note that graphs compare with model and not measured values. 0.001

0.01

0.1

1

0 5 10 15 20 25

Noise Sample

No

n-c

ircu

lari

ty (

a-b

)/a

CL

CL0.5

LCZ3

LI

MN

NN

Orig

Real

Lcz4

Sp3

Sp4


Small Pixel Blurring

Is the resampled image blurred by the resampling? How much larger is the it?

-0.01

-0.008

-0.006

-0.004

-0.002

0

0.002

0.004

0.006

0 5 10 15 20 25

Noise Sample

Blu

rrin

g (1

pix

el =

0.0

1)

CL

CL0.5

LCZ3

LI

MN

NN

Orig

Real

Lcz4

Sp3

Sp4


Big Pixel Detections

0

20

40

60

80

100

120

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Noise Sample

Nu

mb

er o

f o

bje

cts

fou

nd

CL

CL0.5

LCZ3

LCZ4

LI

NN

SP3

SP4

Orig


Big Pixel Astrometry

0.0001

0.001

0.01

0.1

1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Noise Sample

Ch

ang

e in

flu

x

CL

CL0.5

LCZ3

LCZ4

LI

NN

SP3

SP4

Orig


Big Pixel Photometry

0.0001

0.001

0.01

0.1

1

10

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Noise Sample

Ch

ang

e in

flu

x

CL

CL0.5

LCZ3

LCZ4

LI

NN

SP3

SP4

Orig


Big Pixel Morphology

0.001

0.01

0.1

1

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Noise Sample

No

n-c

ircu

lari

ty

CL

CL0.5

LCZ3

LCZ4

LI

NN

SP3

SP4

Orig


Big Pixel Blurring

-0.01

-0.005

0

0.005

0.01

0.015

0.02

0.025

0.03

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Noise Sample

Blu

rrin

g (

1 o

utp

ut

pix

el =

.02

5) CL

CL0.5

LCZ3

LCZ4

LI

NN

SP3

SP4

Orig


Timing

Time to resample a 500x500 sample in a 700x700 grid.(Times are for complete process including I/O and coordinate transformations. 600 MHz processor)

Java C

1.0

10.0

100.0

1000.0


Complexity Low order methods are trivial to program and relatively

easy to test. Some higher order kernel interpolation methods are only

modestly more complex. The initial computation of splines is more challenging, but splines are very easy to evaluate

Previous exact area sampling techniques were complex and time prohibitive. Redistribution methods based on sample kernels can be comparable to interpolation methods. Clipping algorithm is relatively straightforward to understand but use a more algorithmic and less functional paradigm.


Robustness Discontinuities, features, holes are harder to handle directly in higher

order methods Prefiltering or image patching.

Supersampling (averaging multiple sample points within a single resampling pixel) is an easy step towards area resampling for point resampling methods, but requires understanding how many samples should be made within each pixel.

Magnification of well-sampled images best done with high order samplers

High order samplers perform poorly on minified images and where pixel shapes are very different.

Need a quantitative measurement of robustness.


Class 1 Resampling

Resampling pixels are similar in most respects to input pixels: translations, small distortions and scale changes.

• Use high order techniques, e.g., Lanczos or spline methods when data is well sampled. The first is easier to program, the second is faster. Limited return from going beyond cubic spline or Lanczos 3.

• Drizzle approach can limit blurring effects of redistribution while accommodating features or undersampling.

• Noise induced errors typically outweigh sampling errors except for blurring.


Class 2 ResamplingResampling pixels substantially different from original pixels, but more or less constant over image.

• For small resampling pixels (magnification) point sampling techniques should work with well behaved images.

• When minifying, supersampling or inherently adaptive techniques, e.g., exact area or other redistribution techniques are best.

• Sampling errors can easily dominate errors due to noise.

• Balance accuracy and robustness.


Class 3 Resampling

Resampling pixels vary significantly over image. Big holes or features. Non-rectangular, non-contiguous grids?

• Use adaptive resampling techniques. Exact area sampling is a good bet.

• Use best robust techniques.

• Objects are going to have substantial distortions in original or sampled grid.


Concluding thoughts No single best resampling technique.

Probably a small suite algorithms can serve for a wide variety of situations.

Can algorithms self-select? Too empirical

Can robustness be defined quantitatively? Can we predict accuracy of resamplers?

Lots more to consider: Extended objects What is the interplay of resampling with other elements

of processing, e.g., compresion?

pragmatic approaches to image resampling

Documents

original image

image differencing

dss image

copying image

image reversal

fourier transform

resampling problems

sinc function