frequency domain methods for demosaicking of bayer sampled color images eric dubois
DESCRIPTION
Frequency domain methods for demosaicking of Bayer sampled color images Eric Dubois. Problem Statement. - PowerPoint PPT PresentationTRANSCRIPT
-
Frequency domain methods for demosaicking of Bayer sampled color images Eric Dubois
Frequency-domain Bayer demosaicking
-
Problem Statement
Problem: Most digital color cameras capture only one color component at each spatial location. The remaining components must be reconstructed by interpolation from the captured samples. Cameras provide hardware or software to do this, but the quality may be inadequate.Objective: Develop new algorithms to interpolate each color plane (called demosaicking) with better quality reconstruction, and with minimal computational complexity.
Frequency-domain Bayer demosaicking
-
Retinal Cone MosaicThe human visual system must solve a similar problem!
Frequency-domain Bayer demosaicking
-
Construction of color image from color planes+
Frequency-domain Bayer demosaicking
-
Lighthouseoriginal
Frequency-domain Bayer demosaicking
-
Lighthousered original
Frequency-domain Bayer demosaicking
-
Lighthousegreen original
Frequency-domain Bayer demosaicking
-
Lighthouseblue original
Frequency-domain Bayer demosaicking
-
Formation of Color planes
Frequency-domain Bayer demosaicking
-
Lighthousered subsampled
Frequency-domain Bayer demosaicking
-
Lighthousegreen subsampled
Frequency-domain Bayer demosaicking
-
Lighthouseblue subsampled
Frequency-domain Bayer demosaicking
-
LighthouseBayer CFA image
Frequency-domain Bayer demosaicking
-
Color plane interpolationGAGBGLGRGIGreen channel: bilinear interpolation
Frequency-domain Bayer demosaicking
-
Color plane interpolationRCRed channel: bilinear interpolationRNWRNERSWRSERS
Frequency-domain Bayer demosaicking
-
Lighthousered interpolated
Frequency-domain Bayer demosaicking
-
Lighthousegreen interpolated
Frequency-domain Bayer demosaicking
-
Lighthouseblue interpolated
Frequency-domain Bayer demosaicking
-
LighthouseInterpolated color image
Frequency-domain Bayer demosaicking
-
Lighthouseoriginal
Frequency-domain Bayer demosaicking
-
Can we do better?Color planes have severe aliasing. Better interpolation of the individual planes has little effect.
Frequency-domain Bayer demosaicking
-
Lighthousered interpolatedwith bilinear interpolator
Frequency-domain Bayer demosaicking
-
Lighthousered interpolatedwith bicubic interpolator
Frequency-domain Bayer demosaicking
-
Can we do better?Color planes have severe aliasing. Better interpolation of the individual planes has little effect.We could optically prefilter the image (blur it) so that aliasing is less severe.
Frequency-domain Bayer demosaicking
-
Lighthousered interpolatedwith bilinear interpolator
Frequency-domain Bayer demosaicking
-
Lighthouseprefiltered red interpolatedwith bilinear interpolator
Frequency-domain Bayer demosaicking
-
LighthouseInterpolated color image
Frequency-domain Bayer demosaicking
-
Lighthouse Prefiltered & Interpolated color image
Frequency-domain Bayer demosaicking
-
Lighthouse original
Frequency-domain Bayer demosaicking
-
Can we do better?Color planes have severe aliasing. Better interpolation of the individual planes has little effect.We could optically prefilter the image (blur it) so that aliasing is less severe.We can process the three color planes together to gather details from all three components.
Frequency-domain Bayer demosaicking
-
Can we do better?There have been numerous papers and patents describing different algorithms to interpolate the color planes they all work on the three planes together, exploiting the correlation between the three components.Gunturk et al. published an extensive survey in March 2005. The best methods were the projection on convex sets (POCS) algorithm (lowest MSE) and the adaptive homogeneity directed (AHD) algorithm (best subjective quality). We present here a novel frequency-domain algorithm.
Frequency-domain Bayer demosaicking
-
Spatial multiplexing modelsubsamplingmultiplexing
Frequency-domain Bayer demosaicking
-
Spatial multiplexing model
Frequency-domain Bayer demosaicking
-
Frequency-domain multiplexing modelRe-arranging the spatial multiplexing expression
Frequency-domain Bayer demosaicking
-
Frequency-domain multiplexing modelDavid Alleysson, EPFL
Frequency-domain Bayer demosaicking
-
Luma and chrominance components
Frequency-domain Bayer demosaicking
-
Luma and chrominance componentsLuma fLChroma_1 fC1Chroma_2 fC2
Frequency-domain Bayer demosaicking
-
Lighthouse BilinearlyInterpolated color image
Frequency-domain Bayer demosaicking
-
Frequency-domain demosaicking algorithmExtract modulated C1 using a band-pass filter at (0.5,0.5) and demodulate to basebandExtract modulated C2 using band-pass filters at (0.5,0.0) and (0.0, 0.5), demodulate to baseband, and combine in some suitable fashion (the key)Subtract modulated C1 and remodulated C2 from the CFA to get the estimated luma component L.Matrix the L, C1 and C2 components to get the RGB representation.
Frequency-domain Bayer demosaicking
-
Spectrum of CFA signalab
Frequency-domain Bayer demosaicking
-
Using C2a onlyUsing C2b only
Frequency-domain Bayer demosaicking
-
OriginalFrom C2a onlyFrom C2b onlyDemosaicking using C2a only or C2b only -- details
Frequency-domain Bayer demosaicking
-
Demosaicking Block Diagram
Frequency-domain Bayer demosaicking
-
Spectrum of CFA signalab
Frequency-domain Bayer demosaicking
-
Design IssuesHow to choose the filters h1, h2a and h2bFrequency domain design methodsLeast-squares design methodsSize of the filtersHow to combine the two estimates and Choice of features to guide weighting The two above issues may be inter-related.
Frequency-domain Bayer demosaicking
-
Filter designGaussian filters (Alleysson)Window design or minimax designDefine ideal response, with pass, stop and transition bandsApproximate using the window design methodRefine using minimax or least pth optimizationCan design low-pass filters and modulate to the center frequency
Frequency-domain Bayer demosaicking
-
Filter specification u v val0.000 0.00 1.00.110 0.00 1.00.110 0.02 1.00.000 0.10 1.00.030 0.10 1.00.070 0.06 1.00.338 0.00 0.00.338 0.05 0.00.050 0.36 0.00.000 0.36 0.00.184 .205 0.00.500 0.00 0.00.000 0.50 0.00.500 0.50 0.0
Frequency-domain Bayer demosaicking
-
Ideal response perspective view
Frequency-domain Bayer demosaicking
-
Ideal response contour plot
Frequency-domain Bayer demosaicking
-
Window design perspective view
Frequency-domain Bayer demosaicking
-
Window design contour plot
Frequency-domain Bayer demosaicking
-
Least pth filter perspective view
Frequency-domain Bayer demosaicking
-
Least pth filter contour plot
Frequency-domain Bayer demosaicking
-
21 x 21 filters in SPL published algorithmuvh2ah2bh1
Frequency-domain Bayer demosaicking
-
Adaptive weighting of C2a and C2bWe want to form the estimate of C2 by choosing the best between C2a and C2b, or perhaps by a weighted average. We have used
should be near 1 when C2a is the best choice, and near 0 when C2b is the best choice
Frequency-domain Bayer demosaicking
-
Typical scenarios for local spectrumLC2aC2aC2bC2bC1C1C1C1B: C2b is better estimateLC2aC2aC2bC2bC1C1C1C1A: C2a is better estimateuvvu
Frequency-domain Bayer demosaicking
-
Scenario AScenario B
Frequency-domain Bayer demosaicking
-
Typical scenarios for local spectrumLC2aC2aC2bC2bC1C1C1C1B: C2b is better estimateLC2aC2aC2bC2bC1C1C1C1A: C2a is better estimateuvvu
Frequency-domain Bayer demosaicking
-
Weight selection strategyScenario A: average local energy near (fm, 0) is smaller than near (0, fm ).Scenario B: average local energy near (0, fm ) is smaller than near (fm, 0).Let be a measure of the average local energy near (fm, 0), and be a measure of the average local energy near (0, fm ).
Frequency-domain Bayer demosaicking
-
Gaussian filters for local energy measurementuvfm = 0.375
Frequency-domain Bayer demosaicking
-
ResultsResults with this adaptive frequency-domain demosaicking method were published in IEEE Signal Processing Letters in Dec. 2005. All filters were of size 21 x 21. Filters h1, h2a and h2b were designed with the window method, with band parameters determined by trial and error. The method gave the lowest mean-square reconstruction error on the standard set of Kodak test images compared to other published methods.
Frequency-domain Bayer demosaicking
-
Mean square error comparison
Frequency-domain Bayer demosaicking