image color correction and contrast enhancement

Image Color Correction and

Contrast Enhancement

Yu Huang

Sunnyvale, California

yu.huang07@gmail.com

Outline

Image enhancement

Histogram equalization (HE)

CLAHE (Contrast Limited Adaptive HE)

Partitioned HE

HE with mean brightness preservation

Gamma correction (GC)

GC with nonlinear masking

Denoising:

Bilateral, Anisotropic diffusion;

Nonlocal Means (NLM), BM3D;

Joint Deblurring and Denoising;

Histogram-based stretching

Auto-color equalizat. (ACE)

Scaling the DCT coefficients

DCT histogram shifting and alpha

rooting

Unsharp masking

Multi-scale decomposition-based

detail enhancement

High dynamic range (HDR) images

by tone mapping

Inverse tone mapping: from LDR to

HDR;

App. A: Shad.+Reflect. Decomp.

App. B: Radiance+Airlight Decomp.

App. C: Learning-based Restoration

App. D: Learning-based Enhancing

Image Enhancement Spatial domain

Global Gamma correction

Histogram equalization

Stretching in the specified interval

Local Gamma correction with nonlinear masking

Contrast limited adaptive histogram equalization

Edge-based sharpening: unsharp masking

Denoising: bilateral filtering

Frequency domain DCT domain: scaling, alpha-rooting;

Homomorphic filter: image decomposition. Retinex model: color constancy

Separation of luminance component with reflectance component;

Emphasis of reflectance component;

Recombine luminance and reflectance.

Tone mapping: between HDR images and LDR images

Histogram Equalization When the usable data of the image is

represented by close contrast values;

Spread out the most freq. intensity values.

CLAHE (Contrast Limited Adaptive HE)

Contrast limiting applied for each neighbor from which a transformation function

is derived:

Proportional to the cumulative distribution function (CDF) of pixel values;

Contrast limited: clipping histogram at a predefined value before computing CDF.

Efficient computation by interpolation: The image is partitioned into equally sized

rectangular tiles.

redistribution

CLAHE Results

Partitioned HE Static partitioned HE: still use the original dynamic range

Brightness preserving bi-HE (BBHE) [Kim’97]: divide histogram based on mean brightness and then HE for each one;

Dualistic sub-image HE (DSIHE) [Wan’99]: median as the separation point;

Minimum mean brightness error bi-HE (MMBEBHE) [Chen’03]: separation point based on minimum mean brightness error;

Recursive mean-separate HE (RMSHE) [Chen’03]: recursively split into multi sub-histograms (initially from two), based on mean;

Recursive sub-image HE (RSIHE)[Sim’07]: recursively split histogram into more sub-histograms, based on median;

Bi-HE plateau limit [Ooi’09]: clipping based on average number of intensity occurrence.

Dynamic partitioned HE: employ the enhanced dynamic range

Dynamic HE [Wadud’07]: partition histogram based on local minima, and new dynamic range based on pixel number;

Brightness preserving dynamic HE (BPDHE) [Irahim’07]: partition with local maxima, brightness normalization after HE.

Comparison of HE, BBHE & RMSHE

Original HE BBHE RMSHE

Histogram Modification with Mean-

Brightness Preservation Weight and threshold before HE (WTHE) [Wang&Ward’07];

Gray-level grouping: group histogram bins and redistribute groups iteratively[Chen’06];

Histogram modification as an optimization problem to adapt the enhancement level;

Linear black and white (B&W) stretching: Decrease histogram bin length for dark and bright end

Histogram smoothing for spikes: Backward-difference of histogram as measure of smoothness

Weighted histogram approximation: Average local variance of all the pixels with the same gray-level is

used to weight the approximation error, so that avoid spikes further.

B&W stretching

Modification of histogram

Measuring input contrast

Limitation of very low slope

Comparison Results

Original image HE

Weighted thresholded HE HE mean brightness preservation

Gamma Correction Power-Law Transformations:

Gamma compression or expansion.

CRT intensity-to-voltage response follows a power function.

original 0.5<gama<1

gama >1 gama <0.5

Gamma Correction with Masking

Bilateral Filter: Edge Preserving

Anisotropic Diffusion Filter

c(p, t) is large when p is not a part of an edge c(p, t) is small when p is a part of an edge

Nonlocal Means

Patch-based (not pixel-based like bilateral filter);

Averaging with nearby pixels of similar texture;

Pixel-wised

Patch-wised

More discriminative, still blurry in finer details.

Note: Intel OpenCV implements NLM for denoising.

NLM Results

BM3D (Block Matching 3-D) Filter

• For each patch, find similar patches in neighboring; • Group the similar patches into a 3-d stack;

• SSD, SAD or kernel-based.

• Perform a 3-D transform (approximated by 2-D + 1-D) and coeff. thresholding (sparsity in transform domain);

• DCT, WT, Walsh-Hadamard transform,…

• Apply inverse 3-D transform (1-D + 2-D);• Calculate the thresholded pixels for weighting;

• Also combine multiple patches in a collaborative way (aggregation);

• Two stages: hard -> wiener (soft).• Extension to color domain:

• YCbCr space;

• Grouping only uses Y component, but apply for Cb, Cr components.

BM3D Flowchart

BM3D Results

Image Deblur with Denoising

Motion blur (camera or object): degradation by convolution of a latent image with a blur kernel during exposure; Averaging of unaligned images along the motion trajectory;

Deblurr is an inverse problem: estimate point spread function, i.e. PSF; Multiple images or single image:

[Yuan et al., 2007]: Noisy/Blurred image pairs, kernel estimated from noisy first;

Hardware-based: hybrid imaging (camera motion), coded aperture (blur kernel);

Presence of noise is a big problem (how to detect blur and noise?);

Ringing artifacts or amplification of noise in deblurring.

Non-blind deconvolution for single image deblur (kernel known): Lucy-Richardson, Wiener filter, LS, TV (total variation) etc.;

[Yuan et al. 2008]: multiscale LR with bilateral filter;

[Chan & Wong, 2009]: Laplacian prior as Total Variation regularizer in LR;

[Xu & Jia, 2010]: spatial prior for blurred edge scale for TV regularization;

Blind deconvolution (BD) for single image deblur (no kernel clue) Ill-posed, only solved by assumptions or priors;

Smoothness, gradient or color priors (the two-color model) for MAP or regularization;

Image Deblur with Denoising

Spatially invariant: uniform BD [Fergus et al. 2006]: variational Bayesian method with gradient prior from natural

image statistics;

[Jia 2007]: alpha matte for kernel estimation;

[Shan et al. 2008]: regularization with high order partial derivatives;

[Cho & Lee, 2009]: edge prediction with a shock filter for kernel estimation;

[Levin et al., 2011]: efficient marginal likelihood maximization;

[Krishnan et al., 2011]: kernel estimation with normalized sparsity (L1 by L2);

Spatially variant: non-uniform BD [Shan et al., 2007]: in-plane rotation estimation for deblur with iterative optimization;

[Whyte et al., 2010]: variation Bayesian with a geometric model for rotation;

[Gupta et al., 2010]: motion density basis for kernel in RANSAC-based optimization.

Optical aberration: lens imperfection [Schuler et al., 2012]: optic blur with a set of bases.

De-blur and de-noise together: How to separate noise from blur?

[Joshi et al., 2009]: two color model with Gaussian mixture for BD;

[Tsai & Lin, 2012]:; denoising first with noise estimation in blur kernel;

[Zhong et al., 2013]: directional low pass filter.

Deblur Results

Input [Fergus et al. 2006] [Joshi et al., 2008] [Cho & Lee, 2009] [Xu & Jia, 2010]

Input [Shan et al. 2008] [Krishna et al., 2011] [Cho & Lee, 2009]

Deblur and Denoising Results

Non-Blind Deconvolution for Deblurring

Blurred + Noise Basic Total Variation (TV)

Laplacian TV Bilateral TV Bilateral Laplacian TV

Non-Blind Deconvolution for Deblurring

Blurred + Noise Basic Total Variation (TV)

Laplacian TV Bilateral TV Bilateral Laplacian TV

Blind Deconvolution for Deblurring

kernel estimation with normalized sparsity

kernel estimation with normalized sparsity

Blind Deconvolution for Deblurring

kernel estimation with normalized sparsity

Histogram-based Stretching Saturated a percentage S1% of the dark pixels and a

percentage S2% of the bright pixels.

Histogram-based stretching interval estimation.

Auto Color Equalization Rizzi, Gatta, Marini proposed in 2002-2004; Similar to the Retinex color perceptual model;

Color constancy to adjust RGB for HVS [Land & McCann1971]; Adapt local contrast

expanding or compressing the dynamic range

Adapt the image to obtain global white balance.

A simplified model of HVS, smoothed local HE. Acceleration by polynomial approximation of slope function

or intensity level interpolation.

slope function

neighboring

ACE Results

Scaling the DCT Coefficients Use of same scale factor for both DC and AC coefficients;

Scales chromatic components with same factor.

Adjustment of background illumination; Use of DC coeff.;

Preservation of local contrast; Enhancement factor fixed;

Preservation of colors: same scaling proportion in YUV space;

8x8 blocks for DCT (overlapping to avoid blocking artifact);

DCT Histogram Shifting

with Alpha Rooting Adapting spatial domain technique into transform domain;

Logarithmic DCT coefficient histogram;

Applying shift in positive direction;

Equalization of DCT coefficient after affine transform;

Combining histogram shifting with alpha rooting

DCT Histogram Shifting Results

Unsharp Masking

Edge preserving filter

Adaptive gain control

A gain function in YENI [Arici’06]

a=1,b=7, c=21. K=1

),(),(),( mnxmnxmnz smooth

),(),(),(),( mnzmnmnxmny

),(),(),( mnxmnamny

Gain defined [Polesel’00]

Adaptive UM Results

Generalized Unsharp Masking [Deng, IEEE T-IP, 2011];

Enhance both contrast and sharpness by the model component and the residual;

Reduce the halo effect by edge-preserving filter;

Avoid out-of-range by log-ratio & tangent operations.

Detail Enhancement

For log-ratio

Generalized Unsharp Masking Result

original only with CE only with DE with CE and DE

Image-Decomposition based

Detail Enhancement Smooth base layer (large scale variations);

Residual detail layer (small scale details);

Contrast expansion/compression;

Compression in base layer;

Enhance in detail layers.

Edge preserving filter to extract the base layer;

Total Variation-based filter;

Bilateral filter;

EAW (Edge Avoiding Wavelet) filter;

Weighted LS filter;

Domain transform;

Multiple scales for flexible detail manipulation.

Some Results of DT-based Detail

Manipulation

(a) Input (b) Detail layer D0 (c) Detail layer D1 (d) Detail layer D2

Some Examples of WLS-based

Detail Enhancement

Original Image Fine Medium Coarse Combined

HDR Imagery A starlit night has an average luminance level of around 10^-3 candelas/m^2,

and daylight scenes are close to 10^5 cd/m^2 ;

Humans can see detail in regions that vary by 1:10^4 at any given adaptation level, over which the eye gets swamped by stray light (i.e., disability glare) and details are lost;

CRT monitor's max display luminance is only around 100 cd/m^2;

A high-quality xenon film projector is still two orders of magnitude away from the optimal light level for acuity and color perception;

HDR (high dynamic range) image save each pixel with 4 bytes (32bits);

Two main methods for generating HDR imaging: Use physically based renderers, generating basically all visible colors;

Take photographs of a particular scene at different exposure times, apply radiometric calibration to camera, combine calibrated images.

Tone Mapping (Tone Reproduction) Match one observer model applied to the scene to another

observer model applied to the desired display image;

HDR needs tone mapping to display colors on LDR monitor;

Tone Mapping operators' 4 Categories global (spatially uniform): Compress images using an identical

(non-linear) curve for each pixel;

local (spatially varying): Achieve dynamic range reduction by modulating a non-linear curve for each pixel independently (local neighborhood);

frequency domain: Reduce the dynamic range of image components selectively, based on their spatial frequency;

gradient domain: Modify the derivative of an image to achieve dynamic range reduction. (Gradient domain HDR compression);

Video tone mapping: temporal coherence;

Note: Inverse tone mapping from LDR to HDR;

Tone Mapping Based on Multi-Scale

Decomposition for HDR Images

Input.img = Detail.img * Base.img: I(x,y) = R(x,y)*L(x,y);

Reflectance layer (large scale variations);

Illuminance layer (small scale details);

Apply edge preserving filter for decomposition too!

Bilateral, WLS, Domain Transform, BM3D,…

Compression in illumination layer (tone mapping);

Unchanged in reflectance layers.

HDR Images

-

Edge

Preservingexp()

Ln()LDR images+

Compress* k

Some Results of HDR Tone Mapping

Bilateral WLS BM3D

Some Results of HDR Tone Mapping

Bilateral WLS BM3D

inverse Tone Mapping: LDR to HDR How to display LDR image to HDR display screen?

Dodging and Burning: inspired bothTM and iTM; Dodging: blocking light from some image areas while illuminating others;

HDR companding [Li, Siggraph’05]: compress the contrast through tone

mapping, then expand it;

Linear contrast scaling [Akyues, Siggraph’07]: a global expansion

method;

Inverse tone mapping [Banterle, SCCG’08]: Map LDR to middle dynamic range;

Find saturated areas by median cut, create expand map from density estimate;

Reconstruct lost luminance by interpolation (weights);

Attenuate artifacts by cross bilateral filtering (luminance-expand map).

Reverse tone mapping [Rempel, Siggraph’07]: similar to iTM. Inverse gamma and noise filter for contrast stretching;

Smooth brightness and edge stopping for enhancement of saturated regions.

Illustrat. of Reverse Tone Mapping

Original (LDR)

HDR tone mappingHDR (bright)

HDR(dark)

Illustrat. of Reverse Tone Mapping

Original (LDR)

HDR tone mappingHDR (bright)

HDR(dark)

Illustrat. of Reverse Tone Mapping

HDR tone mappingHDR (bright)

Original (LDR) HDR(dark)

Illustrat. of Reverse Tone Mapping

HDR tone mappingHDR (bright)

Original (LDR) HDR(dark)

Hue Preserving & Saturation Enhancement

Chromaticity diagram: gamut;

Hue preserving: HSI (hue-saturation-intensity) from RGB;

Methods of saturation enhancement:

1. Increase saturation by fraction;

2. Increase saturation to the maximum based on hue and lumin.;

3. Reduce lumin. by fraction, then increase saturat. by fraction;

4. Reduce lumin. by fraction, then increase saturat. to maximum;

Too strong color enhancement result in poor quality;

Increased saturation introduces noise in uniform areas;

“Out of gamut” problem in saturation scaling;

Saturation clipping: clip before transform back to RGB;

S-type transformation: nonlinear hue preserving.

Saturation normalization: histogram equalization in normalized HSI.

Gamut in CIE-XY Chromaticity Diagram

Hue Saturation Intensity (HSI) Space

NNNNNN

NNN

NNNNNN

BGBRGR

BGRH

I

BGRSBGRI

2

1

2

2cos

2

1

,,min1

3

1

Hue Preserving Saturation Enhancement

(a) original

(b) linearly enhanced (c) s-type function

(d) clipping (e) normalization

Hue Preserving Saturation Enhancement

(a) original

(b) linearly enhanced (c) s-type function

(d) Clipping (e) Normalization

References K. Zuiderveld: Contrast Limited Adaptive Histogram Equalization. Graphics Gems IV, 1994.

Tomasi, C., Manduchi, R. Bilateral filtering for gray and color images. ICCV’98.

N.Moroney.Local Color Correction Using Non-Linear Masking, IS&T Color Imaging, 2000.

C. Gatta, A. Rizzi, D. Marini, ACE: An automatic color equalization algorithm, European Conference on

Color in Graphics Image and Vision (CGIV02), 2002.

Durand, F., Dorsey, J. Fast bilateral Filtering for the display of high-dynamic-range images. ACM T-

Graphics. 21(3), 2002.

A. Buadess, B. Coll, J. Morel. A non local algorithm for image denoising. CVPR, 2005.

R Fergus, B Singh, A Hertzmann, S. T. Roweis, W T. Freeman, "Removing camera shake from a single

image", Siggraph, 2006

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Image denoising by sparse 3-D transform-domain

collaborative filtering. IEEE T-IP, 16(8):2080–2095, 2007.

A Rempel, Trentacoste, Seetzen, Young, Heidrich, Whitehead, Ward. Ldr2Hdr: On-the-fly Reverse Tone

Mapping of Legacy Video and Photographs. Siggraph, 2007.

L Yuan, J Sun, L Quan, HY Shum, "Image deblurring with blurred/noisy image pairs", SIGGRAPH , 2007.

J Jia, "Single Image Motion Deblurring Using Transparency", CVPR, 2007

Z. Farbman et al., Edge-preserving decompositions for multi-scale tone and detail manipulation,

ACM Siggraph, 2008.

J Mukherjee, S Mitra, Enhancement of color images by scaling the DCT coefficient, IEEE T-IP,

17(10), 2008.

Q Shan, J Jia, A Agarwala, "High-Quality Motion Deblurring From a Single Image", SIGGRAPH 2008.

References T Arici, S Dikbas,Y Altunbasak, A Histogram Modification Framework and Its Application for Image

Contrast Enhancement. IEEE T-IP, 18(9), 2009.

D. Krishnan and R. Fergus. "Fast Image Deconvolution using Hyper-Laplacian Priors". NIPS, 2009.

S Cho, S Lee, "Fast Motion Deblurring", Siggraph Asia, 2009.

N. Joshi, C.L. Zitnick, R Szeliski, and D Kriegman, "Image Deblurring and Denoising using Color Priors",

CVPR 2009.

O. Whyte, J. Sivic, A. Zisserman J. Ponce, "Non-uniform Deblurring for Shaken Images", CVPR 2010.

Q Shan, J Jia, S B Kang, Z Qin, "Using Optical Defocus to Denoise", CVPR, 2010

A Gupta, N Joshi, C. L. Zitnick, M Cohen, B Curless, "Single Image Deblurring Using Motion Density

Functions", ECCV, 2010.

S. Cho, J. Wang, S. Lee. Handling Outliers in Non-blind Image Deconvolution. ICCV, 2011

A. Levin, Y. Weiss, F. Durand, W. T. Freeman. "Efficient Marginal Likelihood Optimization in Blind

Deconvolution". CVPR, June 2011.

E. Gastal, M. Oliveira, Domain transform for edge-aware image and video processing, ACM

SIGGRAPH 2011.

K. Panetta, J. Xia, S Agaian. Color image enhancement based on the discrete cosine transform

coefficient histogram, J. of Electronics Imaging, 21(2), 2012.

Y Tai, S Lin, "Motion-aware noise filtering for deblurring of noisy and blurry image", CVPR, 2012.

C Schuler, "Blind Correction of Optical Aberrations", ECCV'2012.

L Zhong, S Cho, D Metaxas, S Paris, J Wang, "Handling Noise in Single Image Deblurring using

Directional Filters", CVPR, 2013.

Appendix: Image Decomposition for

Illumination and Reflectance (IR)

Intrinsic Image Proposed in [Barrow & Tenenbaum, 1978];

Decomposition of illumination and reflectance;

Illumination (shading as shadow and indirect lighting): amount of light incident at a point, i.e. irradiance;

Reflectance (albedo): how the object reflects light.

Application: colorization, retexturing, shadow removal, tone mapping, matting, white balancing, de-hazing, detail enhancement etc.

• It is an ill-posed problem;

Rely on user indications or precise geometry (rough depth) to disambiguate it [Bousseau et al., 2009][Shen et al., 2011];

Single image source: various assumptions about both illumination and reflectance, such as structural sparsity and neighboring smoothness.

Intrinsic Image Retinex theory [Land & McCann, 1971];

Reflectance is piecewise constant while illumination is smooth;

A variational method [Funt et al., 1992];

An edge-preserving filter does the factorization task;

Uniform albedo polyhedra, diffuse ambient lighting for shading [Sinha & Adelson, 1993]; Discriminate Junctions: T junctions as reflectance variation, arrow

and Y junctions as illumination;

Learning-based [Tappen et al., 2005]; Trained on image derivatives to classify gradients.

Non-local texture prior [Shen et al., 2008]; A set of neighbor pixels share the same texture config.;

Follow a Retinex algorithm [Kimmel, 2003], added with the texture constraint.

Similar chromaticity [Shen & Yeo, 2011]; Same reflectance for neighboring pixels as sparsity.

Intrinsic Image Decomposition Comparison

Reflectance (left to

right): [Shen et al.’08]

– [Shen & Yeo’11] –

ground truth

Illuminance (left to

right): [Shen et al.’08]

– [Shen & Yeo’11] –

ground truth

Original

Appendix: Image De-Hazing for

Radiance, Airlight and Transmission

Image Degradation Model

Dichromatic atmospheric scattering model: I(x) = t(x)J(x)+(1-t(x))A; J(x): surface radiance vector (haze free);

A: constant airlight color vector (veiling light);

t(x): medium transmission along the ray.

t(x)=exp(-r*d(x)) with d(x) as depth and r as scattering coeff.

The incoming light blended with the airlight (by atmospheric particles); Haze induces its visual effect is blurring of distant objects ( a clue for

depth inference, i.e. aerial perspective);

The loss of contrasts and colors due to haze or fog as image averaging with a constant color A.

De-hazing or de-foggy is ill-posed too! multiple images (polarization) under different lighting [Joshi & Cohoen,

2010] or additional information (depth or geometry) [Kopf et al.’2008];

Assumptions or strong prior, esp. for single image.

De-noise and de-haze simultaneously or sequentially.

De-hazing (or De-foggy)

[Fattal, 2008]: every patch has uniform reflectance, the appearance of the pixels within the patch expressed in terms of shading and transmission. shading and transmission signals unrelated;

estimate the appearance of each patch by ICA;

Failure when mag. of the surface reflectance is much smaller than that of the air light;

[Tan, 2008]: Divide the image into a series of small patches and the corresponding patch in radiance should have a higher contrast; neighboring pixels should have similar transmission values formulated in a

Markov Random Field, solved by graph-cut or belief propagation;

Produce over enhanced images in practice.

[He et al., 2009]: dark channel prior (easy to apply bright channel prior); The transmission of each patch estimated as the minimum color compo;

Soft matting to ensure neighboring pixels had similar transmission values.

[Kraz & Nishino, 2009]: use natural statistics of albedo and depth; Factorial MRF with statis. independent latent layers for albedo and depth;

Solved a MAP problem by EM.

[Tarel & Hautiere, 2009]: bilateral filter to factor radiance and airlight; Works not well at the depth discontinuities; could be better with guided filter?

Guided Image Filtering

Problems in bilateral filter

Complexity

Gradient distortion: preserve edge, not gradient;

Guided filter: in all local windows , compute the linear coefficients

and the average of in all local windows that covers pixel

Gradient preserving: q has an edge only if I has an edge

Integral images

O(1) time

Non-approximate

Naturally O(N) time independent of the window radius

mean

variance

Guided Image Filtering

Linear regression

Bilateral/joint bilateral filter does not

have this linear model!

Noise/texture

De-hazing Results’ Comparison

Fast Single Image Dehazing with Dark

Channel Prior and Guided Filter Results

Fast Single Image Dehazing with Dark

Channel Prior and Guided Filter Results

Fast Single Image Dehazing with Dark

Channel Prior and Guided Filter Results

Appendix: Machine Learning-based Image

Deblur and Denoising

Sparse Coding in Image Filtering

• A cost function for : Y = Z + n

• Solve for: Prior term

• Break problem into smaller problems

• Aim at minimization at the patch level.

Proximity of

selected

patch

Sparsity of the

representations

Global

proximity

K-SVD Dictionary Learning

Extract overlapping patches from a single image;

clean or corrupted, even reference (multiple frames)?

for example, 100k of size 8x8 block patches;

Applied the K-SVD, training a dictionary;

Size of 64x256 (n=64, dictionary size k).

Lagrange multiplier namda = 30/sigma of noise;

The coefficients from Orthogonal Matching Pursuit;

the maximal iteration is 180 and noise gain C=1.15;

the number of nonzero elements L=6 (sigma=5).

Denoising by normalized weighted averaging:

Image Denoising by Conv. Nets Image denoising is a learning problem to training Conv. Net;

Parameter estimation to minimize the reconstruction error.

Online learning (rather than batch learning): stochastic gradient Gradient update from 6x6 patches sampled from 6 different training images

Run like greedy layer-wise training for each layer.

Image Denoising by MLP Denoising as learning: map noisy patches to noise-free ones;

Patch size 17x17;

Training with different noise types and levels: Sigma=25; noise as Gaussian, stripe, salt-and-pepper, coding artifact;

Feed-forward NN: MLP; input layer 289-d, four hidden layers (2047-d), output layer 289-d.

input layer 169-d, four hidden layers (511-d), output layer 169-d.

40 million training images from LabelMe and Berkeley segmentation!

1000 testing images: Mcgill, Pascal VOC 2007;

GPU: slower than BM3D, much faster than KSVD.

Deep learning can help: unsupervised learning from unlabelled data.

Image Restoration by CNN Collect a dataset of clean/corrupted image pairs which are then used to train a

specialized form of convolutional neural network.

Given a noisy image x, predict a clean image y close to the clean image y*

the input kernels p1 = 16, the output kernel pL = 8.

2 hidden layers (i.e. L = 3), each with 512 units, the middle layer kernel p2 = 1.

W1 512 kernels of size 16x16x3, W2 512 kernels of size 1x1x512, and W3 size 8x8x512.

This learns how to map corrupted image patches to clean ones, implicitly capturing the characteristic appearance of noise in natural images.

Train the weights Wl and biases bl by minimizing the mean squared error

Minimize with SGD

Regarded as: first patchifying the input, applying a fully-connected neural network to each patch, and averaging the resulting output patches.

Image Restoration by CNN Comparison.

Image Deconvolution with Deep CNN Establish the connection between traditional optimization-based

schemes and a CNN architecture;

A separable structure is used as a reliable support for robust

deconvolution against artifacts;

The deconvolution task can be approximated by a convolutional

network by nature, based on the kernel separability theorem;

Kernel separability is achieved via SVD;

An inverse kernel with length 100 is enough for plausible deconv. results;

Image deconvolution convolutional neural network (DCNN);

Two hidden layers: h1 is 38 large-scale 1-d kernels of size 121×1, and h2 is

381x121 convolution kernels to each in h1, output is 1×1×38 kernel;

Random-weight initialization or from the separable kernel inversion;

Concatenation of deconvolution CNN module with denoising CNN;

called “Outlier-rejection Deconvolution CNN (ODCNN)”;

2 million sharp patches together with their blurred versions in training.

Image Deconvolution with Deep CNN

Reference V. Jain and H.S. Seung. Natural image denoising with convolutional networks.

Advances in Neural Information Processing Systems, 21:769–776, 2008.

Burger, Schuler, Harmeling, Image Denoising: Can Plain Neural Networks Compete

with BM3D?, CVPR, 2012.

Xie, J., Xu, L., Chen, E. Image denoising and inpainting with deep neural networks.

Advances in Neural Information Processing Systems 25, 2012.

K. Cho, Simple Sparsification Improves Sparse Denoising Autoencoders in Image

Denoising, ICML, 2013.

D Eigen, D Krishnan, R Fergu, Restoring An Image Taken Through a Window

Covered with Dirt or Rain. ICCV’13;

M Ranzato, V Mnih, J M. Susskind, G E. Hinton, Modeling Natural Images Using

Gated MRFs, IEEE T-PAMI, 2013;

Li Xu, Jimmy S. Ren, Ce Liu, Jiaya Jia, Deep convolutional neural network for image

deconvolution, NIPS, 2014.

Appendix: Machine Learning-based Image

Enhancement

Image Restoration using Online Photo

Collections

image restoration leverages a large database of images gathered from the web;

efficient visual search to find the closest images which define the input’s visual

context;

visual context as an image-specific prior for image restoration;

applications: white balance correction, exposure correction, and contrast

enhancement.

Personalization of image enhancement

First observe user preferences on a training set, and then learn a model

of these preferences to personalize enhancement of unseen images;

The challenge of designing the system lies at intersection of computer

vision, learning, and usability;

Active selection of an instance subset that share the highest mutual information

with the rest of the high-dimensional space: defined as a sensor placement

problem, i.e. each instance can be thought of as a possible sensor location, where

a probe is placed in order to “sense” the space of images;

Distance metric learning: Learning a distance metric between images that reflects

how far two images should be in terms of their enhancement paras.

Enabling seamless browsing for training: a user interface that explores

the space of possible enhancements for each of the training images and

indicate users’ preferences.

Personalization of image enhancement

Basic idea

control parameters

Personal photo enhancement using

example images

correcting special types of images (faces of a photographer’s family and friends ) and

use common faces across images to automatically perform both global and face-

specific corrections;

Apply face detection to align faces between “good” and “bad” photos.

Personal photo enhancement using

example images

Personal image enhancement pipeline

Example-based image color and tone style

enhancement

Learn implicit color and tone adjustment rules from examples;

Discover the underlying mathematical relationships optimally connecting the

color and tone of corresponding pixels in all image pairs;

Locally approximated with a low-order polynomial model.

Example-based image color and tone style

enhancement

Collaborative personalization of image

enhancement personalized preference in image enhancement tend to cluster?

whether users can be grouped accordingly?

such clusters do exist!

derive methods to learn statistical preference models from a group of users;

Collaborative filtering to automatically enhance novel images for new users.

Collaborative personalization of image

enhancement

Overview of collaborative personalization of image enhancement

Learning photographic global tonal

adjustment Creation of a high-quality reference dataset;

Predict a user’s adjustment from a large training set; Describe a set of features and labels;

Single luminance remapping curve applied independently to each pixel;

Regression techniques such as linear least squares, LASSO, and GPR;

Sensor placement: select a small set of representative photos;

Difference learning: models and predicts difference between users;

On this photo, the editors have produced diverse of outputs, from a sunset mood (b) to a day light look (f).

Context-based automatic local image

enhancement

A local method: relies on local scene descriptors and context in addition to high-level

image statistics;

Searching for the best transformation for each pixel in the given image and then

discovering the enhanced image using a formulation based on Gaussian Random

Fields.

Context-based automatic local image

enhancement

Illustration of basic concept. An example highlighting the local enhancement

Overview of coarse-to-fine local enhancement method.

Learning-to-rank approach for image color

enhancement

Take into account the intermediate steps taken in the enhancement

process, which provide detailed information on the person’s color

preferences;

Formulate the color enhancement task as a learning-to-rank (LTR)

problem in which ordered pairs of images are used for training, and then

various color enhancements of a novel input image can be evaluated from

their corresponding rank values;

Ranking is the central component of many information retrieval (IR);

LTR method employed is Multiple Additive Regression Trees (MART) .

From the parallels between the decision tree structures used for ranking

and the decisions made by a human during the editing process, breaking a

full enhancement sequence into individual steps can facilitate training.

Learning-to-rank approach for image color

enhancement

Automatic Photo Adjustment Using Deep

Learning

Explore the use of deep learning in the context of photo editing;

Introduce an image descriptor (pixel, context and global) that accounts for the local

semantics of an image.

Middle (from top to bottom):

input image, semantic label

map and the ground truth for

the Local Xpro effect;

Left and right: color mapping

scatter plots for four semantic

regions.

Automatic Photo Adjustment Using Deep

Learning

The architecture of the DNN

Multi-scale spatial pooling schema

Pipeline for constructing the semantic label map

Automatic Photo Adjustment Using Deep

Learning

Three Stylistic Local

Effects:

1. Local Xpro,

2. Foreground Pop-Out,

3. Watercolor.

Reference K. Dale, M. K. Johnson, K. Sunkavalli, W. Matusik, H. Pfister. Image restoration using

online photo collections. ICCV, 2009;

S B Kang, A Kapoor, D Lischinski, Personalization of image enhancement. CVPR’10;

N Joshi, W Matusik, W., E H Adelson, D J Kriegman, Personal photo enhancement

using example images. ACM T-Graph. 29, 2 (April), 2010;

B Wang, B., Y Yu, Y-Q Xu, Example-based image color and tone style enhancement.

ACM SIGGRAPH 2011;

J Caicedo, A Kapoor, S B Kang, Collaborative personalization of image

enhancement. CVPR, 2011;

V Bychkovsky, S Paris, E Chan, F Durand, Learning photographic global tonal

adjustment with a database of input/output image pairs. CVPR ’11;

S J Hwang, A Kapoor, S B Kang. Context-based automatic local image

enhancement. ECCV’12;

J. Yan, S. Lin, S.B. Kang, X. Tang, A learning-to-rank approach for image color

enhancement, CVPR 2014;

Z Yan, H Zhang, B Wang, S Paris, Y Yu, Automatic Photo Adjustment Using Deep

Learning, ACM Siggraph Asia, 2014.

Appendix: Chroma Contrast Preserving

in Color to Grayscale Conversion Color to Grayscale Conversion: Dimension reduction 3D to 1D;

Display has olny Y-range for contrast condensing (isoluminant color problem);

Local method: Local changes, contradictions, more comput. costs

Color contrast map to enhance gray image;

Same color may output different gray value, haloing artifacts may produce.

Global method: Same luminance for the same RGB triplets;

Speed, naturalness, luminance range, color feature optimized in conversion;

Mostly, color order is strictly satisfied, might be ambiguous (culture, person).

Bala & Eschbach [2004, Color Imaging Conf.]: high-frequency chrominance;

Rasche et al. [2005, IEEE CG&A and EG]: MDS in color quantization;

Gooch et al. [2005, ACM Siggraph]: Saliency preserving on local contrasts;

Grundland & Dogson [2005, PR]: image dependent piecewise linear mapping

Neumann et al. [2007, CAGVI]: Gradient-inconsistency correction

Smith et al. [2008, Eurograph]: enhance greyscale to reproduce original contrast

Lu, Xu & Jia [2012, ICCP]: relax color order constraints by human perception;

Comparison of Color2Gray Conversion

Original color image CIELab’s Conversion. Lu, Xu, Jia's Conversion.

Gooch05's conversion Grundland05's Conversion. Smith08's Conversion

Rasche05's conversion. Bala04's Conversion. Neumann07’s Conversion

Appendix: Matlab Image functions

Histogram equalization: J = histeq(I,hgram);

CLAHE: contrast limited adaptive HE adapthisteq(L,'NumTiles',[4 4],'ClipLimit',0.005, 'Alpha', 0.6)*255;

Clipping with gamma correction: J = imadjust(I,[low_in; high_in],[low_out; high_out],gamma);

Unsharp masking (binary mask BW is defined first) h = fspecial('unsharp'); I2 = roifilt2(h,I1,BW);

Bilateral filter [k,j] = meshgrid(-r:r,-r:r);

h = exp( -(j.^2 + k.^2)/(2*sigma_s^2) ) .* ...

exp( -(w - w(r+1,r+1)).^2/(2*sigma_d^2) );

y(m,n) = h(:)'*w(:) / sum(h(:));

Gamma correction Gamma=1.5; Correction = 255 * (x/255).^ Gamma;

Appendix: OpenCV Image functions Brightness and contrast adjustments

saturate_cast<uchar>( alpha*( image.at<Vec3b>(y,x)[c] ) + beta );

Gamma correction: LUT can accerlerate it. cvPow(img, trans, gamma);

cvConvertScaleAbs(trans, trans, 1, 0);

Gpu::gammaCorrection();

Histogram Equalization: single channel cvtColor( src, src, CV_BGR2GRAY );

equalizeHist( src, dst );

Unsharp masking GaussianBlur(im, tmp, Size(5,5), 5);

addWeighted(im, 1.5, tmp, -0.5, 0, im);

Box filter for sharnpening boxFilter(image, bImg, -1, ksize, anchor, normalize, BORDER_REFLECT);

edgeImage = image - bImg;

image = image + gain*edgeImage;

Contrast Limited Adaptive Histogram Equalization

openCv2.4.5 version;

Gpu::CLAHE::apply();