spie proceedings [spie 2012 international conference on graphic and image processing - singapore,...

A Fast First-order Continuation Total Variation Algorithm for Image

Denoising

Dongfang Chen, Ningtao Zhang

College of Computer Science & Technology, Wuhan University of Science & Technology

Wuhan, China

[email protected], [email protected]

ABSTRACT

This paper describes a new denoising algorithm which named Continuation total-variation denoising (CTVD)

algorithm. The algorithm is based on the novel Nesterov’s first-order method, most notably Nesterov’s smoothing

technique, tailored to the image processing application. The algorithm also introduces a fixed point continuation

technique to increase the convergence rate, and in particular when to deal with high dynamic range signals and large-

scale problems.

Keywords—Total-variation; image denoising; continuation technique; fixed point technique; first-order method; smooth

technique

1. INTRODUCTION

Image denoising techniques have become important tools in signal applications and many computer vision systems

that require sharp images obtained from noisy. The total variation (TV) formulation could provide a good mathematical

basis for several basic operations in image processing application [1], such as image denoising, image reconstruction,

image impainting, image deblurring, etc.

Using the TV formulation cannot easily solve the large scales of the image problems and the non-smoothness of the

objective dimensions. Many state-of-the-art methods solve the large scale accurately, but with low computational

complexity, such as sub-gradient methods [6, 7], dual formulations [8] primal-dual methods [9], etc.

Recently, Nesterov published a seminal paper [2-5] which couples smoothing techniques with an improved gradient

method to derive first-order methods. As a consequence of this breakthrough, many techniques has been improved for

dealing with special problems in image processing, and the result of these techniques demonstrate that they are fast,

accurate and robust in sense. [10] introduces a fixed point continuation technique which has been used with some

success to increase the speed of convergence, the result is better when dealing with large scale problem.

Our paper builds on recently published both first-order methods developed by Nesterov and fixed point continuation

technique. We would like to the demand for high accuracy and low computation complexity to introduce an accurate and

fast CTVD algorithm to handle noise signals. Our new first-order algorithms have /1O complexity, where is the

problem’s accuracy. Compared to [11, 12], we introduce the continuation technique and provide practical complexity

bounds. The main contribution of the paper consist in showing that this algorithm obeys robust, accuracy and flexibility.

As early emphasized, our algorithm is based Nesterov’s method and continuation technique. In Section 2, our paper

gives a brief but essential summary of Nesterov’s method and continuation technique. Then we present our method for

TV-based continuation denoising in Section 3. We report on several experiments and compare with some other existing

fast order methods in Section 4. At last Section we demonstrate the computational and accuracy of our methods.

International Conference on Graphic and Image Processing (ICGIP 2012), edited by Zeng Zhu, Proc. of SPIE Vol. 8768, 87685Z · © 2013 SPIE

CCC code: 0277-786X/13/$18 · doi: 10.1117/12.2012485

Proc. of SPIE Vol. 8768 87685Z-1

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/03/2013 Terms of Use: http://spiedl.org/terms

2. NOTATION, NESTEROV’S METHOD AND CONTINUATION

2.1 Notation

Image nmRx 0 degradation model write the form:

zxb 0, (1)

where 0x is the original image of interest, z is a white noise term, and b is the degradated observation. We first define

two nm arrays '

cX and '

rX with the finite-difference approximations to the partial derivatives in the directions of the

columns and rows:

T

nrmc XDXXDX '' , . (2)

where mD and nD are the discrete approximations gradient operators. Then the gradient approximation of the pixel can

be writer

12

'

'

)(

R

X

XxD

ijr

ijc

ij . mn

ij RD 2 (3)

mnmn

mn

R

D

D

D

2

11

. (4)

In [17], it show that the 2-norm of this matrix satisfies 82

2D , and the gradient norm approximately satisfies

2'2'2

2 ijrijcij XXxD . (5)

So the total variation (TV) discrete gradient approximations of image could define as:

m

i

n

j

ij xDxT1 1

2)( . (6)

In image denoising, a standard approach attempts to denoise x by solving

2

1 12

min

bxtosubject

xDimizem

i

n

j

ij

(7)

2.2 Nesterov’s method

In [3],[ 5], Nesterov presents a subtle algorithm to minimize any smooth convex functions f on convex sets pQ .

)(min xfpQx (8)

where pQ is a feasible set. The function f is assumed to be differentiable and its gradient )(xf is Lipschitz and

obeys

22

)()(

bxLyfxf . (9)



where L is an upper bound on the Lipschitz constant. With these assumptions Nesterov’s algorithms minimizes f

over

pQ by iteratively estimation three sequences kk yx ,

and kz while the feasible set pQ

. The algorithm takes the

following form (figure 1):

2.3 Continuation technique

Inspired by these homotopy algorithms which find the solution to the lasso problem for values of ranging in an

interval. [13] introduces a fixed point continuation technique which solves 1 -penalized least-square problem. The point

of the technique is that it has been noticed that solving the lasso is faster when is low. The idea is very easy: propose a

warm start , 1 and use 1 kk to solution the next problem. This technique has been successfully used in

[14, 15]. So it has been shown to be a usefully tool to increase the speed of computer convergence.

Figure 1. Nesterov’s method

3. CONTINUATION TOTAL-VARIATION DENOISING

3.1 The first-order algorithm

Interior point algorithms is accuracy but intractable. So we introduce a new fast first-order algorithm to solve image

denoising build upon Nesterov’s method, which is an efficient scheme for minimization of saddle point problems on

bounded convex sets. The main point of the method is to make a smooth )( -approximation with Lipschitz

continuous derivatives to the non-differentiable TV function, and then subsequently minimize this approximation using

an optimal first-order method for minimization of convex functions with Lipschitz continuous derivatives. We use the

continuation technique to control the speed of convergence in our algorithms. So the algorithm satisfys to deal with the

large-scale problems.

Following [16] and rewrite (7) as a saddle point problem of the form

Initialize 0x . 00 x , For 0k

1) Evaluate )( kxf .

2)Compute ky :

)()(2

minarg2

2k

T

kkQxk xfxxxxL

yp

3)Compute kz :

k

i i

T

iipQxk xfxxxfLzp 0

)()()(minarg

4)Update kx : kkkkk yzx )1(

Stop when a given criterion is valid.

Where 2

02

1)(

xxxf p ;

2

1

ii ;

3

2

Kk



DxuT

QxQx dp maxmin , (10)

where the feasible sets

},,1,,,1,1|{

},|{

2

2

njmiuuQ

bxxQ

ijd

p

(11)

and the corresponding prox-function, which we choose as,

2

22

1)( bxxf p and

2

22

1)( uufd . (12)

The two functions’ bounded are 2

2

1)(max

xf p

Qxp

p

and mnxfdQx

dd 2

1)(max

.

(13)

As a smooth approximation for )(xf we then use an additive modification of )(xf with the prox-function

associate with dQ :

)}(max{)( ufDxuxf d

T , (14)

where )(xfd is our dual prox-function. uf is the well-known Huber function and particularly )(xf is given by

.]),[sgn(

,][],[])[(

1

otherwiseix

ixifixixf

(15)

Nesterov’s optimal first-order method for minimizing the convex function )(xf with Lipschitz continuous

derivatives is listed in Section 2.2. We terminate the algorithm when the duality gap satisfies

DyuuDxD TT

ij

m

i

n

j

ij2

1 12

. (16)

Following Nesterov’s method, we need to solve the smooth constrained problem, and rewrite (10) as

)(min xfpQx

. (17)

Once )(xf at kx is computed, Updating the Step 2 and the Step 3 are auxiliary iterates.

3.2 Updating kyand kz

To update ky , we need to deal with

)()(2

minarg2

2k

T

kkQxk xfxxxxL

yp

(18)

where kx is given. The Lagrangian formation for this problem is

)(2

)()(2

222

),(22

xbxfxxxxL

L k

T

kkx (19)

From the Karush-Kuhn-Tucker (KKT) conditions[21], ky is the solution to the linear system



kkk xfL

xbL

yL

11 , (20)

so

kkk xf

Lxb

LL

Ly

1 , (21)

and we similarly obtain

i

ki

ik xfL

xbLL

Lz

10

(22)

3.3 Accelerating algorithms with continuation

For a fixed value of , the convergence rate of the algorithm obeys

2

2

02

2)()(

k

xxLxfyf k

, (23)

where

x is fmin over pQ . Using continuation-like algorithm in figure 2:

Figure 2. Continuation total-variation denoising algorithm

We demonstrate the continuation-inspired algorithm by applying Nesterov’ method with continuation to deal with a

image denoising problem. The acceleration algorithms seem to be a better candidate for high accuracy and quite stable to

solving denoising problem. The number of continuation steps is dependent with the dramatically the number of iterations,

and we have observed that choosing 8,7,6,5 lead to reasonable results.

4. NUMERICAL RESULTS

In this section we illustrate the performance of CTVD compared to the basic TV denoising [17,18] algorithms, the

recent FISTA algorithm of [19] and the algorithm in [20]. In our experiment zxb 0 is an observed image of

222 512,256,128N pixels contaminated by a zero-mean additive white Gaussian noise z of standard

deviation006.0 x (PSNB=12.2dB) .

In Figure 1, it show the Original image and denoising image which size is 512 ×512, which include a standard

Gaussian noise. We test our algorithm with other state-of-the-art algorithms on the same level. To demonstrate the

computational performance of our CTVD algorithm, we created several smaller problems by the original clean image,

and in each instance we added the same Gaussian white noise with standard deviation. Then we solved the same problem

by TV denoising algorithm and CTVD algorithm. Our algorithms introduce the fixed-point continuation technique and

decreasing smoothing parameters to accelerate computational rate. Figure 2 shows the solution computed by CTVD

Initialize 0 , 0x and continuation times T.

1.) Apply Nesterov’s first-order algorithm with t

and 10 txx ;

2.) Decrease the value of tt 1: with 1 ;

Stop when reach the desired value of f .



Original image Noisy image

CND image FISTA denoised image

(Left) and FISTA (Right). FISTA is a state-of the-art method on image processing, and it also builds on the Nesterov’s

method.

Figure 3. Left: clean images of 512×512. Right: noisy image

Figure 4. The result of CTVD algorithms. Right: FISTA’ s result.

TABLE I: TIMING STUDY RESULTS FOR TV DENOISING AND CTVD ALGORITHM. SHOWS THE COMPUTATIONAL PERFORMANCE OF TV

DENOISING ALGORITHMS WITHOUT CONTINUATION AND WITH CONTINUATION. size time MSE Prediction

Error

TV denoising 512*512 7.15 s 4.4278 0.0194

CTVD 512*512 3.79 s 4.2188 0.0189

TV denoising 256*256 1.81 s 3.4847 0.0184

CTVD 256*256 0.94 s 3.3724 0.0181

5. CONCLUSION

We focused on total variation formulations for 2D image processing. Build on Nesterov’s first-order method with

smooth and nonsmooth on convex sets, fix-point continuation technique, and compressed sensing’s novel technique, we

present efficient a fast first-order continuation total variation algorithm. Our algorithm also uses recently the gradient

decent technique to accelerate compute. The performance of algorithm is comparable or better than state-of-the-art

methods.

Our algorithm only requires little computation with complexity /1O .

In the paper, we have used some recently introduced techniques that be motivated by compressed sensing. We have

taken some steps towards the new theory, and compressive signal processing will be a new study field in signal

processing. In the future we hope to analysis of data obtained form compressive measurements.



ACKNOWLEDGMENT

Supported by the National Natural Science Foundation of China (60975031)

REFERENCES

[1] Chan, T. F, Shen, J: Image processing and analysis: vaiational, PDE, wavelet, and stochastic methods. SIAM,

Philadelphia , 2005

[2] Nesterov, Yu.: Introductory lectures on convex optimization. Kluwer, Dordrecht, 2004

[3] Nesterov, Yu.: Smooth minimization of nonsmooth functions. Math. Program. Ser. A 103, 127–152, 2005

[4] Nesterov, Yu.: Excessive gap technique in non-smooth convex optimization. SIAM J. Optim. 16, 235–249, 2005

[5] Nesterov, Yu: Gradient methods for minimizing composite objective functions. CORE Discussion Papers series,

Université Catholique de Louvain, Center for Operations Research and Econometrics. http://www.uclouvain.be/en-

44660.html 2007

[6] Alter, F, Durand, S, Froment, J: Adapted total variation for artifact free decompression of JPEG images. J. Math.

Imaging Vis. 23, 199–211, 2005

[7] Combettes, P. L, Pennanen, T: Generalized mann iterates for constructing fixed points in Hilbert spaces. J. Math.

Anal. Appl. 275, 521–536, 2002

[8] Krishnan, D, Ping, L, Yip, A. M: A primal-dual active-set methods for non-negativity con-strained total variation

deblurring problems. IEEE Trans. Image Process. 16, 2766–2777, 2007

[9] Darbon, J, Sigelle, M: Image restoration with discrete constrained total variation. Part I: fast and exact optimization.

J. Math. Imaging Vis. 26, 261–276, 2006

[10] E. T. Hale, W. Yin, and Y. Zhang, A fixed-point continuation method for l1-regularized minimization with

applications to compressed sensing, Technical Report - Rice University, 2007

[11] Aujol, J. F: Some first-order algorithms for total variation based image restoration. J. Math. Imaging Vis. 34, 307–

327, 2009

[12] Weiss, P., Blanc-Féraud, L., Aubert, G.: Efficient schemes for total variation minimization under constraints in

image processing. SIAM J. Sci. Comput. 31, 2047–2080, 2009

[13] E. T. Hale, W. Yin, and Y. Zhang, A fixed-point continuation method for l1-regularized minimization with

applications to compressed sensing, Technical Report - Rice University, 2007

[14] E. Van Den Berg and M. P. Friedlander, Probing the Pareto frontier for basis pursuit solutions, SIAM Journal on

Scientific Computing, 31 , 890-912,2008

[15] E. Van Den Berg and M. P. Friedlander, Probing the Pareto frontier for basis pursuit solutions, SIAM Journal on

Scientific Computing, 31 (2008), pp. 890 - 912

[16] Boyd, S, Vandenberghe, L: Convex Optimization. Cambridge University Press, Cambridge, 2004

[17] M. A. Figueiredo, R. Nowak, and S. J. Wright, Gradient projection for sparse reconstruction: Application to

compressed sensing and other inverse problems, IEEE Journal of Selected Topics in Signal Processing, 1, pp. 586-

597, 2007

[18] J. Dahl, P. C. Hansen, S. H. Jensen, T. L. Jensen, Algorithms and software for total variation image reconstruction

via first-order methods, Numer Algor, 53, pp. 67–92 , 2010

[19] A. Beck and M. Teboulle, Fast iterative shrinkage-thresholding algorithm for linear inverse problems, SIAM J.

Imaging Sciences, Vol. 2, No. 1, pp. 183–202, 2008

[20] P. Weiss, G, Aubert, L. B. Féraud, Efficient schemes for total variation minimization under constraints in image

processing. version 3 - 7 Mar 2008

[21] R. T. Rockafellar, Convex analysis, Princeton Landmarks in Mathematics and Physics, Princeton University Press,

1970



spie proceedings [spie 2012 international conference on graphic and image processing - singapore,...

Documents