on the rate of convergence of certain methods of … · select a z0 such that 2.1 is satisfied and...

Marhe111atical Programming 2 (1972) 230 257 North-Holland PublishinKCompany

ON THE RATE OF CONVERGENCE OF CERTAIN METHODS OF CENTERS*

0. PIRONNEAU and E. POLAK U11i11ersiry o[Califomia. Berkeley. California, U.S.A.

Received 22 March 1971 Revised manuscript received 13 September 197 1

It is ~hown in this paper that a theoretical method of centers, introduced by Huard, con­verges linearly. It is also shown, by counter-example, that a modified me thod of centers due to I luard and a method of feas ible direction due to Topkis and Veino! canno t converge linearly even under convexity assumpt ions. Because of this, a new modified method of centers is intro­duced wh ich uses a quadratic programming direct ion finding su broutine. In most uses this new method 1s not more complicated than ll uard's modified method of centers. But it does converge linearly. A method for implementing it without Joss of rate of convergence is also discussed.

0. Introduction

The family of optimization algorithms known as methods of centers were introduced by Huard [ 4]. They d iffer from one another only in the distance function used to establish a cen ter at each iteration. In their o riginal form these algorithms were of a theoretical nature, and because of this, various implem entations, or modified methods of cen­ters, have been proposed. The best known of these modified methods of ce nters is also due to Huard [6 ] (These algorithms are also di scussed in l 14] ). In this paper we shall show that a theore tica l me thod of cen­ters, presented in [5 ] , and a new modified method of centers converge linea rly on a class of problems.

The new modified method of centers to be discussed in this paper is a variation of the algorithm described in [6] Although we shall not estab­lish the rate of convergence of the algorithm in [6 ] in this paper, we wish to mention that we were able to show that the algorithm in [6]

*Research sponsored by the Joint Services Electronics Program, Grant AF-AFOSR-68-1488 and the National Aeronautics and Space Administration, Grant NGL-05-003-016.

'

On the rate of convergence of certain methods of centers 231

converges at least as fast as I /0 under the same assumptions und er which our algorithm converges linearly. Furthermore, the example in th e appendix indicates that the algorithm in [6] cannot converge linear­ly under the assumptions used in this paper.

After a few preliminary results in sec tion I, we shall obtain. in sec­tion 2, the rate of convergence of a theore tical method of centers based on a distance fun ction defined by formula (7) in l5.l. In sec tion 3, we shall describe a new modified method of centers and we shall establish a bound on its rate of convergence. In addition we shall desc ribe an e f­ficien t imple mentation for this new method.

As we shall see. our analysis depe nds heavily on an ex tension of Wolfe's duality theorem [ 18] , l 9]. This ex tension is due to G eoffrion [5]. and is restated for the 1;,ake of convenience in Appendix B. It may be interesti ng to observe that results closely related to duality theory have also been used by Faure and Huard f 31, Tremolieres [ 17], Loots­ma [8] and Mifnin [JO] in the study of convergen ce and rate of con­vergence of methods of centers based on a distance fun c tion defined by formula (6) in (5].

Since we shall be exclusively interested in rate of convergence, we sha ll assume that the read er is familiar with methods of ce nters and their ' convergence properties. In any event, th e read er will find these described in con siderable de tail in [ 14] .

I . Preliminaries

We shall consider in this paper algorithms which solve the following problem.

I. I. Problem. min {.t°(z)lp°(z) ~ 0, j = 1, 2, .. ., m}, where Ji: IR"_. IR 1

for j = 0, I , 2, ... , mare continuously differentiable.

We shall assume that Problem I. I has at least one solution ~. We shall show tha t a number of these algorithms are R-linearly converge nt (see [ 11], p. 29 1). For our purposes, it suffices to recall that if a sequence {Z ;} f:o in IR /1 co nverges to a point z and there exist a k E (0. I) and a Q > 0 such that

( 1.2) 11.:; - Zll~Qk; fo r i=O, 1,2, ...

then {z;} co nverges to z, R-linearly.

232 0. Pironneau and E. Polak

Given {zi}i':'o constructed by any one of the algorithms to be consi­dered, we shall begin by showing that there exists an l E (0, 1) such that

( 1.3) J°(::1+

1) - J°(z) ~ l[J'>(zi) f° (z)] for i=O, 1, 2, ....

Obviously, ( 1.3) implies that

( 1.4) !° (z i) - !° (z) < Li[!° (z 0 ) !°Cz)L ; = o, 1, 2, ...

We shall then make use of the lemma below, which requires somewhat

stronger assumptions, to deduce ( l. 2) from ( 1.4 ).

l .S. Lemma. Suppose that (i) the fimctions Ji, j = 0, 1, ... , m, are twice collti11uously differentiable and there exist constants€> 0, m

0 E (0, l]

such that

a2!°(z) ,. ( 1.6) m 0 llyll2 ~ y, y for ally ElR. 11

, for all z E B(z, €), az 2

where z is a solution of ( 1.1) and B(z. €) = {zl nz- zll ~ E} ; (ii) the Kuhn­Tucker constraint qualification is satisfied al z; (iii) {zi} i;O is a sequence in C~ {zlf(z)~ 0,j= 1, 2, ... , m}. which converges to z and satisfies

(1.7) J°(z 1 ) - !°Cz)~ /i[J°(z 0 ) - J°(Z)I for i= 0, 1, 2, ...

Thell there exists an integer i0 such that

( 1.8) llz. - 2112 ~ ki io 2 [fo(zi ) - fo(z)] for all i ?-_ io . ' mo o

Proof Since {z;} converges to:, there exists an i0 such that zi E B(z, E)

for all i ~ i0

. Without loss of generality, we may assume that i0 = 0. Ac­cording to the Taylor expansion formula, for any zi E {z1} and any

)... E IR. 111 + l, there exists a 8 A. (z) E (0, l) such that

Ill 111

(1.9) L) t./[f(zi) - fi(z)J = <z1- z, L) t!Vf(z)> j =O i=O

Ill

+ 1 (Zi . a 2 fi ,.. z L) ?.! - cncz,-z)) '

' az 2 j=O

with~= eA. (zi)z; + [ 1-eA. (zi)lZ.

On the rate of convergence of certain methods of centers 233

If )... ~ 0 is chosen to be an optimal multiplier for Problem 1. I at z. ( 1 . 9) becomes

Ill

(l .10) L) t!J1(zi) + )...0 [J°(zi) - t°(z)l j= l

111 2 . i " 'I>' .af' " = 2 (Z; - Z, LI ?J - (n(zi-Z))

i=O az 2

Therefore, it follows from ( 1.10) and ( 1.6) that

( 1.11) )...0 [J°(z)-J'>(z)J 2!m0 )...0 llzi-211 2 .

Thus, by induction, (I. 7) and ( 1.1 1) imply that

llz1 - 211 2 ~ 2_ ki(J°(z0 ) -!°CZ)), mo

which completes our proof.

2. Rate of convergence of a theoretical method of centers

In this section we shall present two theorems. The first theorem shows that a Huard method of centers [SJ converges at least linearly. The second theorem shows that this method of centers co nverges at most linearly.

Here we shall consider the Problem 1.1 and Huard's algorithm under the following hypotheses.

2.1. Assumptions. We shall assume that there existsz0 EC= {zlf(z) < 0, j = I, ... , m}, such that

(i) the set {z E Cl!° (z) < t° (z0 ) } is compact and convex, ( ii) there exists a compact convex set C(z0 ) containing {z E Cl!°(z) <

t°(z0 )} in its interior such that Ji, j = I , ... , m, are convex in C(z0 ) and (iii) t° is strictly convex in C(z0 ).

(iv) C' = {zlfi(z) < 0, j = I, ... , m} is not empty.

2.2. Algorithm (Huard [SJ). Step 0. Select a z 0 such that 2.1 is satisfied and set i = 0.

'

234 0. Pironneau and£. Polak

Ste/J I. Compute a solution zi+t of

(2.3) 8(.:i) ~ min{d(:, .:: 1)1.: E C(z 0 )} ,

where

(2.4) d(.::,zi)~max{J°(:) .r°(::i);[i(z), j=l, ... ,m} .

Step 2. lf 8(zi) = 0, setz = Z; and stop; else, i;et i = i+ I and go to step 1.

2.5. Comment. Note that an alternative expression fo r 6(z), is given by

(2.6) 6(.:)= min {61f0 (z) - f 0 (:i) 8 ::; 0; (Z' /:i)

[i(z) - 8::; 0 ; j = I, ... , 111: .: E C(z 0 )} .

2.7. lelllma (dual conl'ergence). Suppose that {::i} f:o is a sequence con­structed by Algorithm 2.2 i11 so/Pi11g Problem I. I, which colll'erges to z, the unique solution of Problem I. I. ler 11 i E IR /11

+I be any solution of

the dual uf(2.6), i.e. of

(2.8)

Ill

n;ax (mi~1 {(I - 6 111) 8 + 11° [f0

(z) - J°Czi) ] u _ O (/:i,_) 1=0

zE:C(zol

Ill }

+ ~ uif(z)} .

Then a11.1· accu11111latio11 point of {u)j:0 belongs to the set

(2.9)

Ill

/\(:) ~{)\EIR. 111 +1 I6r/'Vfie)=0, j=O

Ill

6 r/ = I ; A. > 0; r/ fi (:) = 0. i = I , ... , m} .

1=0

On the rate of convergence of certain methods of centers 235

Lemma 2.7 becomes intu itively clear once it is shown that for suf­ficiently large, (11i, zi+ t ) is a solution of

(2.10) 8(zi) =max {11° [J°(z) - J°(zi)] (u, z)

Ill Ill m

+ 6 t/fi(z)l 6 z/ = l; u ~ O; 6 tii'V f(z) = O}. j= I j=O j=O

Since the proof of Lemma 2. 7 is quite complicated, we omit it. The interested reader wi ll find a proof in [ 12] , pp. 17-23 . Similar lemmas have been established for other methods of centers. (Sec [ 81, Theorem 5. I.)

2. 11. Theorem (Linear convergence at least). Let {zJ j:0 be an i11fi11ite sequence generated by Algorithm 2. 2, in solving Problem I. I , and sup­pose that assumptions 2. I are satisfied. Then, given any a E (0, I), there exists an integer i0 (a) such that

(2.12) !°(zi+l) - !°(z)::; [ l - x0 o - a)] [J°(zi) -J°(z) ]

fora// i?::. i0 (a),

where 2 is the unique solution of Problem 1.1 , and X0 is de.fined by

(2 .1 3)

Ill m

~o = min { 'Ao I 6 r/ 'V Ji (z) = 0 ; 6 r/ fi(z) = 0 ; j=O j= l

m

6 r/ = I; r! > 0 , i = 0, l, ... , m} . j=O

Proof To obtain (2.12) we shall make use of three facts. The first is that ~ (2.6), [O(z;+ 1 ) - fO(z1) < 8(zi). The second is that there exists a u (z) E [O, l J such that 8(zi)::; ll°(zi) [J°(z) - .r0czi)J, and the third is that for every a E [0, 1 ), there exists an integer i0 (a) such that u 0 (z) > X° (1-a) for all i ~ i0 (a). We now proceed to establish the last two facts.

Since the set {(z, 8)1.f°(z) - .r°Czi) - o < O;Ji(z) - 6 < 0, j = I, .. ., m}

-··

236 0. Pironneau and E. Polak

is not· emp ty for all zi E C(z 0 ), the strong duality theorem B.5 can be applied to (2.6). Parts (a)(i) and ( ii) of Theorem B.5 , as applied to (2.6),

imply that

(2. 14)

m m

8(.:i) = max {inf u ?: O zEC(zo)

liE" \R

{(1-p 1/) 8+ ~ l/f(z) 1=0 1-1

+ u0 [J°(z) - .r°(zi) l}}

Let ii(zi) = (i1-0 (zi), ... ,um (zi)) E Rm+! be a solution of (2.14) , then from

(a)(iii) of the strong duality theo rem B.5

(2. I 5) 8(.::) =min { 1 (li,z)

Ill m

~ lii(zi))o + ~ lii(z)Ji(z) j=O j= l

+ u0 (.:i)[ fo (:) f° (: i)] I: E C(z 0 )} .

Clearly, equa tion (2.15) cannot hold unless

(2. 16)

Ill

~ lii(zi) = 1 . j=O

Consequently,

(2.1 7)

Ill

o(z)= min { ~ ui(.:i)fi(z)+u0 (zi)[f-O(z) - J°(zi)]}. ; E C(;o) j= l

Upon replacing.: by z in (2.17), we obtain the following bound on o(zi):

(2. I 8)

Ill

8(:::)::; ~ ui(zi)Ji(z) + uO(z)[J°(z) -J°(z)] . j= I

S. ~ C ff(~)< 0 · = 1 · d ( ) > 0 "' 111 -i( )ff( ) < 0 mcc z E , z _ , / , ... , m, an u zi _ , £..;i=I u zi zi - .

Hence,

On the rate of convergence of certain methods of centers 237

(2. 19) <-0- r{)A rO _ o(zi) ll (t.)[1 - (z) J - (_i) ] .

This is the first inequality we set o ut to prove. Next, from Lemma 2. 7 and (2. I 3) we deduce that

(2.20) li1!1 infUo(.:i)> A,0. 1~00

Since by Lemma B. 11 , X-0 > 0, (2.20) implies that, given a ny ex E (0, I), there exists an i0 (ex) such that

(2 .2 1) l(O(zi)>A.0( 1 ex) forall i">.i0 (cx).

Combining (2.18) and (2 .2 1) we now obtain

(2.22) o(z) < A.0 ( I ex)[/-O(z) .f°(zi) ] for all i > i0 (ex) •

which is the second inequality we needed to prove. Finally. from 2.5 and (2.2 I ) we obtain

(2.23) f-O(zi+l) /-O(: i) < 8(.::)::; X0( I - ex) [ fl(~) / 1(:i)]

forall i>i0 (ex).

Rearra nging (2.23) we fi nal ly obtain

(2 .24) .r°(zi+l) -J0 (Z:) ::; [ 1 X0 (1 ex ) l l /-O(zi) - / 0 (z) J

for all (:> i0 (ex),

which com pletes o ur proof.

2.25. Corollary. Suppose that the assumptions of Theore111 2. I I and of lemma I. 5 are satisfied. Then { z i} f:o converges to z linear(11•

We now establish an upper bound on the rate of convergence of Algorithm 2.5.

2.26. Theorem. Let {zJ j:0 be an infinite sequence generated by Algo­rithm 2. 2 in sobiing Problem 1. 1. le I z be the solution of 1 . I and lf!f

'

238 0. Pironneau and E. Polak

'A.0 be defined by (2.13). If assumptions 2.1 are satisfied, then, either X° = I and tlze sequence { J°(zi)}j:0 conl'erges superlinearly to J°(z), or r..0 < I and there exists an integer i 1 such that

(2.27) flC.:;+1) - J°e);:::(1 - X0 HI°Cz;) J°(z) I

for all i ;::: i I ·

Proof Applying part (b)(ii) of the strong duality theorem B.5 to (2.6), we condude that a i/0 (.:i) defined as the first component of a solution

of (2.8) must satisfy

(2.28) i/0 (.:;)[J°(.:i+l) f 0 (.:i) o(z;)I = 0.

Now. according to (2.21). i!0 (z;);::: (I a)X0 for all i;::: i 0 (a). Therefore, by making use of Lemma B.11, u0 (:::;) > 0 for all i > i0 (a). Hence, from

(2.28). we obtain that

(2.29) /J<.:;+1) - !°<.:;) = o(.:i).

Next, according to (2.10), for i = 0, I, 2, ... ,

(2.30)

111 Ill

o(z;) = 111.c.LX {inf {(1 - _6 ui)o + _6 l/fi(z) 11.::;-0 z E C(zo) j=O j= l

/j t- IR

+ uo [fl(.:) .f°(z; J}} .

Setting 11 =A., some clement of J\(z) (defined by (2.9)), we conclude

that

(2.31)

Ill

o(z;)Z inf { _6 '!JJi(.:)+t..0 [J°(z) - fO(z;)]} .:E C<zo> /= 1

The infimum in (2.31) is achieved at z because "if'~o ">/V' f'<z) = 0. There­

fore

On the rate of convergence of certain methods of centers 239

Thus

(2.32)

Ill

o(z;);::: .6 Ai/(:)+ r..0 [J°(z) - J°(z;)J = r..0 [J°(z) f 0 (z;)l j= l

for all A. E J\(E) .

o(z;);:::X0 rt°(z) - J0 cz;)J, for i=O, 1,1 ....

Finally setting i 1 = i0 (a), for so me a E (0, I), (2.29) and (2.32) com­bine to give

(2.33) t 0 czi+l) - f 0 (z);::: ( 1 X0 )[J°(z;) - t°(z)] , for a ll i;::: i1 '

which proves the second part o r the theorem; the first part follows di­rect ly from Theorem 2.1 I. This completes our p roof.

Combining (2. 12) and (2.27), we see that whenever Algorithm 2.2 constructs an infinite sequence {.:;};':0 , we must have

(2.34) o ) 1·0 e) f (.:;+1 · - = 1 - 'A° · lim - -

;~~ fo(zi) - 10 (z)

Therefore the larger the value of X0 E (0, I ), the faster Algorithm 2.2 converges. When the unconstrained optimum of / 0 is in C(z0 ) , then X° = I and Algorithm 2.5 converges superlinearly (or in a finite number of steps). When X0 is small (which is the case when llV' J°(,:) II is large compared to llV'/(z)ll or when the Kuhn-Tucker constraint qualification is only marginally satisfied), Algorithm 2.2 will be slow.

(2.34) should be compared with a similar evaluation obtained by Faure and Huard [3] for the method of centers based on the distance functions.

(2.35)

Ill

d(z,z;) ~(- 1)111 [f0 (z)-J0 (z;) I n f'(z) . j=l

They showed that if {z;}j:0 is an infinite sequence generated by this method of ccn ters, then

'

240 0. Pironneau and t:. Polak

(2.36) !°(:i+1) - f0 (~) 111

lim - -- - = - -i~~ f 0 (:i) [°(2) m + 1

where nl is the number of active constraints at:. Therefore when Ill is large algorithm 2.5 is likely to be somewhat faster 1

: while the method of centers based on (2.35) will be faster than Algorithm 2.5 when the Kuhn-Tucker constraint qualification is only marginally satisfied. ln any event, as we sha ll sec, the major advantage of Algorithm 2.2 is that it can be implemented, without losing its rate of convergence, with less computational effort than the method of ce nters based on (2.35).

3. Modified method of centers

The method of centers 2.5 requires that at each iteration we solve

the following problem.

3.1. Problem. min max {fO(:) - fO(zi)Ji(z), j = I , ... , m} .

In a modified method of centers [6], this problem is replaced by two subproblems: a direction finding su!JtJroblem

(3.2) min {tz 0 l<V' /°(:i), h> < tz0 ;ji(zi) (lzo, fll

+<V'Jf(zi),h><h0 ,;= l , ... ,m;hES} ,

where S = {/z E IR 11 I I/ii I~ 1, i = l, 2, ... , 11 }. which y ields a solution (11° (.:), /i(:i)), followed by a step size determination subproblem

(3.3) 01z (zi) ~minmax{f0 (zi+ µh(zi)) - !° (zi);

Ji(:i + µh(zi)), j = 1, ... , m } ,

whose solution is µ(:i). The final result is Z;+J = zi + µ(zi)h(zi).

1 fl(z i+ 1 - fl(z)

Faure and Huard proposed a way out for decreasing lim - - -- . However :\lifflin /J(Z i) - fl(z)

showed [ 101 that the proposed mod ification makes the computation of an "€-center" much

more time consum ing.

On the rate of convergence of certain methods of cenrers 241

Referring to the example in the appendix, we see that th e a lgorithm using 3.2 and 3 .3 does not converge linearly. Therefore, we modified 3 .2 to obtain the following new algorithm.

3.4. Algorithm (modified method of centers). Step 0. Set i = 0. Step I. Compute a so lu tion (11°(::.i)h(z)) of

(3 .5) mi n{h0 +111'1 11 2 1< V' J°(zi),h>~ho;

[i(zi) + (\i' Ji(zi), h> < 1zO, j = l , ... , m} .

Comment. To find a solu tion o f (3.5), solve (B.19) and use (13. 22) (see Appendix B). Note that whenever n > 3111 and if Wolfe's algorithm is used [ 19] , (B. I 9) is easier to solve than 3. 2.

Step 2. lf h-O(zi) = 0, set z = zi, and stop; else, compute a solution (o,,(z), µ(zi)) or

(3.6) oh(zi) = min max{/0 (.:i + µh(zi)) - J°(:i); µ

Ji(zi + µh(zi)), j = I, ... , m} .

Step 3. Set zi+I = zi + µ(zi)h(z;) and go to step 1.

It is easy to show that under assumption 2. 1, Algorith m 3.4 converges in the same manner as 3.2, i.e. , either it stops at z or it generates an infi­nite sequence whose accumu lation points are solutions of Prob lem 1. 1. For a more general situation, the converge nce properties of Algorithm 3 .4 are given in [ I 3] .

We shall now prove that Algorithm 3.4 h as a better rate than algo­rithm 3 .2. Throughout this section, the fu nc ti on Ji, ; = 0, 1, ... , mare assumed to be convex 2 and twice continuously differentiable.

3. 7. Lemma. Suppose that assumptions 2. l (i)- ( iii) are satisfied. L et oh (z), (h0(z), h(z)), and z be defined as solutions of (3.6), (3.5) and Problem I. I respectively. Then, given E > 0 there exists a -y1 (E) > 0

2 Ac tually, the convexity of Ji, j = 0, 1, ... , 111, is needed only in a compact convex subse t oflR17

containing C(zo) in its interior. ll is on ly for the sake of simplicity that we assumed global convexity.

'

242 0. Pirom1eau and£. Polak

such that 811 (::::).:; iir 1 [h0 (z) +t ll/z(.::)11 2 1 for all:::: E B(z, -y 1 (E)) n C, with M defined by -

(3.8) a 2 !1

ill= max {l; II- (::::)II ,j = 0, I, ... , mlz E B(:,E)}. a:::2

Proof According to (3.6),

(3.9) 811 (z)= min {o,,1f0 (z+µh(z))-/0 (z).:;011 (/) 1z.µ)

Ji(z + µ/z(:::)).:; 81z. j = I , ... , m},

It is straightforward to show that the strong duality Theorem B.5 ap­plies to (3.9). Therefore. we obta in from Theorem B.5 (a)(ii), that, for

any::: EC.

111

(3.10) o,, (::::) = m~x {inf {(I - ~ wi) 811 w~O (01z,µ) 1=0

Ill

+ ~ wi[i(:+µlt(z))+w 0 iJ"(z+ µ/z{z)) -f0 (z) l}).

Ill

(3. 11 ) =m~x{inf {~ wifi(::::+ µh(z))+w0 [.f°(z+ µh(z)) w - 0 µ l= I

- !"(:)I} I ??. J " I ) .

Next. because of (3.8), the right hand side of (3.11) expand s as fol­

lows.

Ill

(3.12) ~ wifi(::::+µfz(z))+w0 [f0 (z+µh(z) - J°(z)) j= 1

111 Ill

.:; ~ wifi(z) + µ ~ wi<'V [i(z), h(z)> + ~ µ 2 M llh(z)ll 2 ,

j= I j=O

On the rate of convergence of certain methods of centers 243

for all w E IR. 111 + 1 such that w ~ 0, L,/~o wi = 1, and all (z, µ)such that z+ µh(z)E B(z,E). From Lemma B.1 6. (see append ix), there ex ists 'Yt (E) such that (3.12) is valid for al l w ~ 0, sa tisfy ing L,j'~ 0 wi = I. for all µ E [0, I] and for all z E B(z, -y 1 ) n C. Therefore . fo r any z E B(z, -y 1 ) n C, (3.10) and (3. l I) combine to give

111 111

(3 . I 3) 01z(z).:; max {inf { ~ wiji(z) + µ ~ wi<'Vfi(:::), h(z)> w ?: O µ E 10,l] j= I j=O

111 } +!µ2 Mll'7(:::) 11 2 } 1 ~ wi= J . 1=0

\

By settingµ= M- 1 in (3.13) (and deleting the inf operation) we obtain

(3.14)

Ill Ill

811 (z).:; max { ~ wifi(:::) + ~ 6 wi<'V [i(z), Ji(::::)> w ~O j= 1 i=O

1 Ill

+ 2M llh(z)ll 2 I ~ wi = I} . i=O

Since for z E C, fi(z).:; 0, j = I , ... , 111; and si nce M :;::.:_ I, it follows that [i(z).:;M- 1[i(z), j= l, ... ,111 . Therefore, for any zEB(5,-y 1 (t:))nC

(3 .15)

111 m

8h(z)< ~ m;x { ~ wiji(z)+ 6 wi<'VJi(z),h(z)> w~O 1=1 1= 0

m

+t ll/z(z)ll2 Ii~ wi = I}.

By definition, (h0 (z). /i(z)) is a solution of (3.5) with ::::i =:::.Therefore

<'VJ°(z), h(z)>.:; h 0 (z); and /(z) + <'V p·(::::), /i(z )) <ho(:::).

j= I , ... ,111.

Hence, from (3. 15),

'

244 0. Pironneau and E. Polak

m m

8;,(z)~ ~max { 6 wihO(z)+~ llh(z)ll 2 ! 6 wi= l} , w > 0 i=O - i=O

i.e., 8;, (.::) -:::= M 1 [!z0 (.::) + !- llh(z)ll 2 ), which proves the lemma.

3.16. Theorem. let {z;}f:,0 be a sequence generated by Algorithm 3.4 ill the process of so!Fing Problem 1.1, and suppose that assumptiOllS 2.2 alld those stated in lemma 1.5 are satisfied. Then, given any a E (0, I), there ex is ts an ill teger i 2 (a) such that for all i > i 2 (a),

(3.17) f-O(zi+I) J0 (z) ~ [I - ''k ["J;.:0(1-a)] 2] [J°(z;) - f 0(.z)) ,

where X0 . m0 . Mare as in (2.13), (1.6) and (3.8), respectiPely.

Proof Our strategy for establishing (3.17) will be as follows. From step 2 in Algorithm 3.4, we have

(3.18) J°(z,+ 1 ) - f0 (z;)~ 8,,(z;), i= 0, 1, 2, ...

We shall show that for every a E (0, l) there exists an integer i 1 (a) such that for all i > i 1 (a)

(3.19) 1zO (z i) + -t 11/z(z;) 11 2 ~ mO°XO (1 - a) 8(z;) ,

where 8(z 1) is as in (2.3). Inequality (3.17) will the n follow from (3.18), Lemma 3. 7, (3.19) and (2.22).

Since the bound m 0 in ( 1.6) is only good for a ball of radius e about ~' we shall have to restrict ourselves to this ball in establishing the rate of convergence. Now, for any z EIR n let

(3.20) r(z) = {z'IJ°(z') - f 0 (z) ~ 8(z);Ji(z') ~ o(z'), j = I , 2, ... , m}

Then, from the example in [ l] , p. 111 , it follows that r is an upper semi-continuous mapping. Since r(z) = {z} and r is u.s.c., it follows by definition, that there exists a "(2 E (0, e ] such that r(z) c B(z, e) for a.II z E B(z, "( 2 ). Consequently, for all .:: E B(z, "(2 ), we must have

(3.21) 8(z) =min {81j-O(z') - J°(z) ~ 8;/(z') ~ 8,j = I, ... , m; (Ii ,z')

z' E B(z, e)} ,

for all z.E B(z, -y2 ) n C.

On the rate of convergence of certain methods of centers 245

Upon applying the strong duality theorem B.5 to (3.21), we obtain

(3.22)

(3.23)

m

8(z)=m;x{i~f. { 6 vif(z')+v0 [f-O(z') u _ O zEB(z,c) i=l

oEIR

m

- f 0(z)] +(I - i~ vi) 8}}

m m

=max {inf { 6 vifi(z') + v0 [fO(z')- fO(z)]} I ~ vi= l} v>o z'EB(z,e) i=l 1=0

Therefore for all v ~ 0 such that "2;~0 vi= I and all z E B(z , "(2 ) n C,

Ill

(3.24) 8(z)~inf { 6 vi/(z')+v0 [J°(z') - !°(z)J}. z'EB(z,~e) i= I

Expanding f 0 (z') - J°(z) to second order terms and making use of the convexity of the functions Ji, j = l, ... , m, we obtain for all v ~ Osuch that Lj1~0 vi = I , and all z E B(i, "(2 ) n C,

m m

(3.25) 8(z) ~ inf { 6 vij1.(z) + 6 vi(''V /i(z), z' z> z' E:B(z,e) i= I i=O

+tm0 v0 llz' -z/1 2}.

By deleting the constraint z' E B(z, e) in (3.25), we obtain

(3.26)

Ill

8(z) :> 6 vi/(z) i= l

I m

-2-,n-ovo 11 6 Pi'V fi(z)/12 1=0 '

!:_or all v ~ 0 such that v0 > 0, L~0 11i = I an? ~II z E B(z, 'Y2 ). n ~'. Let v(z)EIR. 111

+1 be a solution of max {"2;j1~ 1 u1fl(z) - t /1"2;j1; 0 u1Vf(z)ll 2 1

ll 2' 0

'

246 0. Pironneau and E. Polak

"L7!0

µi = I}, which is equivalent to the dual o!:_.(3.5.) (see appendix (B.17)-(B.22)). Then. since 111°pO(.:) < 1 a~d "Lj'~ 1 P1(.:)/1(.:) ::;; 0, we must have. ~?~ 1 1•/(.:)j'l(.:) ~ (llZOl;Q(.:)) l ry~l l'l(z)j'l(.:); aSSL~ing that pO(.:)> 0. Hence. from (3.26) and (B.22) (with 11 replaced by 1')

(3.27) -) > -o(_ - 01 ::v, c-)

Ill -

m m ]

l~ J.l°c.:)/(.:) - } II ~ J.i(z) V' fi(z)ll 2

j= 1 j=O

['1°(7) + l mOpO(z) - 2 1117 (.::)1121

By following the same pattern of reasoning as in the theoretical me thod of cente rs. for Lemma 2. 7. it is easy to prove that

(3.28) lim inf J.0 (.:) 2 A.o • ' :-:

:EC

where XV is defined by (2. 13) and , according to Lemma B.11 satisfies A.-o > 0. It now follows from (3.28) that. given any a E (0 , 1 ), there exists ,,3 (a) E (0. )'2 ) such that J.0(.:) > X0 c 1 - a)> 0. for all.: E 8(3, 'Y:i(a))nC. Therefore (3.27) implies that o(.::)> (111o~o (l -a)) 1 [ho(.::)+ }°llh(.:)11 2 ). for all.: EB(:. 'Y3 (a)) n C. ll ence by making use of Lemma 3. 7, we conclude that, given any a E (0, 1 ). the re exists an integer i 1 (a)

such that

(3.29) 1

f-O(zi+J)-f0 (zi)<o11 (zi)::;; M(h 0 (z)+1 llh(z;)ll 2)

o-o < !.!..!.._.!::._ ( 1 - a) s: 7

.)

- M u(-, ,

for all i ~ i 1 (a). Finally. from (2.22). there exists an integer i 2(a)2 max(i1 (a). i0 (a)), such that

(3.30) ~(Zi+I) - ~(z)::;; moxo2 ( l -a)2 [~(zi)- ~(z )] ,

for all i 2 i 2 (a) .

Rearranging (3.30), we obtain (3. 17), which completes our proof.

On th e rate of convergence of certain methods of centers 247

3.31 Corollary. Suppose that the assumptions of Theorem B. 5 are sa­tisfied, then {zi}f:o con11erges to z linearly.

3.32 R emark. Note that if Algorithm 3.4 is used for solving an uncon­strained problem, Algorithm 3.4 reduces to the method of steepest descent. For this special case, X0 = 1 and Theorem 3.16 agrees with well known results for the method of steepest qescent.

To implement Algorithm 2. 2, Huard [ 6] proposed to accept as a successor to z; , any point zi+l satisfying

(3.33) o(z;)::;; d(zi+I > Z;)::;; p o(z;),

where o and d arc as in (2.3) and (2.4) and p E (0, 1). To compute such a Z;+ 1 , one can use k; iterations of Algorithm 3.4 on (2.6) 3

. We initialize the computation by setting z = Z;· Tracing through the proof of Theorem 3.16, we find that this theorem can be adapted to cover also this particular case, to give, for i sufficiently large,

(3.34) 0 ~ d(zi+l > Z;) - o(z;)

[ XOmO ~ k;

~ 1 - 4M (I -a) J (d(z;, Z;) - o(z;)),

where according to (2.4), d(z;. t;) = 0. Since k; in (3.34) is the smallest integer for which (3.33) holds, we must have 4

(3.35) r, - o 0

k; ~ log(l - p) Llog (t A. r:,~l -a)2)] -I

By inspection of (2 .22) and (3.34 ), the constant of ex ponentiation of the rate of convergence of this implementation which constru cts z i+ 1 to satisfy (3.33) is (I - p""f...O). Since one iteration of the implementation is

3 This suggestion is very close to lluard's which was to use a finite nu mber of iterations of a "partial" algorithms [ 6]. Unfortunately we do not have a:iy bound on the rate of convergence for this partial algorithm.

4 Note that this resu lt is much better than the one obtained by Mifflin [ 10] for the method of center based on (2.33) and implemented by k; iteration of the method of steepest descent to find z i+ 1 sati sfying a relatio n similar to (3 .34 ).

'

0. Pironneau and E. Polak 248

as dif_!:.icult to perform as ki iterations of _!..~orithm 3.4, the f.onstant 1 - pt...0 should be compared with [1-(f...O mO(l - a)2 /M)1 '.From (3.35), we can deduce that l pXO > [l - (X0

2m0(1-a)2/M)lk;, for

all p 6 (0, 1 ). This shows that there is no advantage in choosing a p

which will result in k i > l. Next we note that (3.6) also cannot be solved exactly in a finite

number of iterations of any of the applicable algorithms. However, an approximation to a solution of 3.3, compatible with preservation of linear convergence , can be found by means of the subprocedure defined in step 2', below, which is a cross between a procedure due to Arnijo { 141 and the Golden Section Search. As we shall see shortly, in solving (3.6) this subprocedure is highly efficient and, in all likelyhood, out performs such well known methods as the Davidon cubic interpolation

scheme.

3.36. Step 2'. If i = 0, select a i3 > 0 and go to(*); else go to(*) . (*)if 11°(z ;) = 0, set z = z; and stop; else define e;: IR-+ IR by

(3.37) 8;(µ) = max{.r°Cz, + µh(z;)) - f 0 (zi);Ji(z; + µh(z;)),

j = l , ... , 111} .

Se t l; = {j le(l) = Ji (z ; + h(::; ;)) - o(j)fi(z;), j = l , ... , m}. (o(j) = 1 if j = 0, 8(j) = 0 , otherwise, and go to(**).

(**)Compute

de;(t+) . - d - =max {<V' f (z; + µh(z)), h(z;)>}

µ jE=l;

If de ;Idµ( 1 +) S 0, set µ(z ;) = 1; else use the Golden Section Search with initial interval [0, 11 to find two points µ(z;) < µ'(z;) such that

(3.38) O(µ(z; )) S J3[µ'(z;) - µ(z;)l < V' f° (z;), h(z1)).

3.39. Theorem 5 . Let P; be the number of evaluations of e;(µ) which

5 The process of replacing the computation of a minimizerµ of a convex function e: (0, + 00

)-+ (-

00, 01 by the computation of two pointsµ, µ' such that 0 < µ S µ S µ ' and such that

0(µ) < {3(µ' - µ) ~ (0 +) dµ

constitutes a general procedure for implementing algorithms of the type of (3 .9), without loss

of linear convergence.

On the rate of convergence of certain methods of centers 249

are required to find (µ(z;), µ'(z;)) in step 2', giPe11 z;. Then the conclu­sion of Theorem 3.16 remains valid wizen step 2 of Algorithm 3.4 is replaced by ste/J 2', provided (3. 17) is replaced by

(3.40) f°Cz;+i) - J0 (z) s [l -~X0 (1-a)J 2 _J3_ [/>(z.) M l+J3 I

t°Cz)J .

Furthermore, for all i = 0, l, 2, ...

(3 .41) P; S 1 + 6 log [ 1.36 L (1 + /3)] .

The proof of Theorem 3.39 is rather involved and we therefore omit it. The interested reader will find it in [ 13 J.

Note that when l = I, if we choose J3 = I 0 in step 2', the rate of con­vergence of the implementation of (3.9) using step 2' is about the same as that of Algorithm 3.4 and we are guaranteed to find a step size µ(zi)

with at most 7 evaluation of 8;(µ). Th is is quite reasonable and it is doubtful that any other method would find a step size µ(z;) satisfying (3.35) with fewer evaluation of e;(µ).

4. Conclusion

Thus, we have presented three results in this paper. We have shown that a theoretical method of centers due to Huard [ 5] converges linear­ly; that an implementation of this method converges sublinearly , and that a new modified method of centers, which we have proposed , con­verges linearly. This new method not only converges linearly, but it also has the nice feature that its direction finding subproccdure is essentially insensitive to the dimension of the space on which the optimization problem is defined. Because of this, it was possible to adapt it for the solution of optimal control problems (which are problems in L

2 n L~)

[ 13]. In actual computations, the new modified method of centers can also be combined with standard interpolation schemes which utilize the fact that the successive centers constructed by a theoretical method of centers lie on an analytic curve. However, it is not possible to make any theoretical statements as to the rate of convergence of such a hybrid scheme.

'

250 0. Pironneau and E. Polak

Appendix A

Counter example to the linear convergence of the modified method of centers in [6] and of the Topkis-Yeinnott method of feasible direc­tions [18].

I. Topkis-Veinnott method of feasible directions [ 18] Consider the following problem.

A. I. Problem. min {x2 + y 2 ly ~ O} .

Let z0

= (x0 , Yo) be ~uch that 0 < lx0 I~ Yo ~ t· To solve a problem of the type min {J°(z)lf'(z) ~ 0, j = l, ... , m} the Topkis-Veinnott meth­od of feasible directions, computes a feasible direction h(zi) at zi as a solution of min {max{<'vt°(zi), h>; fi(z;) + <'V/i(z), h>, j = l, ... ,m}}

lhkl " l

which, for J°(z) = x 2 + y 2, f 1 (z) = - y, m = 1, becomes, at z 0 ,

(A.2)

Hence

(A.3)

min {max{ 2x0 h 1 + 2y0h 2 , - h2 - y 0}). - l:Shl:S1

- l:Sh2 :Sl

Yo - 21x0 I hl(z;)=-sgnxo; h2(z;)=- 1+2yo

The step size µ(z;) is computed as the solution of min J°(zi + µ h(z;))I f(z; + µ h(z;)) ~ 0, j = 1, ... , m . Therefore µ(z 0) is found as the solu­tion of

{ [ Yo - 21x0 I J 2

(A.4) mJn (x0 - µ sgn Xo)2 + Yo - µ 1 + 2 Vn Iyo

y 0 - 21x0 1 }

- µ >o '+ 2yo - .

From (A.3) and (A.4), (see fig. 1), it is clear that the constraint in (A.4) is not active at µ(z 0). Therefore µ(z 0) satisfies

On the rate of convergence of certain methods of centers

y

z i-1

x2+ y2 = y1-!

I I •

I

z

Fig. l.

I

I I I

I I

25 1

x

(A.5) Yo - 21x0 1 ( y0 -2L~0 1)

(lxol - µ(zo))+ I 2 .l'o -µ(zo) l 2 =O. + Yo + Yo

Let z 1 = (x 1 ,y 1 ) be the next point computed by the algorithm. Then from (A.3),

(A.6) Yo - 2 lx0 I

x 1 =x0 - µ(z 0)sgnx0 ;y 1 =y0 - µ(z 0) 1

_,_? -Yo

Hence, from (A.5) and the assumption on x 0 , Yo,

(A.7) 1~1= (y 0 - 21x0 1) Yi 1+2y 0

which implies that

(A.8) ,~,. ~Yo· I Yi

=Yo

lx0 I 1 - 2 -

Yo

l + 2Yo

Let {z;} be the sequence generated by the Topkis-Yeinnott method of feasible direction, in solving problem A. I, and startin g from z0 = (x0 ,y0)

~52 0. Pironneau and E. Polak

;uch that O< lx0

1 '.Sy0

:S1 . At each iteration the algorithm decreases the cost, therefore if Yo is chosen such that Yo ~ t, by making repeated use of (A.8), we obtain 0 < lx;I ~ Y; ~ t for all i >- l. From (A.3) it fol­lows that h2 (z;) :S 0 for all i >- 1. Hence (see fig. 1)

(A.9)

where

ll.:;+1 - z 11

llz;-zll =cos 'Y; ,

(A. I 0) 'Yi = 1T - P;+ l - P; .

From (A.8), tan P;+i = Y;+i /x;+J >- 1/Y;· Hence, 13;-+ 1-rr as i-+ 00

, and from (A.10), 'Y;-+O as i-+ 00 . Therefore, from (A.9), lim llz;+ 1 -zll ·

i~ 00

· llz; - .fn- 1 = 1, which proves that z; cannot converge linearly.

2. Modified method of centers Huard's modified method of centers [ 61 has the same direction de-

termination subproblem as the Topkis-Veinnott algorithms just des­

cribed, but the step size µ(z;) is given by

min {max{f°Cz; + µl!(z;)) -J°(zi);/(zi + µh(z;)) , µ

j= J, ... ,m}}.

which for Problem A. I becomes, at z0 ,

. f ( - 2 (. Yo - 21xol) 2 (A.II) mJn\max('l:o - µsgn.\o) + .1 0 - µ 1+2y

0

I' _ .Vo - ~~}}. · o µ I + 2.1 ·0

Consequently, µ(z0 ) satisfies either (A.5) or

(A.12)

y 0 -2ix0 1) 2 Yo -21x0 1

(lxo I - µ)2 + (l'o - µ 1 + 2Yo =Yo - µ 1 + 2yo

On the rate of convergence of certain methods of centers 253

Therefore, the pointz 1 =(x 1 ,y 1 ) computed by the Huard modified method of centers, from z

0, is given by (A.6) with µ(z 1) > 0 being the

smallest strictly positive number which satisfies either (A. 5) or (A.12). Note that if µ(z 1 ) is given by (A.12) then (x 1 , y 1 ) satisfies xy +YI = Y 1 · Hence it is clear (see fig. 1) that if (x0 ,y0 ) is close enough to (0, 0), X will never be given by (A. 12), which implies that, for Problem A. l, and starting from z0 such that 0 < lx0 I~ Yo ~ E, the Topkis-Veinnott and t~e Huard algorithms compute the same sequence of points and hence neither of those two algorithms can converge linearly .

Appendix B. Miscellaneous results

l. Duality theory

Consider the problem

(B. l) min {go (z)lz E n} ,

with

(B.2) n={zEIR. 11 lgi(z)~O, j= 1, ... ,p; zEC},

where gi: C-+ IR, j = 0, 1, ... , p, are convex and continuously differen­tiable functions, and C is a convex subset oflR 11

• The following problem is called the dual of (B. l ).

B.3. Problem. max {inf{g0 (z)+~f= 1 u igi(z)}} (u=(u 1 ,u2 , ... ,uP)EIR.P). u 2: 0 zEC

Let¢ :IR.P-+ fR U {- 00 } , be defined b y,

p

(B.4) ¢(u) =inf {g0 (z) + ~ uigi(z)} . zEC j= I

Definition. Any z En, satisfying go(z) = min{g0 (z)lz En}, will be cal­led a solution of (B. l). Any ti 2: 0, satisfying, ¢(i'1) =max ¢(u), will be called a solution of Problem B.3. u2:o

B.5. Strong duality theorem. Let SP be the set of all solutions of(B. l), i.e.,

'

254 0. Pironneau and E. Polak

(B.6) SP={:' E .Qlg0 (z') =min g0 (z)} . : E n

Suppose that S11 is TZot empty and that

(B.7) .Q' = { z I gi (z) < 0, j = 1, .. ., p}

is TZOt empty. Then (a) (i) Problem B.3 has at least one solution;

p

(B.8) (ii) max {inf {g0 (z) + 6 uigi(z)}} =min {g0 (z)lgi(z)::::; 0 ,

(B.9)

11 >o zE C j= 1 zE C

j=l. .... p};

(iii) for any u. a solution of Problem B.3, and 1or any z, a solution of (B. I).

p

g0 <.:) =min {g0 (.:) + E uigi(z)}: : E-C j = I

(b) a 1•ectf!f Ii ~ 0 i11 IR P is a solution of Problem B.3 if and only if there exists a: in .Q such that

p p

(B.10) (i) gO(z) + E uigi(i) = min{g0 (z) + E uigi(z)}; j= I zE C j= I

(ii) uigi(~) = 0, j = l, ... , p.

This theorem is a particular case of the strong duality theorem stated in Geoffrion [ 41 . Related theorems can be found in Rockafellar [I 5], Mangasarian [9].

2. A property of optimal multipliers

B. 1 1. Lemma. If assumptions 2.2 are satisfied then

(B.12) x0 =min {<A., e>I A. E J\(z)} > 0,

where e = ( l, 0, 0, ... , 0) E IR. 111 +1 and J\(z) is as in (2.9).

On the rate of convergence of certain methods of centers 255

Proof From (2.9) and the fact that z is a solution of Problem 1.1 , J\(z) is a nonempty [2] compact subset of IR m +l and therefore there exists a XE J\(z) such that X0 = <X, e>. Since the functions/i, j = 0, 1, ... , m are convex and continuously differentiable, we must have

(B.13) fi(z)?.fi(z)+<Vf(z),z - z>, j=O, 1, ... ,m,forallzEIRn.

Multiplying (B.13) by 'Al, for j = 0, 1, ... , m, and adding the results, we obtain

m m m

(B.14) E 'i!f(z)?. E 'i/fi(z)+ <E 'i!Vf(z),z - z>. j=O i=O j=O

Thus, it follows from (B.12) and (2.9) that

(B.15)

m

E 'f:!ji(z)?. X° t° (z) . for all z E IR n .

j=O

Hence, if X0 = 0, (B.15) contradicts assumption 2.2(iv). Consequently the lemma must be true.

3. Justification of (3.12)

B. I 6. Lemma. There exists a -y1 (e) such that z + µh(z) E B(z, 'Yi) for all µ E [O, 1 J and all z E B(z, 'Yi (e)).

Proof It follows from the strong duality theorem B.5, as applied to (3.15), that

(B.17)

(B.18)

h0(z) +-t llh(z)ll 2 =max {inf

u~o (ho,h)

m m

m

{O-Eui) ho j=O

+ i~ uifi(z) + 2o ui <V fi(z), h> +-!- llh11 2}}

m m m } =m;x finf{ ~ uifi(z)+ ~ ui<V'fi(z),h>+-!-llh11 2 } I~ ui=J ,

u_O , h l=l 1=0 1=0

'

256 0. Pironneau and E. Polak

and hence that

(B. 19)

Therefore

(B.20)

111 Ill

1i0 (:) +-~ llh(z)ll 2 =max{ 6 uifi(z) + 6 ui <V' f(z), h> 11:>0 j=I j=O

111

+t111111 2 1 6 11i= j=O

Ill

m

6 ui V' Ji(z) + lz = O}. i=O

11°(:) +t 1117(:)112 =max{ 6 uifi(z) - 11?°0 j=I

Ill 111

-111 6 uiV'f (:)11 21 6 ui = 1} .

j=O i=O

Let u E 1R111+1 be a solution of (B.20), then

Ill Ill

(B.2 1) 1zO (:) + t ll/z(z)ll2 6 [iij(z) - -!- II 6 [ii yi/(z)ll2

j= I

and from (B. 19) it follows that

(B.22)

Ill

lz(z) = - 6 [iiyi f (z) 6

j=O

Hence, if: E C,

(B.23)

Ill

11°(z) + lllz(:)ll2 = 6 uifi(z) ~ 0, j=I

j=O

6 This condition is stronger than the one used by Geoffrion [ 4] .

On the rate of convergence of certain methods of cen/ers 257

and therefore ! lllz(z)ll 2 < - [h0 (z) +! llh(z)ll2]. From (B.20), it is clear that 11°( ·) + t llh( ·) 11 2 is continuous, negative in C and that it takes the value zero at-z. Therefore there exists -y 1(E)E (O,! €] such that max { [h 0 (z) +1 llh(z) ll 2 J lz E B(z, 'Yi) n C } ~ i €

2. Hence, it follows that

(3.24) llh(z)ll~!Eforall zEB(z1 ,-y1 )nC .

References

[I ] C. Berge, Topological spaces (MacMillan, New York, 1963). [ 2] M. Canon, C. Cullum and E. Polak, Theory of optimal control and mathematics program­

ming (McGraw-Hill Co., New York, 1970) Theorem 19, p. 4 and Lemma I, p. 68. [3] P. Faure and P. lluard, "Resultats nouveaux relatif a la methode des centres," Paper pre­

sented at the Fourth International Conference on Operations Research, Boston, 1966. [4] A. Geoffrion, "Duality in nonlinear programming," SIAM Review 13 (1971) 1. [51 P. Huard, "The method of centers," in: Nonlinear programming, Ed. J. Abadie (North­

Holland Publ. Co., Amsterdam, 1967). [6 ] P. Huard, "Programmation mathematique convex," RIRO, R 7 (1968) 43 -59. (71 H.W. Kuhn and A.W. Tucker, "Nonlinear programming," in: Proc. second berkeley sym·

posh1111 math. stat. probability (University of California Press, Berkeley, Californ ia, 1951) pp. 481 - 492.

[ 8] F.A. Lootsma, "Constrained optimization via parameter free penalty functions," Philips Research R eports 23 (1968) 424-437 and suppl. 3 (1970).

[91 0. Mangasarian, Nonlinear programming (McGraw-Hill Co., New York, 1969). [ l O] R. Mifflin, "Convergence rates for a method of center algorithm," Thesis Operation

Research Center, University of California, Berkeley, 1971. [ 111 J. Ortega and W. Rheinboldt, Iterative solution of nonlinear equations in several variables

(Academic Press, New York, 1970). [ 121 0. Pironneau and F. Polak, "On the rate of convergence of certain method of centers."

Memorandum No. ERL-M296. University of California, Berkeley 1971. [13 1 0. Pironneau and E. Polak, "A dua l method for optimal control problems with initial and

final boundary constraints," Memorandum No. ERL-M299, University of California, Ber­keley 1971.

[ 14 I E. Polak, Computalional methods of optimization (Academic Press, New York, 1971). [ 15 I R.T. Rockafellar, Convex analysis (Princeton University Press, Princeton, New Jersey,

1970). [ 16 J D. Topkis and A. Veinnott, "On the convergence of some feasible direction a lgorithms for

nonlinear programming," SIAM Journal on Control 5 (2) (1967) 268- 79. 117 I R. Tremolieres, "La methode des centres a lroncature variable," These, Paris, 1968. ( 18] P. Wolfe, "A duality theorem for nonlinear programming," Quarterly of Applied Mathe­

matics (1968) 239-244. (191 P. Wolfe, "The simplex method for quadratic programming," Econometrica 27 (1959)

382-397. (20] G. Zoutendijk, Method of feasible direction (Elsevier Publish ing Co., Amsterdam, 1960).

'