on the rate of convergence of certain methods of … · select a z0 such that 2.1 is satisfied and...
TRANSCRIPT
Marhe111atical Programming 2 (1972) 230 257 North-Holland PublishinKCompany
ON THE RATE OF CONVERGENCE OF CERTAIN METHODS OF CENTERS*
0. PIRONNEAU and E. POLAK U11i11ersiry o[Califomia. Berkeley. California, U.S.A.
Received 22 March 1971 Revised manuscript received 13 September 197 1
It is ~hown in this paper that a theoretical method of centers, introduced by Huard, converges linearly. It is also shown, by counter-example, that a modified me thod of centers due to I luard and a method of feas ible direction due to Topkis and Veino! canno t converge linearly even under convexity assumpt ions. Because of this, a new modified method of centers is introduced wh ich uses a quadratic programming direct ion finding su broutine. In most uses this new method 1s not more complicated than ll uard's modified method of centers. But it does converge linearly. A method for implementing it without Joss of rate of convergence is also discussed.
0. Introduction
The family of optimization algorithms known as methods of centers were introduced by Huard [ 4]. They d iffer from one another only in the distance function used to establish a cen ter at each iteration. In their o riginal form these algorithms were of a theoretical nature, and because of this, various implem entations, or modified methods of centers, have been proposed. The best known of these modified methods of ce nters is also due to Huard [6 ] (These algorithms are also di scussed in l 14] ). In this paper we shall show that a theore tica l me thod of centers, presented in [5 ] , and a new modified method of centers converge linea rly on a class of problems.
The new modified method of centers to be discussed in this paper is a variation of the algorithm described in [6] Although we shall not establish the rate of convergence of the algorithm in [6 ] in this paper, we wish to mention that we were able to show that the algorithm in [6]
*Research sponsored by the Joint Services Electronics Program, Grant AF-AFOSR-68-1488 and the National Aeronautics and Space Administration, Grant NGL-05-003-016.
'
On the rate of convergence of certain methods of centers 231
converges at least as fast as I /0 under the same assumptions und er which our algorithm converges linearly. Furthermore, the example in th e appendix indicates that the algorithm in [6] cannot converge linearly under the assumptions used in this paper.
After a few preliminary results in sec tion I, we shall obtain. in section 2, the rate of convergence of a theore tical method of centers based on a distance fun ction defined by formula (7) in l5.l. In sec tion 3, we shall describe a new modified method of centers and we shall establish a bound on its rate of convergence. In addition we shall desc ribe an e fficien t imple mentation for this new method.
As we shall see. our analysis depe nds heavily on an ex tension of Wolfe's duality theorem [ 18] , l 9]. This ex tension is due to G eoffrion [5]. and is restated for the 1;,ake of convenience in Appendix B. It may be interesti ng to observe that results closely related to duality theory have also been used by Faure and Huard f 31, Tremolieres [ 17], Lootsma [8] and Mifnin [JO] in the study of convergen ce and rate of convergence of methods of centers based on a distance fun c tion defined by formula (6) in (5].
Since we shall be exclusively interested in rate of convergence, we sha ll assume that the read er is familiar with methods of ce nters and their ' convergence properties. In any event, th e read er will find these described in con siderable de tail in [ 14] .
I . Preliminaries
We shall consider in this paper algorithms which solve the following problem.
I. I. Problem. min {.t°(z)lp°(z) ~ 0, j = 1, 2, .. ., m}, where Ji: IR"_. IR 1
for j = 0, I , 2, ... , mare continuously differentiable.
We shall assume that Problem I. I has at least one solution ~. We shall show tha t a number of these algorithms are R-linearly converge nt (see [ 11], p. 29 1). For our purposes, it suffices to recall that if a sequence {Z ;} f:o in IR /1 co nverges to a point z and there exist a k E (0. I) and a Q > 0 such that
( 1.2) 11.:; - Zll~Qk; fo r i=O, 1,2, ...
then {z;} co nverges to z, R-linearly.
232 0. Pironneau and E. Polak
Given {zi}i':'o constructed by any one of the algorithms to be considered, we shall begin by showing that there exists an l E (0, 1) such that
( 1.3) J°(::1+
1) - J°(z) ~ l[J'>(zi) f° (z)] for i=O, 1, 2, ....
Obviously, ( 1.3) implies that
( 1.4) !° (z i) - !° (z) < Li[!° (z 0 ) !°Cz)L ; = o, 1, 2, ...
We shall then make use of the lemma below, which requires somewhat
stronger assumptions, to deduce ( l. 2) from ( 1.4 ).
l .S. Lemma. Suppose that (i) the fimctions Ji, j = 0, 1, ... , m, are twice collti11uously differentiable and there exist constants€> 0, m
0 E (0, l]
such that
a2!°(z) ,. ( 1.6) m 0 llyll2 ~ y, y for ally ElR. 11
, for all z E B(z, €), az 2
where z is a solution of ( 1.1) and B(z. €) = {zl nz- zll ~ E} ; (ii) the KuhnTucker constraint qualification is satisfied al z; (iii) {zi} i;O is a sequence in C~ {zlf(z)~ 0,j= 1, 2, ... , m}. which converges to z and satisfies
(1.7) J°(z 1 ) - !°Cz)~ /i[J°(z 0 ) - J°(Z)I for i= 0, 1, 2, ...
Thell there exists an integer i0 such that
( 1.8) llz. - 2112 ~ ki io 2 [fo(zi ) - fo(z)] for all i ?-_ io . ' mo o
Proof Since {z;} converges to:, there exists an i0 such that zi E B(z, E)
for all i ~ i0
. Without loss of generality, we may assume that i0 = 0. According to the Taylor expansion formula, for any zi E {z1} and any
)... E IR. 111 + l, there exists a 8 A. (z) E (0, l) such that
Ill 111
(1.9) L) t./[f(zi) - fi(z)J = <z1- z, L) t!Vf(z)> j =O i=O
Ill
+ 1 (Zi . a 2 fi ,.. z L) ?.! - cncz,-z)) '
' az 2 j=O
with~= eA. (zi)z; + [ 1-eA. (zi)lZ.
On the rate of convergence of certain methods of centers 233
If )... ~ 0 is chosen to be an optimal multiplier for Problem 1. I at z. ( 1 . 9) becomes
Ill
(l .10) L) t!J1(zi) + )...0 [J°(zi) - t°(z)l j= l
111 2 . i " 'I>' .af' " = 2 (Z; - Z, LI ?J - (n(zi-Z))
i=O az 2
Therefore, it follows from ( 1.10) and ( 1.6) that
( 1.11) )...0 [J°(z)-J'>(z)J 2!m0 )...0 llzi-211 2 .
Thus, by induction, (I. 7) and ( 1.1 1) imply that
llz1 - 211 2 ~ 2_ ki(J°(z0 ) -!°CZ)), mo
which completes our proof.
2. Rate of convergence of a theoretical method of centers
In this section we shall present two theorems. The first theorem shows that a Huard method of centers [SJ converges at least linearly. The second theorem shows that this method of centers co nverges at most linearly.
Here we shall consider the Problem 1.1 and Huard's algorithm under the following hypotheses.
2.1. Assumptions. We shall assume that there existsz0 EC= {zlf(z) < 0, j = I, ... , m}, such that
(i) the set {z E Cl!° (z) < t° (z0 ) } is compact and convex, ( ii) there exists a compact convex set C(z0 ) containing {z E Cl!°(z) <
t°(z0 )} in its interior such that Ji, j = I , ... , m, are convex in C(z0 ) and (iii) t° is strictly convex in C(z0 ).
(iv) C' = {zlfi(z) < 0, j = I, ... , m} is not empty.
2.2. Algorithm (Huard [SJ). Step 0. Select a z 0 such that 2.1 is satisfied and set i = 0.
'
234 0. Pironneau and£. Polak
Ste/J I. Compute a solution zi+t of
(2.3) 8(.:i) ~ min{d(:, .:: 1)1.: E C(z 0 )} ,
where
(2.4) d(.::,zi)~max{J°(:) .r°(::i);[i(z), j=l, ... ,m} .
Step 2. lf 8(zi) = 0, setz = Z; and stop; else, i;et i = i+ I and go to step 1.
2.5. Comment. Note that an alternative expression fo r 6(z), is given by
(2.6) 6(.:)= min {61f0 (z) - f 0 (:i) 8 ::; 0; (Z' /:i)
[i(z) - 8::; 0 ; j = I, ... , 111: .: E C(z 0 )} .
2.7. lelllma (dual conl'ergence). Suppose that {::i} f:o is a sequence constructed by Algorithm 2.2 i11 so/Pi11g Problem I. I, which colll'erges to z, the unique solution of Problem I. I. ler 11 i E IR /11
+I be any solution of
the dual uf(2.6), i.e. of
(2.8)
Ill
n;ax (mi~1 {(I - 6 111) 8 + 11° [f0
(z) - J°Czi) ] u _ O (/:i,_) 1=0
zE:C(zol
Ill }
+ ~ uif(z)} .
Then a11.1· accu11111latio11 point of {u)j:0 belongs to the set
(2.9)
Ill
/\(:) ~{)\EIR. 111 +1 I6r/'Vfie)=0, j=O
Ill
6 r/ = I ; A. > 0; r/ fi (:) = 0. i = I , ... , m} .
1=0
On the rate of convergence of certain methods of centers 235
Lemma 2.7 becomes intu itively clear once it is shown that for sufficiently large, (11i, zi+ t ) is a solution of
(2.10) 8(zi) =max {11° [J°(z) - J°(zi)] (u, z)
Ill Ill m
+ 6 t/fi(z)l 6 z/ = l; u ~ O; 6 tii'V f(z) = O}. j= I j=O j=O
Since the proof of Lemma 2. 7 is quite complicated, we omit it. The interested reader wi ll find a proof in [ 12] , pp. 17-23 . Similar lemmas have been established for other methods of centers. (Sec [ 81, Theorem 5. I.)
2. 11. Theorem (Linear convergence at least). Let {zJ j:0 be an i11fi11ite sequence generated by Algorithm 2. 2, in solving Problem I. I , and suppose that assumptions 2. I are satisfied. Then, given any a E (0, I), there exists an integer i0 (a) such that
(2.12) !°(zi+l) - !°(z)::; [ l - x0 o - a)] [J°(zi) -J°(z) ]
fora// i?::. i0 (a),
where 2 is the unique solution of Problem 1.1 , and X0 is de.fined by
(2 .1 3)
Ill m
~o = min { 'Ao I 6 r/ 'V Ji (z) = 0 ; 6 r/ fi(z) = 0 ; j=O j= l
m
6 r/ = I; r! > 0 , i = 0, l, ... , m} . j=O
Proof To obtain (2.12) we shall make use of three facts. The first is that ~ (2.6), [O(z;+ 1 ) - fO(z1) < 8(zi). The second is that there exists a u (z) E [O, l J such that 8(zi)::; ll°(zi) [J°(z) - .r0czi)J, and the third is that for every a E [0, 1 ), there exists an integer i0 (a) such that u 0 (z) > X° (1-a) for all i ~ i0 (a). We now proceed to establish the last two facts.
Since the set {(z, 8)1.f°(z) - .r°Czi) - o < O;Ji(z) - 6 < 0, j = I, .. ., m}
-··
236 0. Pironneau and E. Polak
is not· emp ty for all zi E C(z 0 ), the strong duality theorem B.5 can be applied to (2.6). Parts (a)(i) and ( ii) of Theorem B.5 , as applied to (2.6),
imply that
(2. 14)
m m
8(.:i) = max {inf u ?: O zEC(zo)
liE" \R
{(1-p 1/) 8+ ~ l/f(z) 1=0 1-1
+ u0 [J°(z) - .r°(zi) l}}
Let ii(zi) = (i1-0 (zi), ... ,um (zi)) E Rm+! be a solution of (2.14) , then from
(a)(iii) of the strong duality theo rem B.5
(2. I 5) 8(.::) =min { 1 (li,z)
Ill m
~ lii(zi))o + ~ lii(z)Ji(z) j=O j= l
+ u0 (.:i)[ fo (:) f° (: i)] I: E C(z 0 )} .
Clearly, equa tion (2.15) cannot hold unless
(2. 16)
Ill
~ lii(zi) = 1 . j=O
Consequently,
(2.1 7)
Ill
o(z)= min { ~ ui(.:i)fi(z)+u0 (zi)[f-O(z) - J°(zi)]}. ; E C(;o) j= l
Upon replacing.: by z in (2.17), we obtain the following bound on o(zi):
(2. I 8)
Ill
8(:::)::; ~ ui(zi)Ji(z) + uO(z)[J°(z) -J°(z)] . j= I
S. ~ C ff(~)< 0 · = 1 · d ( ) > 0 "' 111 -i( )ff( ) < 0 mcc z E , z _ , / , ... , m, an u zi _ , £..;i=I u zi zi - .
Hence,
On the rate of convergence of certain methods of centers 237
(2. 19) <-0- r{)A rO _ o(zi) ll (t.)[1 - (z) J - (_i) ] .
This is the first inequality we set o ut to prove. Next, from Lemma 2. 7 and (2. I 3) we deduce that
(2.20) li1!1 infUo(.:i)> A,0. 1~00
Since by Lemma B. 11 , X-0 > 0, (2.20) implies that, given a ny ex E (0, I), there exists an i0 (ex) such that
(2 .2 1) l(O(zi)>A.0( 1 ex) forall i">.i0 (cx).
Combining (2.18) and (2 .2 1) we now obtain
(2.22) o(z) < A.0 ( I ex)[/-O(z) .f°(zi) ] for all i > i0 (ex) •
which is the second inequality we needed to prove. Finally. from 2.5 and (2.2 I ) we obtain
(2.23) f-O(zi+l) /-O(: i) < 8(.::)::; X0( I - ex) [ fl(~) / 1(:i)]
forall i>i0 (ex).
Rearra nging (2.23) we fi nal ly obtain
(2 .24) .r°(zi+l) -J0 (Z:) ::; [ 1 X0 (1 ex ) l l /-O(zi) - / 0 (z) J
for all (:> i0 (ex),
which com pletes o ur proof.
2.25. Corollary. Suppose that the assumptions of Theore111 2. I I and of lemma I. 5 are satisfied. Then { z i} f:o converges to z linear(11•
We now establish an upper bound on the rate of convergence of Algorithm 2.5.
2.26. Theorem. Let {zJ j:0 be an infinite sequence generated by Algorithm 2. 2 in sobiing Problem 1. 1. le I z be the solution of 1 . I and lf!f
'
238 0. Pironneau and E. Polak
'A.0 be defined by (2.13). If assumptions 2.1 are satisfied, then, either X° = I and tlze sequence { J°(zi)}j:0 conl'erges superlinearly to J°(z), or r..0 < I and there exists an integer i 1 such that
(2.27) flC.:;+1) - J°e);:::(1 - X0 HI°Cz;) J°(z) I
for all i ;::: i I ·
Proof Applying part (b)(ii) of the strong duality theorem B.5 to (2.6), we condude that a i/0 (.:i) defined as the first component of a solution
of (2.8) must satisfy
(2.28) i/0 (.:;)[J°(.:i+l) f 0 (.:i) o(z;)I = 0.
Now. according to (2.21). i!0 (z;);::: (I a)X0 for all i;::: i 0 (a). Therefore, by making use of Lemma B.11, u0 (:::;) > 0 for all i > i0 (a). Hence, from
(2.28). we obtain that
(2.29) /J<.:;+1) - !°<.:;) = o(.:i).
Next, according to (2.10), for i = 0, I, 2, ... ,
(2.30)
111 Ill
o(z;) = 111.c.LX {inf {(1 - _6 ui)o + _6 l/fi(z) 11.::;-0 z E C(zo) j=O j= l
/j t- IR
+ uo [fl(.:) .f°(z; J}} .
Setting 11 =A., some clement of J\(z) (defined by (2.9)), we conclude
that
(2.31)
Ill
o(z;)Z inf { _6 '!JJi(.:)+t..0 [J°(z) - fO(z;)]} .:E C<zo> /= 1
The infimum in (2.31) is achieved at z because "if'~o ">/V' f'<z) = 0. There
fore
On the rate of convergence of certain methods of centers 239
Thus
(2.32)
Ill
o(z;);::: .6 Ai/(:)+ r..0 [J°(z) - J°(z;)J = r..0 [J°(z) f 0 (z;)l j= l
for all A. E J\(E) .
o(z;);:::X0 rt°(z) - J0 cz;)J, for i=O, 1,1 ....
Finally setting i 1 = i0 (a), for so me a E (0, I), (2.29) and (2.32) combine to give
(2.33) t 0 czi+l) - f 0 (z);::: ( 1 X0 )[J°(z;) - t°(z)] , for a ll i;::: i1 '
which proves the second part o r the theorem; the first part follows direct ly from Theorem 2.1 I. This completes our p roof.
Combining (2. 12) and (2.27), we see that whenever Algorithm 2.2 constructs an infinite sequence {.:;};':0 , we must have
(2.34) o ) 1·0 e) f (.:;+1 · - = 1 - 'A° · lim - -
;~~ fo(zi) - 10 (z)
Therefore the larger the value of X0 E (0, I ), the faster Algorithm 2.2 converges. When the unconstrained optimum of / 0 is in C(z0 ) , then X° = I and Algorithm 2.5 converges superlinearly (or in a finite number of steps). When X0 is small (which is the case when llV' J°(,:) II is large compared to llV'/(z)ll or when the Kuhn-Tucker constraint qualification is only marginally satisfied), Algorithm 2.2 will be slow.
(2.34) should be compared with a similar evaluation obtained by Faure and Huard [3] for the method of centers based on the distance functions.
(2.35)
Ill
d(z,z;) ~(- 1)111 [f0 (z)-J0 (z;) I n f'(z) . j=l
They showed that if {z;}j:0 is an infinite sequence generated by this method of ccn ters, then
'
240 0. Pironneau and t:. Polak
(2.36) !°(:i+1) - f0 (~) 111
lim - -- - = - -i~~ f 0 (:i) [°(2) m + 1
where nl is the number of active constraints at:. Therefore when Ill is large algorithm 2.5 is likely to be somewhat faster 1
: while the method of centers based on (2.35) will be faster than Algorithm 2.5 when the Kuhn-Tucker constraint qualification is only marginally satisfied. ln any event, as we sha ll sec, the major advantage of Algorithm 2.2 is that it can be implemented, without losing its rate of convergence, with less computational effort than the method of ce nters based on (2.35).
3. Modified method of centers
The method of centers 2.5 requires that at each iteration we solve
the following problem.
3.1. Problem. min max {fO(:) - fO(zi)Ji(z), j = I , ... , m} .
In a modified method of centers [6], this problem is replaced by two subproblems: a direction finding su!JtJroblem
(3.2) min {tz 0 l<V' /°(:i), h> < tz0 ;ji(zi) (lzo, fll
+<V'Jf(zi),h><h0 ,;= l , ... ,m;hES} ,
where S = {/z E IR 11 I I/ii I~ 1, i = l, 2, ... , 11 }. which y ields a solution (11° (.:), /i(:i)), followed by a step size determination subproblem
(3.3) 01z (zi) ~minmax{f0 (zi+ µh(zi)) - !° (zi);
Ji(:i + µh(zi)), j = 1, ... , m } ,
whose solution is µ(:i). The final result is Z;+J = zi + µ(zi)h(zi).
1 fl(z i+ 1 - fl(z)
Faure and Huard proposed a way out for decreasing lim - - -- . However :\lifflin /J(Z i) - fl(z)
showed [ 101 that the proposed mod ification makes the computation of an "€-center" much
more time consum ing.
On the rate of convergence of certain methods of cenrers 241
Referring to the example in the appendix, we see that th e a lgorithm using 3.2 and 3 .3 does not converge linearly. Therefore, we modified 3 .2 to obtain the following new algorithm.
3.4. Algorithm (modified method of centers). Step 0. Set i = 0. Step I. Compute a so lu tion (11°(::.i)h(z)) of
(3 .5) mi n{h0 +111'1 11 2 1< V' J°(zi),h>~ho;
[i(zi) + (\i' Ji(zi), h> < 1zO, j = l , ... , m} .
Comment. To find a solu tion o f (3.5), solve (B.19) and use (13. 22) (see Appendix B). Note that whenever n > 3111 and if Wolfe's algorithm is used [ 19] , (B. I 9) is easier to solve than 3. 2.
Step 2. lf h-O(zi) = 0, set z = zi, and stop; else, compute a solution (o,,(z), µ(zi)) or
(3.6) oh(zi) = min max{/0 (.:i + µh(zi)) - J°(:i); µ
Ji(zi + µh(zi)), j = I, ... , m} .
Step 3. Set zi+I = zi + µ(zi)h(z;) and go to step 1.
It is easy to show that under assumption 2. 1, Algorith m 3.4 converges in the same manner as 3.2, i.e. , either it stops at z or it generates an infinite sequence whose accumu lation points are solutions of Prob lem 1. 1. For a more general situation, the converge nce properties of Algorithm 3 .4 are given in [ I 3] .
We shall now prove that Algorithm 3.4 h as a better rate than algorithm 3 .2. Throughout this section, the fu nc ti on Ji, ; = 0, 1, ... , mare assumed to be convex 2 and twice continuously differentiable.
3. 7. Lemma. Suppose that assumptions 2. l (i)- ( iii) are satisfied. L et oh (z), (h0(z), h(z)), and z be defined as solutions of (3.6), (3.5) and Problem I. I respectively. Then, given E > 0 there exists a -y1 (E) > 0
2 Ac tually, the convexity of Ji, j = 0, 1, ... , 111, is needed only in a compact convex subse t oflR17
containing C(zo) in its interior. ll is on ly for the sake of simplicity that we assumed global convexity.
'
242 0. Pirom1eau and£. Polak
such that 811 (::::).:; iir 1 [h0 (z) +t ll/z(.::)11 2 1 for all:::: E B(z, -y 1 (E)) n C, with M defined by -
(3.8) a 2 !1
ill= max {l; II- (::::)II ,j = 0, I, ... , mlz E B(:,E)}. a:::2
Proof According to (3.6),
(3.9) 811 (z)= min {o,,1f0 (z+µh(z))-/0 (z).:;011 (/) 1z.µ)
Ji(z + µ/z(:::)).:; 81z. j = I , ... , m},
It is straightforward to show that the strong duality Theorem B.5 applies to (3.9). Therefore. we obta in from Theorem B.5 (a)(ii), that, for
any::: EC.
111
(3.10) o,, (::::) = m~x {inf {(I - ~ wi) 811 w~O (01z,µ) 1=0
Ill
+ ~ wi[i(:+µlt(z))+w 0 iJ"(z+ µ/z{z)) -f0 (z) l}).
Ill
(3. 11 ) =m~x{inf {~ wifi(::::+ µh(z))+w0 [.f°(z+ µh(z)) w - 0 µ l= I
- !"(:)I} I ??. J " I ) .
Next. because of (3.8), the right hand side of (3.11) expand s as fol
lows.
Ill
(3.12) ~ wifi(::::+µfz(z))+w0 [f0 (z+µh(z) - J°(z)) j= 1
111 Ill
.:; ~ wifi(z) + µ ~ wi<'V [i(z), h(z)> + ~ µ 2 M llh(z)ll 2 ,
j= I j=O
On the rate of convergence of certain methods of centers 243
for all w E IR. 111 + 1 such that w ~ 0, L,/~o wi = 1, and all (z, µ)such that z+ µh(z)E B(z,E). From Lemma B.1 6. (see append ix), there ex ists 'Yt (E) such that (3.12) is valid for al l w ~ 0, sa tisfy ing L,j'~ 0 wi = I. for all µ E [0, I] and for all z E B(z, -y 1 ) n C. Therefore . fo r any z E B(z, -y 1 ) n C, (3.10) and (3. l I) combine to give
111 111
(3 . I 3) 01z(z).:; max {inf { ~ wiji(z) + µ ~ wi<'Vfi(:::), h(z)> w ?: O µ E 10,l] j= I j=O
111 } +!µ2 Mll'7(:::) 11 2 } 1 ~ wi= J . 1=0
\
By settingµ= M- 1 in (3.13) (and deleting the inf operation) we obtain
(3.14)
Ill Ill
811 (z).:; max { ~ wifi(:::) + ~ 6 wi<'V [i(z), Ji(::::)> w ~O j= 1 i=O
1 Ill
+ 2M llh(z)ll 2 I ~ wi = I} . i=O
Since for z E C, fi(z).:; 0, j = I , ... , 111; and si nce M :;::.:_ I, it follows that [i(z).:;M- 1[i(z), j= l, ... ,111 . Therefore, for any zEB(5,-y 1 (t:))nC
(3 .15)
111 m
8h(z)< ~ m;x { ~ wiji(z)+ 6 wi<'VJi(z),h(z)> w~O 1=1 1= 0
m
+t ll/z(z)ll2 Ii~ wi = I}.
By definition, (h0 (z). /i(z)) is a solution of (3.5) with ::::i =:::.Therefore
<'VJ°(z), h(z)>.:; h 0 (z); and /(z) + <'V p·(::::), /i(z )) <ho(:::).
j= I , ... ,111.
Hence, from (3. 15),
'
244 0. Pironneau and E. Polak
m m
8;,(z)~ ~max { 6 wihO(z)+~ llh(z)ll 2 ! 6 wi= l} , w > 0 i=O - i=O
i.e., 8;, (.::) -:::= M 1 [!z0 (.::) + !- llh(z)ll 2 ), which proves the lemma.
3.16. Theorem. let {z;}f:,0 be a sequence generated by Algorithm 3.4 ill the process of so!Fing Problem 1.1, and suppose that assumptiOllS 2.2 alld those stated in lemma 1.5 are satisfied. Then, given any a E (0, I), there ex is ts an ill teger i 2 (a) such that for all i > i 2 (a),
(3.17) f-O(zi+I) J0 (z) ~ [I - ''k ["J;.:0(1-a)] 2] [J°(z;) - f 0(.z)) ,
where X0 . m0 . Mare as in (2.13), (1.6) and (3.8), respectiPely.
Proof Our strategy for establishing (3.17) will be as follows. From step 2 in Algorithm 3.4, we have
(3.18) J°(z,+ 1 ) - f0 (z;)~ 8,,(z;), i= 0, 1, 2, ...
We shall show that for every a E (0, l) there exists an integer i 1 (a) such that for all i > i 1 (a)
(3.19) 1zO (z i) + -t 11/z(z;) 11 2 ~ mO°XO (1 - a) 8(z;) ,
where 8(z 1) is as in (2.3). Inequality (3.17) will the n follow from (3.18), Lemma 3. 7, (3.19) and (2.22).
Since the bound m 0 in ( 1.6) is only good for a ball of radius e about ~' we shall have to restrict ourselves to this ball in establishing the rate of convergence. Now, for any z EIR n let
(3.20) r(z) = {z'IJ°(z') - f 0 (z) ~ 8(z);Ji(z') ~ o(z'), j = I , 2, ... , m}
Then, from the example in [ l] , p. 111 , it follows that r is an upper semi-continuous mapping. Since r(z) = {z} and r is u.s.c., it follows by definition, that there exists a "(2 E (0, e ] such that r(z) c B(z, e) for a.II z E B(z, "( 2 ). Consequently, for all .:: E B(z, "(2 ), we must have
(3.21) 8(z) =min {81j-O(z') - J°(z) ~ 8;/(z') ~ 8,j = I, ... , m; (Ii ,z')
z' E B(z, e)} ,
for all z.E B(z, -y2 ) n C.
On the rate of convergence of certain methods of centers 245
Upon applying the strong duality theorem B.5 to (3.21), we obtain
(3.22)
(3.23)
m
8(z)=m;x{i~f. { 6 vif(z')+v0 [f-O(z') u _ O zEB(z,c) i=l
oEIR
m
- f 0(z)] +(I - i~ vi) 8}}
m m
=max {inf { 6 vifi(z') + v0 [fO(z')- fO(z)]} I ~ vi= l} v>o z'EB(z,e) i=l 1=0
Therefore for all v ~ 0 such that "2;~0 vi= I and all z E B(z , "(2 ) n C,
Ill
(3.24) 8(z)~inf { 6 vi/(z')+v0 [J°(z') - !°(z)J}. z'EB(z,~e) i= I
Expanding f 0 (z') - J°(z) to second order terms and making use of the convexity of the functions Ji, j = l, ... , m, we obtain for all v ~ Osuch that Lj1~0 vi = I , and all z E B(i, "(2 ) n C,
m m
(3.25) 8(z) ~ inf { 6 vij1.(z) + 6 vi(''V /i(z), z' z> z' E:B(z,e) i= I i=O
+tm0 v0 llz' -z/1 2}.
By deleting the constraint z' E B(z, e) in (3.25), we obtain
(3.26)
Ill
8(z) :> 6 vi/(z) i= l
I m
-2-,n-ovo 11 6 Pi'V fi(z)/12 1=0 '
!:_or all v ~ 0 such that v0 > 0, L~0 11i = I an? ~II z E B(z, 'Y2 ). n ~'. Let v(z)EIR. 111
+1 be a solution of max {"2;j1~ 1 u1fl(z) - t /1"2;j1; 0 u1Vf(z)ll 2 1
ll 2' 0
'
246 0. Pironneau and E. Polak
"L7!0
µi = I}, which is equivalent to the dual o!:_.(3.5.) (see appendix (B.17)-(B.22)). Then. since 111°pO(.:) < 1 a~d "Lj'~ 1 P1(.:)/1(.:) ::;; 0, we must have. ~?~ 1 1•/(.:)j'l(.:) ~ (llZOl;Q(.:)) l ry~l l'l(z)j'l(.:); aSSL~ing that pO(.:)> 0. Hence. from (3.26) and (B.22) (with 11 replaced by 1')
(3.27) -) > -o(_ - 01 ::v, c-)
Ill -
m m ]
l~ J.l°c.:)/(.:) - } II ~ J.i(z) V' fi(z)ll 2
j= 1 j=O
['1°(7) + l mOpO(z) - 2 1117 (.::)1121
By following the same pattern of reasoning as in the theoretical me thod of cente rs. for Lemma 2. 7. it is easy to prove that
(3.28) lim inf J.0 (.:) 2 A.o • ' :-:
:EC
where XV is defined by (2. 13) and , according to Lemma B.11 satisfies A.-o > 0. It now follows from (3.28) that. given any a E (0 , 1 ), there exists ,,3 (a) E (0. )'2 ) such that J.0(.:) > X0 c 1 - a)> 0. for all.: E 8(3, 'Y:i(a))nC. Therefore (3.27) implies that o(.::)> (111o~o (l -a)) 1 [ho(.::)+ }°llh(.:)11 2 ). for all.: EB(:. 'Y3 (a)) n C. ll ence by making use of Lemma 3. 7, we conclude that, given any a E (0, 1 ). the re exists an integer i 1 (a)
such that
(3.29) 1
f-O(zi+J)-f0 (zi)<o11 (zi)::;; M(h 0 (z)+1 llh(z;)ll 2)
o-o < !.!..!.._.!::._ ( 1 - a) s: 7
.)
- M u(-, ,
for all i ~ i 1 (a). Finally. from (2.22). there exists an integer i 2(a)2 max(i1 (a). i0 (a)), such that
(3.30) ~(Zi+I) - ~(z)::;; moxo2 ( l -a)2 [~(zi)- ~(z )] ,
for all i 2 i 2 (a) .
Rearranging (3.30), we obtain (3. 17), which completes our proof.
On th e rate of convergence of certain methods of centers 247
3.31 Corollary. Suppose that the assumptions of Theorem B. 5 are satisfied, then {zi}f:o con11erges to z linearly.
3.32 R emark. Note that if Algorithm 3.4 is used for solving an unconstrained problem, Algorithm 3.4 reduces to the method of steepest descent. For this special case, X0 = 1 and Theorem 3.16 agrees with well known results for the method of steepest qescent.
To implement Algorithm 2. 2, Huard [ 6] proposed to accept as a successor to z; , any point zi+l satisfying
(3.33) o(z;)::;; d(zi+I > Z;)::;; p o(z;),
where o and d arc as in (2.3) and (2.4) and p E (0, 1). To compute such a Z;+ 1 , one can use k; iterations of Algorithm 3.4 on (2.6) 3
. We initialize the computation by setting z = Z;· Tracing through the proof of Theorem 3.16, we find that this theorem can be adapted to cover also this particular case, to give, for i sufficiently large,
(3.34) 0 ~ d(zi+l > Z;) - o(z;)
[ XOmO ~ k;
~ 1 - 4M (I -a) J (d(z;, Z;) - o(z;)),
where according to (2.4), d(z;. t;) = 0. Since k; in (3.34) is the smallest integer for which (3.33) holds, we must have 4
(3.35) r, - o 0
k; ~ log(l - p) Llog (t A. r:,~l -a)2)] -I
By inspection of (2 .22) and (3.34 ), the constant of ex ponentiation of the rate of convergence of this implementation which constru cts z i+ 1 to satisfy (3.33) is (I - p""f...O). Since one iteration of the implementation is
3 This suggestion is very close to lluard's which was to use a finite nu mber of iterations of a "partial" algorithms [ 6]. Unfortunately we do not have a:iy bound on the rate of convergence for this partial algorithm.
4 Note that this resu lt is much better than the one obtained by Mifflin [ 10] for the method of center based on (2.33) and implemented by k; iteration of the method of steepest descent to find z i+ 1 sati sfying a relatio n similar to (3 .34 ).
'
0. Pironneau and E. Polak 248
as dif_!:.icult to perform as ki iterations of _!..~orithm 3.4, the f.onstant 1 - pt...0 should be compared with [1-(f...O mO(l - a)2 /M)1 '.From (3.35), we can deduce that l pXO > [l - (X0
2m0(1-a)2/M)lk;, for
all p 6 (0, 1 ). This shows that there is no advantage in choosing a p
which will result in k i > l. Next we note that (3.6) also cannot be solved exactly in a finite
number of iterations of any of the applicable algorithms. However, an approximation to a solution of 3.3, compatible with preservation of linear convergence , can be found by means of the subprocedure defined in step 2', below, which is a cross between a procedure due to Arnijo { 141 and the Golden Section Search. As we shall see shortly, in solving (3.6) this subprocedure is highly efficient and, in all likelyhood, out performs such well known methods as the Davidon cubic interpolation
scheme.
3.36. Step 2'. If i = 0, select a i3 > 0 and go to(*); else go to(*) . (*)if 11°(z ;) = 0, set z = z; and stop; else define e;: IR-+ IR by
(3.37) 8;(µ) = max{.r°Cz, + µh(z;)) - f 0 (zi);Ji(z; + µh(z;)),
j = l , ... , 111} .
Se t l; = {j le(l) = Ji (z ; + h(::; ;)) - o(j)fi(z;), j = l , ... , m}. (o(j) = 1 if j = 0, 8(j) = 0 , otherwise, and go to(**).
(**)Compute
de;(t+) . - d - =max {<V' f (z; + µh(z)), h(z;)>}
µ jE=l;
If de ;Idµ( 1 +) S 0, set µ(z ;) = 1; else use the Golden Section Search with initial interval [0, 11 to find two points µ(z;) < µ'(z;) such that
(3.38) O(µ(z; )) S J3[µ'(z;) - µ(z;)l < V' f° (z;), h(z1)).
3.39. Theorem 5 . Let P; be the number of evaluations of e;(µ) which
5 The process of replacing the computation of a minimizerµ of a convex function e: (0, + 00
)-+ (-
00, 01 by the computation of two pointsµ, µ' such that 0 < µ S µ S µ ' and such that
0(µ) < {3(µ' - µ) ~ (0 +) dµ
constitutes a general procedure for implementing algorithms of the type of (3 .9), without loss
of linear convergence.
On the rate of convergence of certain methods of centers 249
are required to find (µ(z;), µ'(z;)) in step 2', giPe11 z;. Then the conclusion of Theorem 3.16 remains valid wizen step 2 of Algorithm 3.4 is replaced by ste/J 2', provided (3. 17) is replaced by
(3.40) f°Cz;+i) - J0 (z) s [l -~X0 (1-a)J 2 _J3_ [/>(z.) M l+J3 I
t°Cz)J .
Furthermore, for all i = 0, l, 2, ...
(3 .41) P; S 1 + 6 log [ 1.36 L (1 + /3)] .
The proof of Theorem 3.39 is rather involved and we therefore omit it. The interested reader will find it in [ 13 J.
Note that when l = I, if we choose J3 = I 0 in step 2', the rate of convergence of the implementation of (3.9) using step 2' is about the same as that of Algorithm 3.4 and we are guaranteed to find a step size µ(zi)
with at most 7 evaluation of 8;(µ). Th is is quite reasonable and it is doubtful that any other method would find a step size µ(z;) satisfying (3.35) with fewer evaluation of e;(µ).
4. Conclusion
Thus, we have presented three results in this paper. We have shown that a theoretical method of centers due to Huard [ 5] converges linearly; that an implementation of this method converges sublinearly , and that a new modified method of centers, which we have proposed , converges linearly. This new method not only converges linearly, but it also has the nice feature that its direction finding subproccdure is essentially insensitive to the dimension of the space on which the optimization problem is defined. Because of this, it was possible to adapt it for the solution of optimal control problems (which are problems in L
2 n L~)
[ 13]. In actual computations, the new modified method of centers can also be combined with standard interpolation schemes which utilize the fact that the successive centers constructed by a theoretical method of centers lie on an analytic curve. However, it is not possible to make any theoretical statements as to the rate of convergence of such a hybrid scheme.
'
250 0. Pironneau and E. Polak
Appendix A
Counter example to the linear convergence of the modified method of centers in [6] and of the Topkis-Yeinnott method of feasible directions [18].
I. Topkis-Veinnott method of feasible directions [ 18] Consider the following problem.
A. I. Problem. min {x2 + y 2 ly ~ O} .
Let z0
= (x0 , Yo) be ~uch that 0 < lx0 I~ Yo ~ t· To solve a problem of the type min {J°(z)lf'(z) ~ 0, j = l, ... , m} the Topkis-Veinnott method of feasible directions, computes a feasible direction h(zi) at zi as a solution of min {max{<'vt°(zi), h>; fi(z;) + <'V/i(z), h>, j = l, ... ,m}}
lhkl " l
which, for J°(z) = x 2 + y 2, f 1 (z) = - y, m = 1, becomes, at z 0 ,
(A.2)
Hence
(A.3)
min {max{ 2x0 h 1 + 2y0h 2 , - h2 - y 0}). - l:Shl:S1
- l:Sh2 :Sl
Yo - 21x0 I hl(z;)=-sgnxo; h2(z;)=- 1+2yo
The step size µ(z;) is computed as the solution of min J°(zi + µ h(z;))I f(z; + µ h(z;)) ~ 0, j = 1, ... , m . Therefore µ(z 0) is found as the solution of
{ [ Yo - 21x0 I J 2
(A.4) mJn (x0 - µ sgn Xo)2 + Yo - µ 1 + 2 Vn Iyo
y 0 - 21x0 1 }
- µ >o '+ 2yo - .
From (A.3) and (A.4), (see fig. 1), it is clear that the constraint in (A.4) is not active at µ(z 0). Therefore µ(z 0) satisfies
On the rate of convergence of certain methods of centers
y
z i-1
x2+ y2 = y1-!
I I •
I
z
Fig. l.
I
I I I
I I
25 1
x
(A.5) Yo - 21x0 1 ( y0 -2L~0 1)
(lxol - µ(zo))+ I 2 .l'o -µ(zo) l 2 =O. + Yo + Yo
Let z 1 = (x 1 ,y 1 ) be the next point computed by the algorithm. Then from (A.3),
(A.6) Yo - 2 lx0 I
x 1 =x0 - µ(z 0)sgnx0 ;y 1 =y0 - µ(z 0) 1
_,_? -Yo
Hence, from (A.5) and the assumption on x 0 , Yo,
(A.7) 1~1= (y 0 - 21x0 1) Yi 1+2y 0
which implies that
(A.8) ,~,. ~Yo· I Yi
=Yo
lx0 I 1 - 2 -
Yo
l + 2Yo
Let {z;} be the sequence generated by the Topkis-Yeinnott method of feasible direction, in solving problem A. I, and startin g from z0 = (x0 ,y0)
~52 0. Pironneau and E. Polak
;uch that O< lx0
1 '.Sy0
:S1 . At each iteration the algorithm decreases the cost, therefore if Yo is chosen such that Yo ~ t, by making repeated use of (A.8), we obtain 0 < lx;I ~ Y; ~ t for all i >- l. From (A.3) it follows that h2 (z;) :S 0 for all i >- 1. Hence (see fig. 1)
(A.9)
where
ll.:;+1 - z 11
llz;-zll =cos 'Y; ,
(A. I 0) 'Yi = 1T - P;+ l - P; .
From (A.8), tan P;+i = Y;+i /x;+J >- 1/Y;· Hence, 13;-+ 1-rr as i-+ 00
, and from (A.10), 'Y;-+O as i-+ 00 . Therefore, from (A.9), lim llz;+ 1 -zll ·
i~ 00
· llz; - .fn- 1 = 1, which proves that z; cannot converge linearly.
2. Modified method of centers Huard's modified method of centers [ 61 has the same direction de-
termination subproblem as the Topkis-Veinnott algorithms just des
cribed, but the step size µ(z;) is given by
min {max{f°Cz; + µl!(z;)) -J°(zi);/(zi + µh(z;)) , µ
j= J, ... ,m}}.
which for Problem A. I becomes, at z0 ,
. f ( - 2 (. Yo - 21xol) 2 (A.II) mJn\max('l:o - µsgn.\o) + .1 0 - µ 1+2y
0
I' _ .Vo - ~~}}. · o µ I + 2.1 ·0
Consequently, µ(z0 ) satisfies either (A.5) or
(A.12)
y 0 -2ix0 1) 2 Yo -21x0 1
(lxo I - µ)2 + (l'o - µ 1 + 2Yo =Yo - µ 1 + 2yo
On the rate of convergence of certain methods of centers 253
Therefore, the pointz 1 =(x 1 ,y 1 ) computed by the Huard modified method of centers, from z
0, is given by (A.6) with µ(z 1) > 0 being the
smallest strictly positive number which satisfies either (A. 5) or (A.12). Note that if µ(z 1 ) is given by (A.12) then (x 1 , y 1 ) satisfies xy +YI = Y 1 · Hence it is clear (see fig. 1) that if (x0 ,y0 ) is close enough to (0, 0), X will never be given by (A. 12), which implies that, for Problem A. l, and starting from z0 such that 0 < lx0 I~ Yo ~ E, the Topkis-Veinnott and t~e Huard algorithms compute the same sequence of points and hence neither of those two algorithms can converge linearly .
Appendix B. Miscellaneous results
l. Duality theory
Consider the problem
(B. l) min {go (z)lz E n} ,
with
(B.2) n={zEIR. 11 lgi(z)~O, j= 1, ... ,p; zEC},
where gi: C-+ IR, j = 0, 1, ... , p, are convex and continuously differentiable functions, and C is a convex subset oflR 11
• The following problem is called the dual of (B. l ).
B.3. Problem. max {inf{g0 (z)+~f= 1 u igi(z)}} (u=(u 1 ,u2 , ... ,uP)EIR.P). u 2: 0 zEC
Let¢ :IR.P-+ fR U {- 00 } , be defined b y,
p
(B.4) ¢(u) =inf {g0 (z) + ~ uigi(z)} . zEC j= I
Definition. Any z En, satisfying go(z) = min{g0 (z)lz En}, will be called a solution of (B. l). Any ti 2: 0, satisfying, ¢(i'1) =max ¢(u), will be called a solution of Problem B.3. u2:o
B.5. Strong duality theorem. Let SP be the set of all solutions of(B. l), i.e.,
'
254 0. Pironneau and E. Polak
(B.6) SP={:' E .Qlg0 (z') =min g0 (z)} . : E n
Suppose that S11 is TZot empty and that
(B.7) .Q' = { z I gi (z) < 0, j = 1, .. ., p}
is TZOt empty. Then (a) (i) Problem B.3 has at least one solution;
p
(B.8) (ii) max {inf {g0 (z) + 6 uigi(z)}} =min {g0 (z)lgi(z)::::; 0 ,
(B.9)
11 >o zE C j= 1 zE C
j=l. .... p};
(iii) for any u. a solution of Problem B.3, and 1or any z, a solution of (B. I).
p
g0 <.:) =min {g0 (.:) + E uigi(z)}: : E-C j = I
(b) a 1•ectf!f Ii ~ 0 i11 IR P is a solution of Problem B.3 if and only if there exists a: in .Q such that
p p
(B.10) (i) gO(z) + E uigi(i) = min{g0 (z) + E uigi(z)}; j= I zE C j= I
(ii) uigi(~) = 0, j = l, ... , p.
This theorem is a particular case of the strong duality theorem stated in Geoffrion [ 41 . Related theorems can be found in Rockafellar [I 5], Mangasarian [9].
2. A property of optimal multipliers
B. 1 1. Lemma. If assumptions 2.2 are satisfied then
(B.12) x0 =min {<A., e>I A. E J\(z)} > 0,
where e = ( l, 0, 0, ... , 0) E IR. 111 +1 and J\(z) is as in (2.9).
On the rate of convergence of certain methods of centers 255
Proof From (2.9) and the fact that z is a solution of Problem 1.1 , J\(z) is a nonempty [2] compact subset of IR m +l and therefore there exists a XE J\(z) such that X0 = <X, e>. Since the functions/i, j = 0, 1, ... , m are convex and continuously differentiable, we must have
(B.13) fi(z)?.fi(z)+<Vf(z),z - z>, j=O, 1, ... ,m,forallzEIRn.
Multiplying (B.13) by 'Al, for j = 0, 1, ... , m, and adding the results, we obtain
m m m
(B.14) E 'i!f(z)?. E 'i/fi(z)+ <E 'i!Vf(z),z - z>. j=O i=O j=O
Thus, it follows from (B.12) and (2.9) that
(B.15)
m
E 'f:!ji(z)?. X° t° (z) . for all z E IR n .
j=O
Hence, if X0 = 0, (B.15) contradicts assumption 2.2(iv). Consequently the lemma must be true.
3. Justification of (3.12)
B. I 6. Lemma. There exists a -y1 (e) such that z + µh(z) E B(z, 'Yi) for all µ E [O, 1 J and all z E B(z, 'Yi (e)).
Proof It follows from the strong duality theorem B.5, as applied to (3.15), that
(B.17)
(B.18)
h0(z) +-t llh(z)ll 2 =max {inf
u~o (ho,h)
m m
m
{O-Eui) ho j=O
+ i~ uifi(z) + 2o ui <V fi(z), h> +-!- llh11 2}}
m m m } =m;x finf{ ~ uifi(z)+ ~ ui<V'fi(z),h>+-!-llh11 2 } I~ ui=J ,
u_O , h l=l 1=0 1=0
'
256 0. Pironneau and E. Polak
and hence that
(B. 19)
Therefore
(B.20)
111 Ill
1i0 (:) +-~ llh(z)ll 2 =max{ 6 uifi(z) + 6 ui <V' f(z), h> 11:>0 j=I j=O
111
+t111111 2 1 6 11i= j=O
Ill
m
6 ui V' Ji(z) + lz = O}. i=O
11°(:) +t 1117(:)112 =max{ 6 uifi(z) - 11?°0 j=I
Ill 111
-111 6 uiV'f (:)11 21 6 ui = 1} .
j=O i=O
Let u E 1R111+1 be a solution of (B.20), then
Ill Ill
(B.2 1) 1zO (:) + t ll/z(z)ll2 6 [iij(z) - -!- II 6 [ii yi/(z)ll2
j= I
and from (B. 19) it follows that
(B.22)
Ill
lz(z) = - 6 [iiyi f (z) 6
j=O
Hence, if: E C,
(B.23)
Ill
11°(z) + lllz(:)ll2 = 6 uifi(z) ~ 0, j=I
j=O
6 This condition is stronger than the one used by Geoffrion [ 4] .
On the rate of convergence of certain methods of cen/ers 257
and therefore ! lllz(z)ll 2 < - [h0 (z) +! llh(z)ll2]. From (B.20), it is clear that 11°( ·) + t llh( ·) 11 2 is continuous, negative in C and that it takes the value zero at-z. Therefore there exists -y 1(E)E (O,! €] such that max { [h 0 (z) +1 llh(z) ll 2 J lz E B(z, 'Yi) n C } ~ i €
2. Hence, it follows that
(3.24) llh(z)ll~!Eforall zEB(z1 ,-y1 )nC .
References
[I ] C. Berge, Topological spaces (MacMillan, New York, 1963). [ 2] M. Canon, C. Cullum and E. Polak, Theory of optimal control and mathematics program
ming (McGraw-Hill Co., New York, 1970) Theorem 19, p. 4 and Lemma I, p. 68. [3] P. Faure and P. lluard, "Resultats nouveaux relatif a la methode des centres," Paper pre
sented at the Fourth International Conference on Operations Research, Boston, 1966. [4] A. Geoffrion, "Duality in nonlinear programming," SIAM Review 13 (1971) 1. [51 P. Huard, "The method of centers," in: Nonlinear programming, Ed. J. Abadie (North
Holland Publ. Co., Amsterdam, 1967). [6 ] P. Huard, "Programmation mathematique convex," RIRO, R 7 (1968) 43 -59. (71 H.W. Kuhn and A.W. Tucker, "Nonlinear programming," in: Proc. second berkeley sym·
posh1111 math. stat. probability (University of California Press, Berkeley, Californ ia, 1951) pp. 481 - 492.
[ 8] F.A. Lootsma, "Constrained optimization via parameter free penalty functions," Philips Research R eports 23 (1968) 424-437 and suppl. 3 (1970).
[91 0. Mangasarian, Nonlinear programming (McGraw-Hill Co., New York, 1969). [ l O] R. Mifflin, "Convergence rates for a method of center algorithm," Thesis Operation
Research Center, University of California, Berkeley, 1971. [ 111 J. Ortega and W. Rheinboldt, Iterative solution of nonlinear equations in several variables
(Academic Press, New York, 1970). [ 121 0. Pironneau and F. Polak, "On the rate of convergence of certain method of centers."
Memorandum No. ERL-M296. University of California, Berkeley 1971. [13 1 0. Pironneau and E. Polak, "A dua l method for optimal control problems with initial and
final boundary constraints," Memorandum No. ERL-M299, University of California, Berkeley 1971.
[ 14 I E. Polak, Computalional methods of optimization (Academic Press, New York, 1971). [ 15 I R.T. Rockafellar, Convex analysis (Princeton University Press, Princeton, New Jersey,
1970). [ 16 J D. Topkis and A. Veinnott, "On the convergence of some feasible direction a lgorithms for
nonlinear programming," SIAM Journal on Control 5 (2) (1967) 268- 79. 117 I R. Tremolieres, "La methode des centres a lroncature variable," These, Paris, 1968. ( 18] P. Wolfe, "A duality theorem for nonlinear programming," Quarterly of Applied Mathe
matics (1968) 239-244. (191 P. Wolfe, "The simplex method for quadratic programming," Econometrica 27 (1959)
382-397. (20] G. Zoutendijk, Method of feasible direction (Elsevier Publish ing Co., Amsterdam, 1960).
'