interior point methods and kernel functions of a linear
TRANSCRIPT
Georgia Southern University
Digital Commons@Georgia Southern
Electronic Theses and Dissertations Graduate Studies, Jack N. Averitt College of
Spring 2009
Interior Point Methods and Kernel Functions of a Linear Programming Problem Latriece Y. Tanksley
Follow this and additional works at: https://digitalcommons.georgiasouthern.edu/etd
Recommended Citation Tanksley, Latriece Y., "Interior Point Methods and Kernel Functions of a Linear Programming Problem" (2009). Electronic Theses and Dissertations. 650. https://digitalcommons.georgiasouthern.edu/etd/650
This thesis (open access) is brought to you for free and open access by the Graduate Studies, Jack N. Averitt College of at Digital Commons@Georgia Southern. It has been accepted for inclusion in Electronic Theses and Dissertations by an authorized administrator of Digital Commons@Georgia Southern. For more information, please contact [email protected].
1
INTERIOR POINT METHODS AND KERNEL FUNCTIONS
OF A LINEAR PROGRAMMING PROBLEM
by
LATRIECE Y. TANKSLEY
(Under the Direction of Goran Lesaja)
ABSTRACT
In this thesis the Interior – Point Method (IPM) for Linear Programming problem (LP) that is
based on the generic kernel function is considered.
The complexity (in terms of iteration bounds) of the algorithm is first analyzed for a class of
kernel functions defined by (3-1). This class is fairly general; it includes classical logarithmic
kernel function, prototype self-regular kernel function as well as non-self-regular functions,
thus it serves as a unifying frame for the analysis of IPM. Historically, most results in the
theory of IPM are based on logarithmic kernel functions while other two classes are more
recent. They were considered with the intention to improve theoretical and practical
performance of IPMs. The complexity results that are obtained match the best known
complexity results for these methods.
Next, the analysis of the IPM was summarized and performed for three more kernel functions.
For two of them we again matched the best known complexity results.
The theoretical concepts of IPM were illustrated by basic implementation for the classical
logarithmic kernel function and for the parametric kernel function both described in (3-1).
Even this basic implementation shows potential for a good performance. Better
implementation and more numerical testing would be necessary to draw more definite
2
conclusions. However, that was not the goal of the thesis, the goal was to show that IPM with
kernel functions different than classical logarithmic kernel function can have best known
theoretical complexity.
3
INDEX WORDS: Interior Point Methods, Linear Programming, Kernel Functions, Log Barrier Functions
INTERIOR POINT METHODS AND KERNEL FUNCTIONS
OF A LINEAR PROGRAMMING PROBLEM
by
LATRIECE Y. TANKSLEY
B.S., Savannah State University, 2000
A Thesis Submitted to the Graduate Faculty of Georgia Southern University in Partial
Fulfillment of the Requirements for the Degree
MASTER OF SCIENCE
STATESBORO, GEORGIA
2009
4
© 2009
LaTriece Y. Tanksley
All Rights Reserved
5
INTERIOR POINT METHODS AND KERNEL FUNCTIONS
OF A LINEAR PROGRAMMING PROBLEM
Major Professor: Goran Lesaja
Committee: Billur Kaymakcalan
Scott Kersey
Yan Wu
Electronic Version Approved:
May 2009
6
ACKNOWLEDGEMENTS
Mathematics has always been of great interest to me. I cannot begin to express the joy and
pleasure that I’ve experienced studying at Georgia Southern University under the direction of
the great faculty here. Dr. Goran Lesaja has played an intricate role in my graduate research. I
could not have made it to this point without his advisement and knowledge. I would like to
thank Dr. Yan Wu and Dr. Billur Kaymakcalan for serving on the graduate committee. I have
had the pleasure of taking previous classes from both instructors and want to thank them for
their encouragement throughout my academic career. I would also like to thank Dr. Scott
Kearsey for serving on the graduate committee. I have not had the pleasure of knowing him as
well as the others, but I appreciate his interest in my graduate work. I would also like to thank
graduate student Jie Chen and Segio Valdrig for their generous help with the implementation
of the algorithms in MatLab. I would like to thank my parents, Algene and Dorothy Tanksley,
for their support throughout this long journey. They have been great role models in my life
and this accomplishment is a direct result of their hard work and commitment towards my
academic success. Finally, I would like to thank the math professors at Savannah State
University who first encouraged me and guided me through undergraduate studies in math. I
am eternally grateful to them for their leadership and interest in my studies.
7
TABLE OF CONTENTS
Page
ACKNOWLEDGEMENTS ........................................................................................................6
LIST OF FIGURES .....................................................................................................................8
LIST OF TABLES.......................................................................................................................9
NOMENCLATURE ..................................................................................................................10
CHAPTER
1 INTRODUCTION ........................................................................................................12
2 INTERIOR POINT METHODS BASED ON KERNAL FUNCTIONS .....................18
3 ANALYSIS OF THE ALGORITHM FOR A CLASS OF KERNEL FUNCTIONS . 27
4 ANALYSIS OF THE ALGORITHM FOR ADDITIONAL KERNAL FUNCTIONS50
5 NUMERICAL RESULTS ............................................................................................68
6 CONCLUSION.............................................................................................................74
REFERENCES ..........................................................................................................................76
APPENDICES
APPENDIX A.................................................................................................................77
APPENDIX B.................................................................................................................82
8
LIST OF FIGURES
Figure 1: Graphical Interpretation of IPM.................................................................................21
Figure 2: Generic Primal-Dual Interior-Point Algorithm for Linear Optimization...................26
Figure 3: IPM Based on generic kernel function .......................................................................70
9
LIST OF TABLES
Table 1: Wyndor Glass Company Data .....................................................................................14
Table 2: Complexity results for long-step methods...................................................................64
Table 3: Complexity results for short-step methods ..................................................................65
Table 4: Numerical results for Example ....................................................................................69
Table 5: Numerical results for randomly generated “small problems”
with dimension less than 10...................................................................................70
Table 6: Numerical results for randomly generated problems
with dimension 200x300........................................................................................71
Table 7: Numerical results for randomly generated problem
with dimension 400x700........................................................................................71
Table 8: More Numerical Results ..............................................................................................72
Table 9: More Numerical Results ..............................................................................................72
10
NOMENCLATURE
nR Euclidean n dim space
nR All vectors of nR with nonnegative components
Ax Element x belongs to set A
x A Euclidian norm of vector nRx ,
n
i
nixx
1
,0 converges to 0
RRf n : A function with n variables
f , f2 A gradient and a Hessian of f
mn RRF : A vector valued function
F A Jacobian of F
)(xdiagX A diagonal matrix that has components of the vector x on
the main diagonal and zeros everywhere else
1,, xs
xxs Component wise operations (product, division, inverse) of
vectors nRsx , . For example, ),,( 11 nn sxsxxs .
))(( nfo There exist a constant C and a function )(ng such that
)()( nCgnf (“small o” notation).
11
))(( nfO There exist a constant C and a function )(ng such that
)()( nCgnf
))(( nf There exist constants KC, and a function )(ng such that
)()()( nKgnfnCg (“big ” notation).
12
CHAPTER 1
INTRODUCTION
Linear Programming Model
The mathematical model of linear programming is useful in solving a wide range of
problems in industry, business, science and government. This is by far the most used
optimization model.
These problems and their linear programming models can often be complex as they
consist of a huge number of variables and constraints ranging up to the hundred thousands. A
linear programming model consists of an objective function, that is a linear function, and the
constraints on that function that are also linear. Linear programming involves the planning of
activities to obtain a result that reaches a specified goal among all feasible alternatives.
A Linear Program (LP) is a problem that can be expressed in standard form as follows:
Minimum xcT
subject to 0
x
bAx(1-1)
where nRx is the vector of variables to be solved for, and matrix mxnRA and vectors
mn RbRc , are input data. . The linear function xcZ T is called the objective function,
and the equations bAx are called functional constraints, while 0x are called nonegativity
constraints. The set 0,| xbAxRxF n is called a feasible set. Geometrically, the set
represents a polyhedron in nR .
13
Many practical problems can be modeled as LP models. To illustrate this fact we list
the following simple example taken from the [Hillier, Lieberman, 2005].
Example: The Wyndor Glass Co. produces high-quality glass products, including
windows and glass doors. It has three plants. Aluminum frames and hardware are made in
Plant 1, wood frames are made in Plant 2, and Plant 3 produces the glass and assembles the
products. Because of declining earnings, top management has decided to revamp the
company’s product line. Unprofitable products are being discontinued, releasing production
capacity to launch two new products having large sales potential:
Product 1: An 8-foot glass door with aluminum framing
Product 2: A 4 x 6 foot double-hung wood-framed window
Product 1 requires some of the production capacity in Plants 1 and 3, but none in Plant 2.
Product 2 needs only Plants 2 and 3. The marketing division has concluded that the company
could sell as much of either product as could be produced by these plants. However, because
both products would be competing for the same production capacity in Plant 3, it is not clear
which mix of these two products would be most profitable. Each product will be produced in
batches of 20, so the production rate is defined as the number of batches produced per week.
Any combination of production rates that satisfies the restrictions is permitted, including
producing none of one product and as much as possible of the other. Profit from selling one
batch of Product 1 (glass doors) is $3000 and profit from selling one batch of Product 2
(windows) is $5000. We assume that all produced batches will be sold. Goal: To determine
what the production rates should be for the two products in order to maximize the total profit,
subject to the restrictions imposed by the limited production capacities available in the three
plants.
14
Of course, this is a simplified real world situation but good enough to illustrate the usefulness
of the model. Formulation of the Linear Programming (LP) Problem
Let1x = number of batches of product 1 produced per week
2x = number of batches of product 2 produced per week
Z = total profit per week in thousands of dollars from producing these two products
The following table summarizes the data gathered:
Wyndor Glass Company Data
Plant Production Time Per Batch Hours Production Time Available
per Week, Hours
Product
1 2
1
2
3
1 0
0 2
3 2
4
12
18
Profit per Batch 3000 5000
Table 1
The objective function is 21 53 xxZ and it represents a total profit measured in thousands of
dollars. The objective function is subject to the restrictions imposed by the limited production
capacities available in each of the plants, and they can be mathematically expressed by the
following inequalities:
15
18233
1222
41
21
2
1
xxPlant
xPlant
xPlant
Thus, the overall linear programming model illustrating the Wyndor Glass Company is
0,0
1823
122
4
53
21
21
2
1
21
xx
xx
x
xtosubject
xxZMaximize
By adding slack variables this problem can be transformed in the standard form (1-1).
0,0,0,0,0
1823
122
4
53
54321
521
42
31
21
xxxxx
xxx
xx
xxtosubject
xxZMaximize
The similar procedure can be done for different inequality formulations of LP.
This example illustrates the applicability of the LP model. The number of problems
that can be modeled as LP is huge and widespread to many areas of science, industry,
business, finance, government, etc. For more examples see [HL] and other Operations
Research textbooks. Therefore, the efficient methods to solve LP models are very important.
In the following sections we will outline main methods that are used to solve LP models.
Methods to Solve LP Models
The first successful general procedure, Simplex Method, for solving a LP problem was
discovered by George Dantzig in 1947 although there were partial results discovered earlier.
Theoretically the main idea of the simplex method (SM) is that it travels from vertex to vertex
16
on the boundary of the feasible region, repeatedly increasing or decreasing the objective
function until either an optimal solution is found, or it is established that no solution exists.
The number of iterations required in the worst case is an exponential function of the number
of variables, as it was first discovered by Klee and Minti in 1972. However, the worst case
behavior has not been observed in practice. On the contrary, the algorithm works very well in
practice, typically requiring )(nO iterations. Highly sophisticated implementations are
available (CPLEX, MOSEK, LINDO, EXCEL SOLVER) and have excellent codes for
simplex algorithms. These codes are capable of solving huge problems with millions of
variables and thousands of constraints. This discrepancy between exponential worst case
complexity and good practical behavior of simplex method prompted the research in two
directions. One direction was a search for the algorithm with the polynomial worst case
complexity and the other direction was the analysis of average complexity of simplex method.
In 1979, Leonid Khaciyan showed that the Ellipsoid Method, created by A.
Nemirovski and D. Yudin for nonlinear programming problems, solves any linear program in
a number of steps which is a polynomial function of the amount of data defining the linear
program. Unfortunately, in practice, the simplex method turned out to be far superior to the
ellipsoid method. However, theoretical importance is significant because it provided a basis to
prove that polynomial methods exist for many combinatorial problems.
In 1982 K. Borgward provided the first probabilistic analysis of the simplex method,
showing that the expected number of iterations is polynomially bounded. Soon afterwards,
other authors provided similar analysis. A relatively simple and complete analysis was
provided by Adler and Megiddo in 1985. Using the clever probability model, they showed that
17
upper and lower bounds on an average number of iterations is a function of 2}),(min{ nm ,
where m is the number of constraints and n is the number of variables.
In 1984, Narendra Karmarkar introduced an Interior-Point Method (IPM) for linear
programming, combining the desirable theoretical properties of the ellipsoid method and
practical advantages of the simplex method. Its success initiated an explosion in the
development of interior-point methods that continue to this day.
These methods do not pass from vertex to vertex along the edges of the feasible region,
which is the main feature of the simplex algorithm; they follow the central path in the interior
of the feasible region. Though this property is easy to state, the analysis of interior-point
methods is a subtle subject which is much less easily understood than the behavior of the
simplex method. Interior-point methods are now generally considered competitive with the
simplex method in most, though not all, applications, and sophisticated software packages
implementing them are now available (CPLEX, MOSEK, LINDO, EXCEL SOLVER).
18
CHAPTER 2
INTERIOR POINT METHODS BASED ON KERNEL FUNCTIONS
The linear optimization problem in standard form is
(P) 0,:min xbAxxcT ,
where nmnxm RcRbmArankRA ,),)(( . The dual problem of (P) is
(D) 0,:max scsyAyb TT ,
where nRs is a dual slack variable.
We can assume that the Interior Point Condition (IPC) is satisfied without loss of
generality; that is, there exists a point ),,( 000 ysx such that 0, 00 xbAx and
0,00 scsyAT , which means that the interiors of the feasible regions of the primal(P)
and dual(D), are not empty. If the problem doesn’t satisfy the IPC, it can be modified so that it
does and even in such a way that esx 00 , where e denotes a vector of all ones. The
details can be found in [Roos, C et. al., 1997].
Optimality conditions for (P) and (D) yield the following system of equations:
,0
0,
0,
xs
scsyA
xbAxT (2-1)
where the vector xs denotes the component wise product of vectors x and s which is also
called Hadamard product.
19
The theory of interior point methods (IPMs) that is based on the use of Newton’s
Method suggests that the third equation in (2-1) has to be perturbed. The third equation is
often called the complementarity condition for the primal and dual, and is replaced by
exs , where is a positive parameter. The optimality conditions (2-1) are transformed to
the following system:
.
0,
0,
exs
scSyA
xbAxT
(2-2)
Since rank (A) = m, this system has unique solution for each >0. We can write this solution
as )(),(),( syx , calling )(x the center of (P) and ))(),(( sy the center of (D).
The set of all centers forms a homotopy path in the interior of the feasible region that is
called the central path.
The main property of the central path can be summarized as follows: if ,0 then
the limit of the central path exists, the limit points satisfy the complementarity condition, and
the limit yields optimal solutions for (P) and (D). This limiting property of the central path
leads to the main idea of the iterative methods for solving (P) and (D): Trace the central path
while reducing at each iteration. However, tracing the central path exactly would be too
costly and inefficient. One of the main achievements of interior point methods was to show
that tracing the central path approximately while still maintaining good properties of the
algorithms is sufficient.
Tracing the central path means solving the system (2-2) using Newton Method on the
function
20
.0),,(
exs
csyA
bAx
syxF T
(2-3)
Tracing the central path approximately means that only one, or at most, a couple of iterations
of a modified (damped) Newton’s Method will be performed for a particular . One iteration
of a Newton Method for the function (2-3) and particular is stated below.
),,( syxF
z
y
x
F
(2-4)
where
XS
IA
A
F T
0
0
00
denotes the Jacobian of F and Tzyx ,, is a Newton’s search direction that we want to
calculate.
Solving (2-4) reduces to solving this system.
xsesxxs
syA
xAT
0
0
(2-5)
Next we update the current iterates syx ,, by taking an appropriate step along the calculated
direction.
21
sssyyyxxx ,,
The step size α has to be chosen approximately, so that the new iterate is in a certain
neighborhood of µ-center. The choice of will be discussed later in the text.
The idea of the algorithm is illustrated in the Figure 1 blow.
Graphical Interpretation of IPM
Figure 1
In order to generalize the algorithm outlined above, we introduce a new vector
Optimal solution
=0)
Central path
Neighborhood
neighborhoo
d
of the
approximate
solution
-centersIterates Directions with
step-size
Feasible region
22
xs (2-6)
which we use to define new scaled directions
s
sd
x
xd sx
:,: (2-7)
where the operations in (2-7) are component-wise product and division of vectors. Using the
above definitions (2-6) and (2-7), the system (2-5) reduces to
1
0
0
sx
s
x
dd
dyA
dA
(2-8)
where )(),(:,1 1 xdiagXvdiagVXAVA
. Note that )(xdiagX denotes a
diagonal matrix that has components of the vector x on the main diagonal and zeros
everywhere else.
The new search direction syx ,, is obtained by first solving the system (2-
8).Once xd and sd are found we apply (2-6) to find x and s . This direction can also be
obtained directly by solving the following system:
(2-9) .0
0
sxxs
syA
xAT
23
This system can be reduced to
ryM (2-10)
where
)(1
1
ASr
XAASM T
(2-11)
and xdiagX , sdiagS , diagV .
Once y is found, s and x are found by back substitutions
yAs T (2-12)
sxsx 1 (2-13)
where products denote component-wise products of vectors.
The following observation is crucial for the generalization of the method. Observe that
1 is a gradient of the following function
.log2
1log
2
1log
2
1 2
2
21
1
2
n
nn
ii
ic
(2-14).
This function is called log-barrier function.
One may easily verify that the Hessian )()( 22 ediagc . Since this matrix is
positive definite, )(c is strictly convex. Moreover, since 0)( ec , it follows that
24
)(c attains its minimal value at e , with 0)( ec . Thus, it follows that )(c is
nonnegative everywhere and vanishes if and only if e , that is, if and only if
)()( ssandxx . Hence, we see that the )()( sandxcenters can be characterized
as the minimizers of the function )(c . Therefore, the function )(c serves as a proximity
measure to the -centers (central path.. The norm based proximity measure that is derived
from )(c is defined as
.)(2
1 c (2-15)
Furthermore, the complimentary equation in (2-8) can be written as
)(csx dd , which is also called the scaled centering equation. The importance of
the equation arises from the fact that it essentially defines the search directions. Since xd and
sd are orthogonal, we will still have 0sxdd if and only if e . The same is true for x
and s .
The main idea of the generalization of the method is to replace the log-barrier function
(2-14) with some other barrier function that has the same properties as log barrier. The choice
of this function will certainly affect the calculation of the search direction, the step size, and
with that the rate of the convergence of the method. It is worth examining the classes of
barrier functions that may lead to the improved behavior of the algorithm. In what follows, we
will consider several such classes.
We will restrict ourselves to the case where the barrier function )( is separable with
identical coordinate functions )( i . Thus,
25
)()(1
n
ii , (2-16)
where ),0[:)( t is twice differentiable and attains its minimum at 1t , with 0)1( . The
function )(t is called a kernel function. The log-barrier function belongs to this class.
The algorithm based on generic kernel function that was outlined above is summarized
in the Figure 2 below. In principle, each barrier function gives rise to a different primal-dual
algorithm. The parameters τ,θ and the step size α in the algorithm should be tuned in such a
way that the number of iterations required by the algorithm is as small as possible. The
resulting iteration bound will depend on the kernel function, and our main task becomes to
find a kernel functions that give a good and possibly best known iteration bound. The question
of finding the kernel function that minimizes the iteration bound is still an open question.
26
Figure 2 Generic IPM
Generic Primal-Dual Interior-Point Algorithm for Linear Optimization
Input:
An input data A, b, c
A threshold parameter ;1
An accuracy parameter 0 ;
A fixed barrier update parameter 1 ;
Iteration:
begin
1:;:;: esex ;
while n do
begin
calculate )1(: ;
calculate ;:
xs
while do
begin
calculate the direction syx ,,
using (2-10) – (2-13) :
calculate step size ;
update
yyysssxxx :,:,: ;
end
end
end
27
CHAPTER 3
ANALYSIS OF THE ALGORITHM FOR A CLASS OF KERNEL FUNCTIONS
The following class of kernel functions will be used to analyze the algorithm, Generic
IPM, discussed in the Chapter 2, Figure 2 .
1.0,0,log1
1
1,1.0,0,1
1
1
1
)(1
11
,
pttp
t
qptq
t
p
t
tp
qp
qp (3-1)
where p is a growth parameter and q is a barrier parameter.
Notice that tq
t q
log1
11
when 1q . This class of kernel functions is fairly general. As
we just explained, it includes log-kernel function as a special case. It also includes so-called
self-regular kernel functions when 1p . These functions have been extensively discussed in
recent literature (Peng, J, et. al, 2002). Moreover, it also includes non self-regular functions
when 10 p . This class of functions was first discussed in (Lesaja G., et.al., 2008). The
results in this chapter follow the results presented in that paper. However, the proofs of several
results that were omitted in the paper are outlined here.
Properties of Kernel Functions
The derivatives of )(t in (3-1) play a crucial role in our analysis. Thus, we write
down the first three derivatives:
28
.)1()1()(
)(
)('
22'''
11''
qp
qp
qp
tqqtppt
qtptt
ttt
(3-2)
In the next several lemmas we will describe certain properties of kernel function and its
derivatives and their relationships in terms of inequalities. These results will be used in the
analysis of the Generic IPM.
The following lemma states the so-called exponential convexity of kernel function which is
crucial in proving the polynomial complexity of the Generic IPM.
Lemma 3.1 212121 2
1,00 ttttthentandtIf .
Proof: It can be shown that the inequality in the lemma holds if and only if
00)(')('' tallforttt . This result is beyond the scope of the thesis and can be found
(Peng, J., et. al., 2002). Using (3-1) one can easily verify that
0
1)1(
1)(')(
11''
q
pq
pq
p
t
qtp
tt
t
qpttttt ,
which completes the proof.
Lemma 3.2 If 1t , then 2)1(2
)()1(2
)('
t
qptt
t .
Proof: If ),(')1()(2)( ttttf then )('')1()(')(' ttttf and
)(''')1()('' tttf . Also 0)1( f and 0)1(' f . Since 0)(''' t it follows that if
29
1t then 0)('' tf whence 0)(' tf and 0)( tf . This implies the first inequality. The
second inequality follows from Taylor’s theorem and the fact that qp )1('' .
Lemma 3.3 Suppose that )()( 21 tt with 21 1 tt . The following statements hold:
i. One has 0)(',0)(' 21 tt , and )(')(' 21 tt .
ii. If 1 , then )()( 21 tt ; equality holds if and only if 1 or 121 tt .
Proof: Proof of (i):
The statement is obvious if 11 t or 2t =1 because then 0)(')( 21' tt implies 121 tt
Thus we may assume that .1 21 tt Suppose the opposite )(')(' 21 tt . By the mean
value theorem we have
),()1()(' 2''
11 tt for some ),1( 22 t
and
- ),()1()( 1''
22' tt for some )1,( 11 t .
Since '' (t) is monotonically decreasing one has )()( 2''
1'' . Then we obtain
,)1()1()1( 2''
11''
12''
2 ttt
Hence since '' 2 >0 it follows that 12 11 tt . Using this and the fact that )(' t is
convex, we may also write
30
).(
)(
)('12
1
)(12
1
)(12
1
)(
)(
1
1 '
11
2'
1
2'
2
1
'
2
1
2
t
d
tt
tt
tt
d
t
t
t
This contradiction proves the first part of the lemma.
Proof of (ii):
Consider, ).()( 12 ttf
One has 0)1( f and ).()( 1'
12'
2' ttttf
Since 0)('' t for all ,0t )(' t is monotonically increasing. Hence, )(')(' 21 tt .
Substitution gives
0))((')(')(')(')(')(' 12221221122 tttttttttttf
The last inequality holds since fortandtt 0)('12 .1t This proves that 0f for
,1 and hence the inequality (ii) in the lemma follows.
If 1 obviously we have equality. Otherwise, if 1 and 0)( f , then the mean value
theorem implies 0)(' f for some ),1( . But this implies )()( 1'
2' tt . Since )(' t
Since )(' t is concave,
Since 0)(11 2'
12 tandtt ,
Since )(')(' 21 tt ,
Since )(' t is concave.
31
is strictly monotonic, this implies that 12 tt , whence 21 tt . Since also 21 1 tt we
obtain 121 tt . This completes the proof of the second part of the lemma.
Lemma 3.4 If 1t , then )('')(2)(' 2 ttt .
Proof : Defining )('')(2)(')( 2 ttttf one has 0)1( f and
0)(''')(2)(''')(2)('')('2)('')('2)(' tttttttttf .
This proves the lemma.
Lemma 3.5 Let ]1,0(),0[:)( s be the inverse function of 1)('2
1 tfort . The
following inequality holds:
qs
s1
)21(
1)(
. (3-3)
Proof: Since )('2
1ts , we have
sstttts pqqp 2122 .
Since )1(t , this implies the lemma.
Lemma 3.6 If 1t and pq 2 , then )(1 ttt .
Proof: Defining 21)( ttttf we have 0)1( f and
.12' ' tttttf
32
Moreover, it is clear that 01' f and
.022222'''2 9'' qpqpp ttptqpttqtpttttf
The second inequality above is due to the fact that q p 2 . Thus we obtain
21 ttt ,
which implies the lemma.
Lemma 3.7 Let ),1[),0[: be the inverse function of )(t for 1t . The following
inequalities hold:
sssssp p 21)(11 21
1
. (3-4)
If pq 2 , then
ssssss 21)( 22 (3-5)
Proof: Since q>1 and t1, we have
1
1
1
1
1
1 111
p
t
q
t
p
tts
pqp
Hence, the first inequality in (3-4) follows.
The second inequality in (3-4) follows by using the first inequality of Lemma 3.2:
.21
2
1111
2
11
2
11
2
1 '
tt
tttttttts qp
33
Hence, solving the following inequality
01122 tst
leads to
.21 2 sssst (3-6)
Finally, let .2 pq By Lemma 3.6 one has .11 tsttt
Substitution of the upper bound for t given by (3-6) leads to
.21 22 ssssss
This completes the proof of the lemma.
Now we will derive a very important bound for normed proximity measure 2
1
in terms of the original proximity measure given by the barrier function )( .
Theorem 3.1 The following inequality holds:
'2
1)( . (3-7)
The proof is beyond the scope of thesis and can be found in (Peng, J., et. al., 2002)
Corollary 3.1 If 1)( , then p
p
1)(6
1)(
34
Proof: Using Theorem 2.1, and the fact that 1)( , we have
1
2
1
2
1'
2
1)( pqp . Note that
tt p 1 is monotonically increasing in t . Thus, by using the first inequality in (3-4), we obtain
1
1
1
1
1
1
1
1
11
)(6
1
)(3
)(
2
1
)(11
)()1(
2
1
)(11
1)(11
2
1
(
1)(
2
1)(
p
p
pp
p
p
pp
p
p
p
pv
Which proves the corollary.
Analysis of the algorithm
The outline of the analysis of the algorithm is as follows.
1. Outer iteration estimates:
Estimate of the increase of the barrier function after the µ update
2. Inner iteration estimates:
Estimator for default step size
Estimate of the decrease of the barrier function during the inner iteration with the
default step size
35
Outer Iteration Estimates
At the start of each outer iteration of the algorithm, just before the update of the
parameter µ with the factor 1 , we have )( . Since the µ vector is updated to
)1( , with 10 , the vector is updated to
1, which in general leads
to an increase in the value of )( . Then, during the subsequent inner iterations, )(
decreases until it passes the threshold τ again. During the course of the algorithm the largest
values of )( occur just after the updates of µ. That is why we need to derive an estimate
for the effect of a µ-update on the value of )( .
Theorem 3.2 Let ),1[),0[: as defined in Lemma 3.7. Then for any positive vector
and any 1 the following inequality holds:
nn
)()(
.
The proof is beyond the scope of this thesis and can be found in [Peng, J., et. al., 2002].
Corollary 3.2 Let 10 and
1
. If , then
2
112
)(
1)(
nnqpn
n . (3-8)
36
Proof: With 1 and 1)( the first inequality follows from Theorem 3.2. The second
inequality follows by using Lemma 3.2 and q
qp )1('' .
The following upper bounds on the value of )( after the µ-update follow immediately
;1,)1(
21
:)(
2
1
qnnn
nL
(3-9)
and
.2,1
21
:)(2
2
2
2
2 pqnnnnn
nL
(3-10)
Default Step-size
In this subsection, we will determine a default step size which not only keeps the
iterations feasible but also gives rise to sufficiently large decrease of )( in each inner
iteration. During an inner iteration, the parameter µ is fixed. After the step in the direction
syx ,, with step size α, the new iterate is
yyyxssxxx ,, (3-11)
And a new -vector is given by
37
sx
. (3-12)
Since
2
),(
),(
xs
dusd
ess
sess
duxd
exx
xexx
ss
xx
we obtain
sx dd .
Next, we consider the decrease in as a function of . We define two functions
f , (3-13)
and
dsdf2
1:1 . (3-14)
Lemma 3.1 implies that
sxsx dddd 2
1.
The above inequality shows that )(1 f is an upper bound of )(f . Obviously,
0)0()0( 1 ff . Taking the derivative with respect to , we get
38
n
isisiixixii ddddf
11 ''
2
1)(' .
From the above equation and using that )( sx dd we obtain
21 )(2)()(
2
1)()(
2
1)0(' vvddvf T
sxT . (3-15)
Differentiating once again, we get
0''''2
1)('' 22
11
sixi
ddddf sii
n
ixii , unless 0 sx dd .
(3-16)
It is worthwhile to point out that during an inner iteration sandx are not both at the center
since 0)( v , so we may conclude that )(1 f is strictly convex in .
Lemma 3.8 The following inequality holds:
2''2)('' min2
1 f . (3-17)
Proof: Since sx dd , and sx dd it is easy to see that .2||,|| sx dd where
)(2
1| Therefore, we have 2|||| xd and .2|||| sd Hence,
,2min aad xi ,2min aad si .1 ni
Using (3-16) and definition of , we get
n
isixi addaaf
1min
"222min
"1 )2(22"
2
1)( .
39
This proves the lemma.
Lemma 3.9 If the step-size satisfies
2)('2' minmin , (3-18)
then 0)('1 f .
Proof: Using the Lemma 3.8,(3-15) and (3-17), we write:
.022
)2(2
2)2(2
)2(22
)()0()(
22
min'
min'2
minmin"2
min"22
"1
'1
'1
d
d
dffaf
a
o
a
o
a
o
This proves the lemma.
Lemma 3.10 The largest possible value of the step-size satisfying the condition of Lemma
3.9 is given by
)2()(2
1:
(3-19)
Proof: We want such that (3-18) holds with as large as possible. Let us denote min as
.1 Since t'' is decreasing the derivative with respect to .1 of the expression which is to
the left side of the inequality (3-18) (i.e. )2 1''
1'' is negative. Hence, fixing ,
the smaller 1 is, the smaller will be. We have
40
.2
1||
2
1||||
2
11
'1
'
Equality holds if and only if .1 is the only coordinate in that differs from 1 and 11 (in
case 01' ). Hence the worst situation for the step size occurs when 1 satisfies
.2
11
' (3-20)
The derivative with respect to of the expression that is the left side of the inequality (3-18)
equals 022 1'' and hence this expression is increasing in Thus, the largest
possible value of satisfying (3-18), satisfies
.222
11
' (3-21)
Due to the definition of , (3-20) and (3-21) can be written as
)2(2),( 11 .
This implies
))2()((2
1))2((
2
11
and the lemma is proved.
Lemma 3.11 Let be defined by (3-19) . The following inequality holds:
2''
1 . (3-22)
41
Proof: By definition of ,
2'
Taking the derivative with respect to , we find
2'"
which leads to
0''
2'
. (3-23)
Hence, is monotonically decreasing. An immediate consequence of (3-19)) is
2
2
'
''
1
2
1 dd (3-24)
where we also used (3-23). To obtain a lower bound for , we want to replace the argument
of the last integral by its minimal value. We would like to know when " is a maximal
for .2, . We know that " is monotonically decreasing. Thus " is maximal for
2, when is minimal. Since is monotonically decreasing this occurs when
2 . Therefore
,2"
1
2
1
''
1"
2
d
and the lemma is proved.
42
Theorem 3.3 We have q
q
qp
1
)41)((
1:~
(3-25)
Proof: Using Lemma 3.11, and the fact that )('' t is monotonically decreasing for
),0( t we have
~
)41)((
1
)41()41(
1
))2((''
1111
q
q
q
q
q
q
qpqpp
and the theorem is proved.
Thus, we can define the following default step-size
q
q
qp1
)41)((
1:~
, (3-26)
Inner Iteration Estimates
Using the lower bound on the step size obtained in (3-25), we can obtain results on the
decrease of the barrier function during inner iteration.
Lemma 3.12 If the step size is such that , then 2)( f . (3-27)
Proof: Let the univariate function h be such that
0)0(0 1 fh , 2'1 2)0(' fh , aah 2"2)( min
2''
43
According to (3-17) we have "" hf and that implies '' hf and hf .
Taking ~ , with ~ defined in (3-26)), we have
0'2'22"22' minmin2
0
min22
dh .
Since t''' decreasing in )(, '' ht is increasing . Using Lemma 3.13 below, we get:
2'1 )0(
2
1)()( ahahaf
.
As we mentioned before, 1f is an upper bound of af , hence, the lemma is proved.
Theorem 3.4 The following inequality holds:
)1(
)1(
)()(60
1)~(
pq
qp
qpf . (3-28)
Poof: According to Lemma 3.12, if the step-size is such that , then 2)( f . By
(3-25) the default step-size ~ satisfies ~ , hence, the following upper bound for )(f is
obtained 2~)~( f . Using Corollary 3.2 and the fact that )~(f is monotonically
decreasing in , we obtain
)~(fq
q
qp1
2
)41)((
44
q
q
p
p
p
p
qp
1
1
1
2
)(3
21)(36
)(
)1(
)1(
)()(60
1
pq
qp
qp
Thus, the proof is complete.
Estimate of the total number of iterations
As we’ve already mentioned, there are two types of algorithms.
The Short-step Algorithms, where the barrier update parameter θ depends on the size of
the problem; that is,
nO
1 .
The Long-step Algorithms, where the barrier update parameter θ is fixed; that is,
)1,0( .
We will give the estimate of the total number of iterations needed for both types of algorithms.
We will need following technical results. The proofs can be found in (Peng, J., et. al., 2002)]
Lemma 3.13 If ]1,0[ and 1t then tt 1)1( .
45
Lemma 3.14 Let )(th be twice differentiable convex function with ,0)0(',0)0( hh and
let )(th attain its (global) minimum at .0* t . If )('' th is monotonically increasing for
*},0{ tt then *0,2
)0(')( tt
thth .
Lemma 3.15 Let kttt ,..., 10 be a sequence of positive numbers such that
1...1,0,11 Kktt kkk
where
0.100
tKThenand .
Long-step Algorithms
Lemma 3.16 The total number of outer iterations in both cases is the same.
n
log1
(3-29)
Proof: The number of outer iterations is the number of iterations K necessary to obtain
n . Previous and new are related as follows )1(: . Thus, n can be
written as n
K )1(0 . We can assume that 10 and by taking the logarithm of both
sides of the inequality we obtain n
K log)1log( . Using the Taylor theorem for
)1log( we obtain n
K log1
proving the lemma.
Now we need to estimate the upper bound on the total number of inner iterations per one outer
iteration for the large-step methods. That number is equal to the number of iterations
necessary to return to the situation )( . We denote the value of after the update
46
as 0 . The subsequent values in the same outer iteration are denoted as Kkk ,,2,1,
where K denotes the total number of inner iterations in the outer iteration. By using (3-9), we
have
.1
21
2
0
nnn
n
Since 1
11
p
tt
p
when 1t and 2
1
)1(1p
, after some elementary reductions, we
obtain:
2
1
2
0
)1)(1(
2)1(1
p
p
nnpnpn
. (3-30)
Now Theorem 3.4 leads to
1,....,.1,0,11 Kkkkk
, (3-31)
where qp
60
1 and ).1(
pq
qp . Using Lemma 3.15 and (3-30) and (3-31) we obtain
the following upper bound on the number K of inner iterations.
47
)1(
2
1
2
)1(0
)1)(1(
2)1()1(
)1(60))(1(60
pq
qp
ppq
qp
p
nnpnpn
pqpqK
(3-
32)
Now we can derive an upper bound on the total number iterations needed by the large-update
version of the Generic IPM in Figure 2.1. According to Lemma 3.16 the number of outer
iterations is bounded above by
n
log1
By multiplying the number of outer iterations and the number of inner iterations obtained in
(3-32) in the lemma above we get an upper bound for the total number of iterations
n
p
nnpnpn
pq
pq
qp
plog
)1)(1(
2)1()1(
)1(60
)1(
2
1
2
.
For large update methods we know that ),1( and )(nO . After some elementary
transformations the iteration bound reduces to the following bound
n
nqO qpq
qp
log)( (3-33)
This result is summarized in the theorem below
48
Theorem 3.5: Given that 1 and nO which are characteristics of the large-update
methods the Generic IPM described in the Figure 2.1 will obtain - appropriate solutions of
(P) and (D) in at most
n
nqO qpq
qp
log)( iterations.
The obtained complexity result contains several previously known complexity results as
special cases.
1. When 1p and 1q , the kernel function )(t becomes the prototype self-regular
function. If in addition, nq log the iteration bound reduces to the best known bound for
self-regular function, which is
n
nnO loglog .
2. Letting 1p and 1q , the iteration bound becomes
n
nO log and )(t represents
the classical logarithmic kernel function.
3. For 2q and 0p , )(t represents the simple kernel function 21
)( t
tt which
is not self-regular. The iteration bound is the same as the one obtained for the logarithmic
kernel function.
Short-Step Algorithms
To get the best possible bound for short-step methods we need to use the bound described in
(3-10).
49
2
2
2
2
2
2
2
2
2
0 11
21
21
21
nnnnnnqpnnnnnn
Using
11
11 the above inequality can be simplified to
2
2
22
0
2
)1(2
nnn
nqp
(3-34)
Following the same line of arguments as in the above subsection 3.3.1 we conclude that the
total number K of inner iterations is bounded above by
)1(
)(2
2
22)1(
0
2
)1())(1(60
pq
qp
pq
qp
nnnn
qppqK
(3-35)
Given the upper bound on the number of the outer iterations as mentioned in the previous
subsection 3.3.1 the upper bound on the total number of iterations is
.log2
)1(
)(60)1(
)(2
2
22
n
nnnn
qpqpq
qp
(3-36)
50
For small update methods it is well known that
n
1 and )1(O . After some
elementary reductions one easily obtains that the iteration bound is
n
nqO log2 . We
summarize this result in the theorem below.
Theorem 3.6: Given that
n
1 and 1O which are characteristics of the small
update methods the Generic IPM described in the Figure 2.1 we will obtain - appropriate
solutions of (P) and (D) in at most
n
nqO log2 iterations.
51
CHAPTER 4
ANALYSIS OF THE ALGORITHM FOR ADDITIONAL KERNEL FUNCTIONS
Summary of the algorithm analysis
Looking carefully at the analysis of the Generic IPM described in Chapter 3 the
procedure can be summarized in the following way.
Step 0: Input a kernel function ; an update parameter 10, ; a threshold
parameter ; and an accuracy parameter .
Step 1: Solve the equation s '2
1 to get )(s the inverse function of
]1,0(,)('2
1 tt . If the equation is hard to solve, derive a lower bound for )(s .
Step 2: Calculate the decrease of )(t in terms of for the default step-size ~ from
2'')~(
2
f .
Step 3: Solve the equation st )( to get )(s , the inverse function of 1),( tt . If the
equation is hard to solve, derive lower and upper bounds for )(s .
Step 4: Derive a lower bound for in terms of )( by using '2
1v .
Step 5: Using the results of Step 3 and Step 4 find a valid inequality of the form
1)()~(f for some positive constants and ]1,0( .
52
Step 6: Calculate the upper bound of 0 from
2
0 11
)1("21
),,(
nnnnnL .
Step 7: Derive an upper bound for the total number of iterations from
nlog0
.
Step 8: Set )1()( andnO so as to calculate complexity bound for large-update
methods, or set )1(O and
n
1 so as to calculate the complexity bound
for small update methods.
Additional Kernel Functions
We will consider the following additional kernel functions.
.1),1(1
)1(
1
2
1)(
12
1
qtq
q
ttt
q
(4-1)
e
eett
t
12
2 2
1)( (4-2)
53
det
tt
1
112
3 2
1)( (4-3)
The growth term in all of them is the same 2
1)(
2
ttg while the barrier term varies
)(tb . The reason for considering this growth term is that according to the analysis above it
seems to give the best complexity results and, thus, will give a more consistent view of the
complexity analysis.
The following lemmas are useful for the above kernel functions. They are variations of
the similar lemmas in the previous chapter and actually they are also valid for the class of
kernel functions (3-1) used in that chapter. Their main purpose is to help facilitate the
summary analysis described in the previous subsection.
Lemma 4.1 When )()( tt i for 31 i , then sss 21)(21 .
Proof: The inverse function of )(t for ),1[ t is obtained by solving for t from the
equation st )( , for 1t . In almost all cases it is hard to solve this equation explicitly.
However, we can easily find a lower and an upper bound for t and this suffices for our goal.
First one has
2
1)(
2
1)(
22
tt
tts b ,
where )(tb denotes the barrier term. The inequality is due to the fact that 0)1( b and
)(tb is monotonically decreasing. It follows that
54
sst 21)(
For the second inequality we use the fact that 1)('' ti for 31 i . Note that )(ti are
nonnegative strictly convex functions such that 0)1( i . This implies that )(ti is twice
differentiable and therefore is completely determined by its second derivative
ddtt
ii 1 1
)('')( (4-4)
Thus
2
1 11 1
)1(2
1)('')( tddddts
tt
ii
which implies
sst 21)( .
This completes the proof.
Lemma 4.2 Let 31 i . Then one has
1
2
2
)1(",,
2n
nL . Hence, if
)1(O and )1(",1
0 Othenn
.
Prof: By Lemma 4.1 we have ss 21)( . Hence, by using Theorem 3.2 and first
inequality in (3-8) we have
55
1
21
1,,0
nnn
nnL (4-5)
By using Taylor theorem and the fact that 0)1(')1( we obtain
32 )1)(("!3
1)1)(1("
2
1)( tt
Given the fact that 0)(''' we obtain for 1t and t 1
2)1)(1("2
1)( tt (4-6)
Applying (4-6) to (4-5)with
1
21
nt we obtain
1
2
2
)1("
1
2
2
)1("1
1
21
2
)1("2
22
0
nnnnn
where we also used the fact that
11
11 . This proves the lemma.
Lemma 4.3 Let ]1,0(),0[: be the inverse function of the restriction of )(' tb to the
interval ]1,0( where )(tb is the barrier term in the kernel functions 31),( iti . Then
).21()( ss
56
Proof: Let ).(st Due to the definition of as the inverse function of
1)('2
1 tfort this means that
1),(')('2 tttts b .
Since 1t this implies
ssttb 212)('
Since )(' tb is monotonically decreasing in all three cases, it follows that
)21()( sst ,
proving the lemma.
Analysis of the Generic IPM with Additional Kernel Functions
In this subsection we will provide the analysis of the Generic IPM using additional kernel
functions stated in the subsection. We will follow the steps described earlier.
Example
Consider the function
.1),1(1
)1(
1
2
1)()(
12
1
qtq
q
tttt
q
Step 1: The inverse function of q
qttb
11)('
is given by
qsq
s1
))1(1(
1)(
.
57
Hence, by Lemma 6.3, qqs
s1
)21(
1)(
.
Step 2: It follows that
q
q
f 1
)41(1)2(
11))2((''
)(2
1
22
. (4-7)
Step 3: By Lemma 6.1 the inverse function of ),1[)( tfort satisfies
sss 21)(21
Omitting the argument , we therefore have
21))(( .
Step 4: Now using the fact that )))((('2
1)( , and assuming 1 , we obtain
211
1212
1
)21(
11
1121(
2
1qq
. (4-8)
Step 5: Combining (4-7) and (4-8) after some elementary reductions, we obtain
q
q
q
q
q
f 2
12
53
1
)41(1
)~( 1
. (4-9)
Thus, it follows that
1,...,1,0,11 Kkkkk
58
with q53
1 and
q
q
2
1 , and K denotes the number of inner iterations. Hence, by Lemma
3.15 the number K of inner iterations is bounded above by
q
q
q
kK
2
1
00
20 106
1
106
. (4-10)
Step 6: To estimate 0 we use Lemma 6.2, with .2)1(" Thus, we obtain
1
22
0
n.
Step 7: Thus, using Lemma 3.16 the total number of iterations is bounded above by
nnqnK q
q
log1
2106log
2
12
. (4-11)
Step8: For large update methods ( with )1()( andnO ) the right hand side expression
becomes
n
qnO q
q
log2
1
. (4-12)
For small update methods ( with )1(O and
n
1 ) the right hand side expression
becomes
n
nqO log . (4-13)
59
Example
Consider the kernel function
e
eettt
t
12
2 2
1)()( .
Step 1: The inverse function of 2
1
1)('
t
et
t
b
is such that 1,
1)(
2
1
tst
ets
t
. It
follows that sstt
e t
22
1
1whence we obtain
sts
log1
1)(
. Hence, by Lemma4.3,
)21()( ss .
Step 2: Since )(" t is monotonically decreasing we have
)41(''))2(('')~(
22
qf
.
Now putting )41( qt we have 1t and we may write
2
2
2
2
11
4
2
11
4
22
151
)41(313
121
1)("
)(
tte
te
t
ttf
tt
.
Since
222
))41(log1()41(
11
t
we finally get
2
2
))41(log1(151)(
f . (4-14)
60
Step 3: By Lemma 6.1 the inverse function of )(t for ),1[ t satisfies
sss 21)(21 . Omitting the argument , we thus have 21 .
Step 4: Now using that ('2
1)( , we obtain
211121
2
1
2121
2
11
21
1
e . (4-15)
Step 5: Substitution of ((4-15) into the (4-14) gives, after some elementary reductions, while
assuming 10
20
2
1
2
2
1
))21(log1(44))21(log1(44)(
f .
Thus, it follows that
1,...,1,0,11 Kkkkk
with 2
0 ))21(log1(44
1
and
2
1 , and K denotes the number of inner iterations.
Hence, by Lemma 3.15 the number K of inner iterations is bounded above by
2
1
02
00 ))21(log1(88
kK .
Step 6: We use Lemma 4.2 with ,4)1(" to estimate 0 .
61
1
22
2
0
n. (4-16)
Substitution of (4-16) in the expression for K gives
1
2
1
221log1288
2
nnK . (4-17)
Step 7: Thus the total number of iterations is bounded above by
nnnnK
log1
2
1
221log1288log
2
. (4-18)
Step 8: For large update methods, when )1()( andnO the right hand side expression
(4-18) becomes
n
nnO loglog 2 . (4-19)
For small update methods, when )1(O and
n
1 , the right hand side expression
(4-18) becomes
n
nO log (4-20)
Example
Consider the kernel function
62
det
ttt
1
112
3 2
1)()( .
Step 1: The inverse function of 1
1
)('
etb is given by s
slog1
1)(
. Hence, by Lemma
6.3,
)21(log1
1)(
ss
.
Step 2: It follows that
2
2
1
1)2(
1
22
))41log(1)(41(1
)2(1
))2(('')(
qe
f
q
.
(4-21)
Step 3: By Lemma 4.1 the inverse function of )(t for ),1[ t satisfies
sss 21)(21 .
Thus we have, omitting the argument ,
21 .
Step 4: Now using that '2
1we obtain
211121
2
121
2
1 121
1
e . (4-22)
63
Step 5: Substitution of (4-22) into the (4-21) gives, after some elementary reductions, while
assuming 10
20
2
1
2
2
1
))1(log1(21))1(log1(21)(
f .
Thus, it follows that
1,...,1,0,11 Kkkkk
with 2
0 ))21(log1(21
1
and
2
1 , and K denotes the number of inner iterations.
Hence, by Lemma 3.15 the number K of inner iterations is bounded above by
2
1
02
00 ))21(log1(42
kK . (4-23)
We use Lemma 4.2, with 2)1(" to estimate 0 . This gives
1
22
0
n. (4-24)
Substitution of (4-24) into (4-24) leads to the following estimate for number K of inner
iterations
1
2
1
21log142
2
nnK . (4-25)
Step 7: By Lemma 3.16 the total number of iterations is bounded above by
64
nnnnK
log1
2
1
21log142log
2
(4-26)
Step 8: For large-update methods when )1()( andnO the right hand side expression
(4-26) becomes
n
nnO loglog 2 . (4-27)
For small update methods, when )1(O and
n
1 , the right hand side expression
(4-26) becomes
n
nO log . (4-28)
Summary of complexity results
The complexity results for Generic IPM with kernel functions defined in (3-1) and in (4-1)-(4-
3) are summarized in the Table for large-step methods and in the table for small-step methods.
For the class of kernel functions (3-1) we consider three special cases,
the logarithmic kernel function: tt
t log2
1)(
2
the classical self-regular function when 1p : 1
1
2
1)(
12
q
ttt
q
65
the linear non-self-regular function when 0p :1
11)(
1
q
ttt
q
Complexity of large-update methods
The complexity results for large-step methods are summarized below. They are obtained by
taking into the account that )1()( andnO .
Complexity results for long-step methods
i Kernel Function )(ti Iteration Bound
1t
tlog
2
12
n
nO log)(
2
1
1
2
1 12
q
tt q
n
qnO q
q
log)( 2
1
3
1
11
1
q
tt
q
n
qnO log)(
4
11
1
1
2
1 12
tq
q
tt q
n
qnO q
q
log)( 2
1
5
e
eet t
1
2
2
1 n
nnO log)log( 2
6 de
t t
1
112
2
1n
nnO log)log( 2
Table 2
Notice that the best bound is obtained in case of 3 and 4 by taking nq log2
1 which gives the
iteration bound of
66
n
nnO loglog , (4-29)
which is currently the best known bound for large-update methods.
Complexity of small update methods
The complexity results for small-update methods are summarized below. They are obtained by
taking into the account that
n
1 and 1O .
Complexity results for short-step methods
i Kernel Function )(ti Iteration Bound
1t
tlog
2
12
n
nO log)(
2
1
1
2
1 12
q
tt q
n
nqO log)( 2
3
11
11
q
tt
q
n
nqO log)( 2
4
11
1
1
2
1 12
tq
q
tt q
n
nqO log)(
5
e
eet t
1
2
2
1 n
nqO log)(
6 de
t t
1
112
2
1n
nqO log)(
Table 3
67
The above table shows that the small-update methods based on listed kernel functions all have
the same complexity, namely
.log
n
nO (4-30)
This is up till now the best iteration bound for IPMs solving LP problems.
Historically most of the IPMs were based on the logarithmic kernel function. Notice that
the gap between theoretical complexity of short-step methods and large-step methods is
significant; the short step methods have much better theoretical complexity. However, in
practical implementations large-step methods work better. This discrepancy was one of the
motivations to consider other kernel functions in hopes to find the kernel function which
would not have a gap or the gap would be smaller. As we can see, this goal has been
achieved; for cases 2 and 4 the gap is much smaller because for these kernel functions large
step method has much better complexity than for the classical logarithmic kernel function.
This is one of the main achievements of considering different classes of kernel functions.
68
CHAPTER 5
NUMERICAL RESULTS
The Generic IPM described in the Figure 1 was implemented in MATLAB 7.6.0 with
the class of kernel functions described by formula (3-1)
1,0,0,log1
1
1,1,0,0,1
1
1
1
)(1
11
,
pttp
t
qptq
t
p
t
tp
qp
qp
This imply that there are two implementations of the algorithm, one for the classical
logarithmic kernel function
tt
t log2
1)(
2
(5-1)
and one for the kernel function with parameters p and q
1,1,0,0,1
1
1
1)(
11
,
qptq
t
p
tt
qp
qp (5-2)
We call the first implementation “Classical Method” and the second implementation “New
Method”. Both codes are listed in the Appendix A.
The algorithm was tested on several examples with different sizes ranging from very
small to moderate size problems. The data was entered in some cases “by hand” and for the
others they were generated randomly.
Example: Consider the following simple LP model.
69
.0,0
3..
22max
21
21
21
xx
xxts
xx
It is easy to see that this problem has infinitely many optimal solutions, they are all the points
on the segment )1,0();0,1( and the optimal value is 6Z . The problem was solved by New
Method with 2.1,8.0 qp with accuracy parameter 001.0 . It took unusually many
iterations (57), however algorithm steadily converged to the expected result
)5,1,5.1(),( 21 xx .
Numerical results for Example
x y s
1 1 1 0 1 1 1
1.383 1.383 0.234
-0.823
4 0.1 0.11.248
9
1.4883
1.4883
0.0234
-1.602
30.065
60.065
61.756
7
.... …. ….. ……. …… …………
…
1.4999
1.4999
0.0002
-2.000
20.000
20.000
22.000
2
Table 4
Objective function value Z = -5.9997
70
This example also illustrates an important feature of the interior-point methods that distinguish
them from simplex-type methods; that in the case of infinitely many optimal solutions they
converge to the center of the optimal set rather than to the vertex. The graphical illustration of
the above example with several first iterations is given below.
IPM Based on generic kernel function
Figure 3
Next, the algorithm was examined on the set of 200 randomly generated “small” problems for with sizes less than 10. The average number of iteration and CPU time is given in the table below.
Classical method New method
Average Number
Of Iterations Time
Average Number
Of Iterations Time
24.6 0.0362 40.7 0.0448
Numerical results for randomly generated “small problems” with dimension less than 10Table 5
71
Next, the algorithm was examined on the set of 200 randomly generated “moderate
size” problems of the size 200x300. The average number of iteration and CPU time is given in
the table below.
Classical method New method
Average Number
Of Iterations Time
Average Number
Of Iterations Time
35.81 2.6626 40 2.8812
Numerical results for randomly generated problems with dimension 200x300
Table 6
The algorithm was then applied to the randomly generated problems of the bigger size
400x700. The result is given in the table below
Classical method New method
Average Number
Of Iterations Time
Average Number
Of Iterations Time
57 35.8048 41 25.4030
Numerical results for randomly generated problem with dimension 400x700
Table 7
Results summarized in tables seem to suggest that Classical Methods works slightly better for
the problems of the smaller size while, as the dimension of the problem increases, the New
Method becomes better. Another feature of the IPM is also visible from these examples and
72
that is that the number of iterations does not increase significantly with the increase in the size
of the problem.
In the sequel the New Method was examined on the set of 200 randomly generated “small”
problems with sizes less than 10 and for different values of parameters p and q. The results
are given in the Table below.
Table 8: More Numerical Results
Next, the New Method was examined on the set of 20 randomly generated problems of
the size 200x300 and for different values of parameters p and q. The results are given in the
Table below.
Table 9: More Numerical Results
73
The table seems to suggest that for the problems of the smaller size the New Method works
the best when 1.1,9.0 qp , which is in line with theoretical expectation . However, for
the problems of the higher dimension it is hard to make conclusion which combination of
parameters works the best. Theory suggests that nqp log,1 where n is the number of
variables gives the best complexity. However for the particular set of problems in the previous
table, it seems that combination 3.1,7.0 qp works the best .
Better implementation and more testing is necessary for more definite conclusions.
However, that was not the intention of the thesis. The goal was to make the basic
implementations that ilustrates theoretical concepts discussed in the thesis. Even on this basic
level the implementation of the New Method shows the potential to work well. Of coure, with
more sophisticated implementation the performance can be further improved.
74
CHAPTER 6
CONCLUSION
In this thesis the Interior – Point Method (IPM) for Linear Programming problem (LP)
that is based on the generic kernel function is considered. The algorithm is described in
Chapter 2.
In Chapter 3 the complexity (in terms of iteration bounds) of the algorithm is analyzed
for a class of kernel functions defined by (3-1). This class is fairly general; it includes classical
logarithmic kernel function, prototype self-regular kernel function as well as non-self-regular
functions, thus it serves as a unifying frame for the analysis of IPMs. Two versions of the
IPMs are considered, the short-step algorithms where barrier parameter depends on the size
of the problem and long-step algorithms where barrier parameter is a fixed constant )1,0( .
Historically most of the IPMs were based on the logarithmic kernel function. Notice
that the gap between theoretical complexity of short-step methods and large-step methods is
significant; the short step methods have much better theoretical complexity. However, in
practical implementations large-step methods work better. This discrepancy was one of the
motivations to consider other kernel functions in hopes to find the kernel function which
would not have a gap or the gap would be smaller. As we can see this goal has been achieved;
for kernel functions 2 and 4, the gap is much smaller than for the classical logarithmic kernel
function. In addition, the complexity results that are obtained match the best known
complexity results for these methods. This chapter is mostly based on the paper (Bai, Y., et al.,
2008) with the addition of most of the proofs that were omitted in the paper.
75
The main contribution of the thesis is contained in Chapter 4. The detailed complexity
analysis of the IPM that was provided in Chapter3 for kernel function (3-1) is summarized and
the analysis of the algorithm was performed for three additional kernel functions (4-1) – (4-3).
For one of them we again matched the best known complexity results for the large-step
methods and for the other two the complexity is slightly weaker, however still significantly
improved in comparison with classical logarithmic kernel function.
The IPM that is theoretically analyzed in Chapter 3 is implemented in Chapter 5 for the
classical logarithmic kernel function (Classical Method) and for the parametric kernel function
(New Method) both described in (3-1). Although the implementation is on the basic level, it
shows potential for a good performance of IPM based on kernel function different than
classical logarithmic kernel function on which most of the commercial codes are based. The
preliminary calculations seem to indicate that IPM with classical kernel logarithmic function
perform better on problems of the smaller size while for larger problems the New Methods
seems to work slightly better. Also, based on the preliminary numerical tests it is hard to make
conclusion which combination of parameters p and q in (3-1) works the best. Better
implementation and more numerical testing would be necessary to draw more definite
conclusions.
However, that was not the goal of the thesis, the goal was to show that IPM with kernel
functions different than classical logarithmic kernel function can have best known theoretical
complexity and to show that they have potential for practical implementations.
76
REFERENCES
[1] Y.Q Bai, C.Roos. A polynomial-time algorithm for linear optimization based on a new
simple kernel function. Optimization Methods Software, 18 (6):631-646, 2003.
[2] Y. Bai, G. Lesaja, C. Roos, G. Wang, M. El Ghami. A Class of Large and Small Update
Primal – Dual Interior-Point Algorithms for Linear Optimization. Journal of Optimization
Theory and Applications, Volume 138, No. 3, 341-359, 2008.
[3] G. Lesaja. Introducing Interior-Point Methods for Introductionary Operations Research
Courses and/or Linear Programming Courses, Open Operational Research Journal, accepted,
2008.
[4] F. Hillier and G. Lieberman. Introduction to Operations Research. Seventh Edition.
McGraw Hill Publishing, 2001.
[5] J. Peng, C.Roos, and T. Telarky. Self- regularity: A New Paradigm for Primal-Dual
Interior Point Algorithms. Princeton University Press, 2002.
[6] C. Roos, T. Telarky, and J.-Ph.Vial. Theory and Algorithms for Linear Optimization. An
Interior-Point Approach. John Wiley &Sons, Chichester, UK, 1997.
[7] S. Wright. Primal-Dual Interior Point Methods. SIAM Publishing, 1997.
[7] Y. Bai, M. El ghami, and C. Roos. A Comparative Study of New Barrier Functions for
Primal – Dual Interior – Point Algorithms in Linear Optimization.
77
APPENDICES
APPENDIX A
MatLab codes for the Classical Method
function [i,xx,yy,ss,z,d]=IpmClassical(A, b, c, epsilon)% % input tau, epsilon, theta% % sizes: A--m*n, b--m, s--n, x--n, y--m, c--n%% To call this function, please set up the problem by defining A, b and% c. Or load the example problem.
[m,n]=size(A); x=1*ones(n,1); s=1*ones(n,1); y=zeros(m,1); mu=x'*s/n; rd=c-A'*y-s; rp=b-A*x;
i=1; xx(i,:)=x; yy(i,:)=y; ss(i,:)=s; while norm(rd)>epsilon||norm(rp)>epsilon||n*mu > epsilon i=i+1; [dx,ds,dy]=SolvesystemClassical(A,b,c,x,y,s,mu); inds=find(ds<0);indx=find(dx<0); alpha=0.9*min(abs([1;s(inds)./ds(inds);x(indx)./dx(indx)])); x=x+alpha*dx; y=y+alpha*dy; s=s+alpha*ds; mu=min(x'*s/n,0.9*mu); rd=c-A'*y-s; rp=b-A*x; xx(i,:)=x; yy(i,:)=y; ss(i,:)=s; d(i,:)=dx'; if i>200 || alpha<1e-11 i; break; end end z=x'*c; %[i,z]end
78
function [dx,ds,dy]=SolvesystemClassical(A,b,c,x,y,s,mu)% This function solves the following system% A*dx = 0% A^T*dy + ds = 0% S*dx + x*ds = - mu*v.*grad(Psi(v))% % Psi(v)=sum((v.^(p+1)-1)/(p+1)+(v.^(1-q)-1)/(q-1));% grad(v)=v.^p-v.^q;
gama=.1; X=diag(x); S=diag(s); S_inv=diag(1./s); rd=c-A'*y-s; rp=b-A*x; M=A*S_inv*X*A'; r=b+A*S_inv*(X*rd-gama*mu*ones(size(A,2),1)); dy=M\r; ds=rd-A'*dy; dx=-x+S_inv*(gama*mu*ones(size(A,2),1)-X*ds); end
MatLab codes for the New Method
function [i,xx,yy,ss,z,d]=IpmNew(A, b, c, epsilon)% %% input tau, epsilon, theta% v>0, 0<=p<=1, q>1, tau>1% sizes: A--m*n, b--m, s--n, x--n, y--m, c--n%% To call this function, please set up the problem by defining A, b and% c. Or load the example problem.
p=1-0.2;q=1+0.2;theta=0.1;%tau=1.5; [m,n]=size(A); x=ones(n,1); s=ones(n,1); y=zeros(m,1); mu=x'*s/n; rd=c-A'*y-s; rp=b-A*x; i=1; xx(i,:)=x; yy(i,:)=y; ss(i,:)=s; while norm(rd)>epsilon||norm(rp)>epsilon||n*mu > epsilon i=i+1; v=sqrt(x.*s./mu); [dx,ds,dy]=SolvesystemNew(A,b,c,x,y,s,mu,v,p,q);
79
delta=1/2*sqrt(sum((v.^p-1./v.^q).^2)); alpha=1/((p+q)*(1-4*abs(delta))^(1+1/q)); alpha=abs(alpha); inds=find(ds<0);indx=find(dx<0); alpha=0.9*min(abs([1;s(inds)./ds(inds);x(indx)./dx(indx)])); x=x+alpha*dx; y=y+alpha*dy; s=s+alpha*ds; rd=c-A'*y-s; rp=b-A*x; mu=(1-theta)*mu; xx(i,:)=x'; yy(i,:)=y'; ss(i,:)=s'; d(i,:)=dx'; if i>200 || alpha<1e-10 i; break; end end z=x'*c; %[i,z]end
function [dx,ds,dy]=SolvesystemNew(A,b,c,x,y,s,mu,v,p,q)% This function solves the following system% A*dx = 0% A^T*dy + ds = 0% S*dx + x*ds = - mu*v.*grad(Psi(v))% % Psi(v)=sum((v.^(p+1)-1)/(p+1)+(v.^(1-q)-1)/(q-1));% grad(v)=v.^p-v.^q;gama=.1; X=diag(x); S=diag(s); S_inv=diag(1./s); rd=c-A'*y-s; rp=b-A*x; M=A*S_inv*X*A'; r=A*S_inv*(X*rd+mu*v.*(v.^p-gama*v.^(-q)))+rp; dy=M\r; ds=rd-A'*dy; dx=-S_inv*(X*ds+mu*v.*(v.^p-gama*v.^(-q)));end
Problem Generator
function R=mytest(NO,m,n)% This function solve the random systems of dimention m*n
80
% input m,n are the dimention of A% input NO is the number of the iterations% % MYTEST(); will apply both ipm methods to a random matrix with random% dimension m*n, where m,n are positive integers less than 10%% MYTEST(N) will iterate ‘MYTEST()’ N times.% % MYTEST(N,m,n) will iterately apply the methods to N random problems% with dimension m*n.
if ~exist(‘NO’) NO=1; end m_ne=0;n_ne=0; if ~exist(‘m’) m_ne=1; end if ~exist(‘n’) n_ne=1; end dim=zeros(NO,2); k=1; while k<=NO if m_ne m=fix(rand(1)*10); end if n_ne n=fix(rand(1)*10); end
A=rand(m,n);b=rand(m,1);c=rand(n,1); if rank(A)<min(m,n) || min(m,n)==0 %fprintf(‘rank(A)<min(m,)’); if m==0 && ~m_ne display(‘STOP, m=0’); return; elseif n==0 && ~n_ne display(‘STOP, n=0’); return; end continue; end tic; [i1(k),x1,yy1,ss1,z1(k)]=IpmNew(A, b, c, 0.01); t1(k)=toc; tic; [i2(k),xx2,yy2,ss2,z2(k)]=IpmClassical(A, b, c, 0.01); t2(k)=toc; dim(k,1)=m;dim(k,2)=n; k=k+1; end R = [dim(:,1),dim(:,2),i1’,z1’,t1’,i2’,z2’,t2’]; myfun®;end
81
Organization of output
function myfun® fprintf(' New method Classical method \n'); fprintf(' idx dim(m*n) iteration Opt time iteration Opt time\n'); fprintf('----------------------------------------------------------------\n'); for k=1:size(R,1) fprintf(' %3d %3d%3d %3d %6.4f %4.4f %3d %6.4f %4.4f\n', ... k,R(k,:)); end R_ave=sum(R)/size(R,1); fprintf('----------------------------------------------------------------\n'); fprintf(' New method Classical method \n'); fprintf(' dim(m*n) iteration time iteration time\n'); fprintf(' average %3.1f %3.1f %3.1f %4.4f %3.1f %4.4f\n', ... R_ave([1,2,3,5,6,8])); fprintf('----------------------------------------------------------------\n');
end
82
APENDIX B
MatLab code Example 5.1
max 3 * x_1 + 5 * x_2
s.t.
x_1 + <=4
2 * x_2 <=12
3 * x_1 + 2 * x_2 <=18
x_1 , x_2 >= 0
A =
1 0 1 0 0
0 2 0 1 0
3 1 0 0 1
(change to min-problem)
c =
-3
-5
0
0
0
b =
4
12
18
83
>> [i,x,y,s,z]=IpmClassical(A, b, c, epsilon)
i =
7
z =
-41.9805
>> [i,x,y,s,z]=IpmNew(A, b, c, epsilon)
i =
60
z =
-41.9993
>> [i,x,y,s,z]=IpmClassical(A, b, c, epsilon)
i =
7
x =
1.0000 1.0000 1.0000 1.0000 1.0000
1.4437 1.7530 0.8831 0.9645 1.0398
2.0334 3.0422 0.7370 0.3825 0.8654
2.9815 4.6039 0.4101 0.0542 0.4968
3.8206 5.7689 0.0779 0.0054 0.1095
3.9803 5.9762 0.0095 0.0020 0.0169
3.9977 5.9975 0.0013 0.0005 0.0028
84
y =
0 0 0
-0.1333 -0.0519 0.0235
-0.2783 -0.5925 -0.1361
-0.5626 -1.3849 -0.4117
-0.8317 -2.0210 -0.6562
-0.9001 -2.1384 -0.6935
-0.9363 -2.1549 -0.6873
s =
1.0000 1.0000 1.0000 1.0000 1.0000
0.4093 0.1000 0.9699 0.8885 0.8131
0.1458 0.0100 0.8931 1.2073 0.7509
0.0146 0.0068 0.8669 1.6891 0.7159
0.0032 0.0026 0.8825 2.0717 0.7069
0.0011 0.0007 0.9052 2.1434 0.6986
0.0003 0.0002 0.9368 2.1554 0.6878
z =
-41.9805
>> [i,x,y,s,z]=IpmNew(A, b, c, epsilon)
i =
60
z =
-41.9993
85
x =
1.0000 1.0000 1.0000 1.0000 1.0000
1.4437 1.7530 0.8831 0.9645 1.0398
2.0409 3.0601 0.7365 0.3780 0.8702
3.0192 4.6730 0.4016 0.0474 0.5044
3.8797 5.8542 0.0624 0.0308 0.1302
3.9559 5.9723 0.0383 0.0293 0.1223
3.9623 5.9859 0.0372 0.0256 0.1236
3.9669 5.9883 0.0330 0.0232 0.1105
3.9702 5.9896 0.0297 0.0208 0.0997
3.9732 5.9906 0.0268 0.0188 0.0897
3.9759 5.9916 0.0241 0.0169 0.0808
3.9783 5.9924 0.0217 0.0152 0.0727
3.9805 5.9932 0.0195 0.0137 0.0655
3.9824 5.9938 0.0176 0.0123 0.0589
3.9842 5.9945 0.0158 0.0111 0.0530
3.9857 5.9950 0.0143 0.0100 0.0478
3.9872 5.9955 0.0128 0.0090 0.0430
3.9884 5.9960 0.0116 0.0081 0.0387
3.9896 5.9964 0.0104 0.0073 0.0348
3.9906 5.9967 0.0094 0.0065 0.0314
3.9916 5.9971 0.0084 0.0059 0.0282
3.9924 5.9973 0.0076 0.0053 0.0254
3.9932 5.9976 0.0068 0.0048 0.0229
3.9939 5.9979 0.0061 0.0043 0.0206
3.9945 5.9981 0.0055 0.0039 0.0185
3.9950 5.9983 0.0050 0.0035 0.0167
86
3.9955 5.9984 0.0045 0.0031 0.0150
3.9960 5.9986 0.0040 0.0028 0.0135
3.9964 5.9987 0.0036 0.0025 0.0122
3.9967 5.9989 0.0033 0.0023 0.0109
3.9971 5.9990 0.0029 0.0021 0.0098
3.9974 5.9991 0.0026 0.0018 0.0089
3.9976 5.9992 0.0024 0.0017 0.0080
3.9979 5.9993 0.0021 0.0015 0.0072
3.9981 5.9993 0.0019 0.0013 0.0065
3.9983 5.9994 0.0017 0.0012 0.0058
3.9984 5.9995 0.0016 0.0011 0.0052
3.9986 5.9995 0.0014 0.0010 0.0047
3.9987 5.9996 0.0013 0.0009 0.0042
3.9989 5.9996 0.0011 0.0008 0.0038
3.9990 5.9996 0.0010 0.0007 0.0034
3.9991 5.9997 0.0009 0.0006 0.0031
3.9992 5.9997 0.0008 0.0006 0.0028
3.9993 5.9997 0.0007 0.0005 0.0025
3.9993 5.9998 0.0007 0.0005 0.0023
3.9994 5.9998 0.0006 0.0004 0.0020
3.9995 5.9998 0.0005 0.0004 0.0018
3.9995 5.9998 0.0005 0.0003 0.0016
3.9996 5.9998 0.0004 0.0003 0.0015
3.9996 5.9999 0.0004 0.0003 0.0013
3.9996 5.9999 0.0004 0.0002 0.0012
3.9997 5.9999 0.0003 0.0002 0.0011
3.9997 5.9999 0.0003 0.0002 0.0010
87
3.9997 5.9999 0.0003 0.0002 0.0009
3.9998 5.9999 0.0002 0.0002 0.0008
3.9998 5.9999 0.0002 0.0001 0.0007
3.9998 5.9999 0.0002 0.0001 0.0006
3.9998 5.9999 0.0002 0.0001 0.0006
3.9998 5.9999 0.0002 0.0001 0.0005
3.9999 6.0000 0.0001 0.0001 0.0005
y =
0 0 0
-0.1333 -0.0519 0.0235
-0.2846 -0.6023 -0.1375
-0.5916 -1.4288 -0.4214
-0.9196 -2.0876 -0.6622
-1.4299 -2.2344 -0.5248
-1.5993 -2.2683 -0.4714
-1.5888 -2.2669 -0.4748
-1.5890 -2.2668 -0.4743
-1.5884 -2.2665 -0.4741
-1.5878 -2.2662 -0.4739
-1.5873 -2.2660 -0.4738
-1.5868 -2.2658 -0.4737
-1.5864 -2.2656 -0.4735
-1.5860 -2.2654 -0.4734
-1.5857 -2.2652 -0.4733
-1.5854 -2.2651 -0.4732
-1.5851 -2.2649 -0.4732
88
-1.5848 -2.2648 -0.4731
-1.5846 -2.2647 -0.4730
-1.5844 -2.2646 -0.4730
-1.5842 -2.2645 -0.4729
-1.5841 -2.2645 -0.4729
-1.5839 -2.2644 -0.4728
-1.5838 -2.2643 -0.4728
-1.5837 -2.2643 -0.4728
-1.5835 -2.2642 -0.4727
-1.5834 -2.2642 -0.4727
-1.5834 -2.2641 -0.4727
-1.5833 -2.2641 -0.4727
-1.5832 -2.2641 -0.4727
-1.5831 -2.2640 -0.4726
-1.5831 -2.2640 -0.4726
-1.5830 -2.2640 -0.4726
-1.5830 -2.2640 -0.4726
-1.5830 -2.2639 -0.4726
-1.5829 -2.2639 -0.4726
-1.5829 -2.2639 -0.4726
-1.5828 -2.2639 -0.4726
-1.5828 -2.2639 -0.4725
-1.5828 -2.2639 -0.4725
-1.5828 -2.2639 -0.4725
-1.5828 -2.2638 -0.4725
-1.5827 -2.2638 -0.4725
-1.5827 -2.2638 -0.4725
89
-1.5827 -2.2638 -0.4725
-1.5827 -2.2638 -0.4725
-1.5827 -2.2638 -0.4725
-1.5827 -2.2638 -0.4725
-1.5827 -2.2638 -0.4725
-1.5827 -2.2638 -0.4725
-1.5826 -2.2638 -0.4725
-1.5826 -2.2638 -0.4725
-1.5826 -2.2638 -0.4725
-1.5826 -2.2638 -0.4725
-1.5826 -2.2638 -0.4725
-1.5826 -2.2638 -0.4725
-1.5826 -2.2638 -0.4725
-1.5826 -2.2638 -0.4725
-1.5826 -2.2638 -0.4725
s =
1.0000 1.0000 1.0000 1.0000 1.0000
0.4093 0.1000 0.9699 0.8885 0.8131
0.1425 0.0100 0.8960 1.2136 0.7489
0.0143 0.0168 0.8812 1.7184 0.7110
0.0220 0.0112 0.9486 2.1166 0.6911
0.0160 0.0110 1.4328 2.2373 0.5277
0.0148 0.0097 1.5996 2.2686 0.4717
0.0132 0.0088 1.5888 2.2669 0.4748
0.0119 0.0079 1.5890 2.2668 0.4743
0.0107 0.0071 1.5884 2.2665 0.4741
90
0.0096 0.0064 1.5878 2.2662 0.4739
0.0087 0.0057 1.5873 2.2660 0.4738
0.0078 0.0052 1.5868 2.2658 0.4737
0.0070 0.0047 1.5864 2.2656 0.4735
0.0063 0.0042 1.5860 2.2654 0.4734
0.0057 0.0038 1.5857 2.2652 0.4733
0.0051 0.0034 1.5854 2.2651 0.4732
0.0046 0.0031 1.5851 2.2649 0.4732
0.0041 0.0027 1.5848 2.2648 0.4731
0.0037 0.0025 1.5846 2.2647 0.4730
0.0033 0.0022 1.5844 2.2646 0.4730
0.0030 0.0020 1.5842 2.2645 0.4729
0.0027 0.0018 1.5841 2.2645 0.4729
0.0024 0.0016 1.5839 2.2644 0.4728
0.0022 0.0015 1.5838 2.2643 0.4728
0.0020 0.0013 1.5837 2.2643 0.4728
0.0018 0.0012 1.5835 2.2642 0.4727
0.0016 0.0011 1.5834 2.2642 0.4727
0.0014 0.0010 1.5834 2.2641 0.4727
0.0013 0.0009 1.5833 2.2641 0.4727
0.0012 0.0008 1.5832 2.2641 0.4727
0.0010 0.0007 1.5831 2.2640 0.4726
0.0009 0.0006 1.5831 2.2640 0.4726
0.0008 0.0006 1.5830 2.2640 0.4726
0.0008 0.0005 1.5830 2.2640 0.4726
0.0007 0.0005 1.5830 2.2639 0.4726
0.0006 0.0004 1.5829 2.2639 0.4726
91
0.0006 0.0004 1.5829 2.2639 0.4726
0.0005 0.0003 1.5828 2.2639 0.4726
0.0005 0.0003 1.5828 2.2639 0.4725
0.0004 0.0003 1.5828 2.2639 0.4725
0.0004 0.0002 1.5828 2.2639 0.4725
0.0003 0.0002 1.5828 2.2638 0.4725
0.0003 0.0002 1.5827 2.2638 0.4725
0.0003 0.0002 1.5827 2.2638 0.4725
0.0002 0.0002 1.5827 2.2638 0.4725
0.0002 0.0001 1.5827 2.2638 0.4725
0.0002 0.0001 1.5827 2.2638 0.4725
0.0002 0.0001 1.5827 2.2638 0.4725
0.0002 0.0001 1.5827 2.2638 0.4725
0.0001 0.0001 1.5827 2.2638 0.4725
0.0001 0.0001 1.5826 2.2638 0.4725
0.0001 0.0001 1.5826 2.2638 0.4725
0.0001 0.0001 1.5826 2.2638 0.4725
0.0001 0.0001 1.5826 2.2638 0.4725
0.0001 0.0001 1.5826 2.2638 0.4725
0.0001 0.0001 1.5826 2.2638 0.4725
0.0001 0.0000 1.5826 2.2638 0.4725
0.0001 0.0000 1.5826 2.2638 0.4725
0.0001 0.0000 1.5826 2.2638 0.4725
z =
-41.9993
>>
92
MatLab code for Table 5.2
random problems with random dimentions less that 10
New method Classical method
idx dim(m*n) iteration Opt time iteration Opt time
----------------------------------------------------------------
36 6 5 12 1.4119 0.0056 13 1.4164 0.0396
37 6 7 15 0.9627 0.0065 16 0.9627 0.0046
38 6 8 17 0.7974 0.0072 24 0.7974 0.0073
39 4 8 17 0.7864 0.0336 21 0.7864 0.0061
40 8 3 15 0.4944 0.0070 38 0.3653 0.0140
41 4 9 72 1.0654 0.0757 27 1.0668 0.0086
42 5 8 18 0.4801 0.0076 24 0.4801 0.0072
43 8 9 14 1.3682 0.0070 19 1.3682 0.0064
44 2 2 13 0.7773 0.0042 14 0.7773 0.0033
45 6 5 13 0.2225 0.0059 14 0.2229 0.0049
46 8 6 14 1.6662 0.0068 20 1.6114 0.0268
47 3 6 17 0.3840 0.0069 30 NaN 0.0101
48 3 7 64 0.5853 0.0260 7 0.5886 0.0018
49 3 3 12 0.8750 0.0044 14 0.8750 0.0038
50 2 7 64 0.6310 0.0458 7 0.6367 0.0017
51 7 7 12 1.7618 0.0047 13 1.7618 0.0036
52 2 7 64 0.1590 0.0244 7 0.1647 0.0018
----------------------------------------------------------------
New method Classical method
dim(m*n) iteration time iteration time
average 4.9 5.4 40.7 0.0448 24.6 0.0362
----------------------------------------------------------------
93
New method Classical method
idx dim(m*n) iteration Opt time iteration Opt time
----------------------------------------------------------------
1 9 1 45 NaN 0.0681 30 NaN 0.0326
2 9 1 45 NaN 0.0674 45 NaN 0.7326
3 7 7 12 3.1727 0.0053 14 3.1727 0.0042
4 7 7 12 2.0364 0.0050 13 2.0364 0.0037
5 5 4 14 0.7954 0.0675 16 0.7257 0.0059
6 6 3 49 0.4286 0.0243 92 0.4560 0.1107
7 6 1 45 NaN 0.1613 45 NaN 0.0612
8 3 9 66 0.2055 0.0284 9 0.1994 0.0024
9 3 1 45 NaN 0.1071 36 NaN 0.0133
10 7 8 13 2.3593 0.0057 14 2.3593 0.0042
11 2 7 64 0.3550 0.0246 8 0.3596 0.0019
12 6 7 14 1.7947 0.0055 15 1.7947 0.0043
13 2 6 62 NaN 0.0637 27 NaN 0.0096
14 3 2 201 0.4458 0.3814 71 0.1206 0.1475
15 7 8 14 2.3068 0.0059 16 2.3068 0.0258
16 7 5 28 0.4972 0.0138 19 0.5465 0.0067
17 4 9 17 1.2697 0.0076 18 1.2697 0.0053
18 2 7 64 NaN 0.0814 17 NaN 0.0044
19 1 4 58 0.0069 0.0650 6 0.0106 0.0015
20 5 5 12 2.0070 0.0045 13 2.0070 0.0036
21 1 5 60 0.0321 0.0245 6 0.0387 0.0026
22 1 2 52 0.2947 0.0739 5 0.2976 0.0012
23 9 6 23 0.7775 0.0118 52 0.5957 0.0617
94
24 5 3 18 1.7080 0.0088 33 1.2630 0.0476
25 4 3 31 0.3020 0.0528 28 0.3146 0.0426
26 5 8 16 2.4185 0.0069 16 2.4185 0.0046
27 4 7 16 0.2408 0.0064 20 0.2408 0.0059
28 9 8 17 1.8185 0.0091 24 1.2417 0.0096
29 1 2 52 0.2448 0.0208 5 0.2478 0.0012
30 3 4 14 1.0822 0.0051 17 1.0822 0.0046
31 4 5 14 1.4672 0.0054 16 1.4672 0.0044
32 4 2 201 0.0760 0.3930 59 0.0470 0.1529
33 5 8 25 0.5901 0.0115 28 0.5901 0.0087
34 1 9 66 0.0042 0.0293 7 0.0082 0.0018
35 3 7 15 0.8637 0.0060 201 0.2481 0.3810
36 6 5 12 1.4119 0.0056 13 1.4164 0.0396
37 6 7 15 0.9627 0.0065 16 0.9627 0.0046
38 6 8 17 0.7974 0.0072 24 0.7974 0.0073
39 4 8 17 0.7864 0.0336 21 0.7864 0.0061
40 8 3 15 0.4944 0.0070 38 0.3653 0.0140
41 4 9 72 1.0654 0.0757 27 1.0668 0.0086
42 5 8 18 0.4801 0.0076 24 0.4801 0.0072
43 8 9 14 1.3682 0.0070 19 1.3682 0.0064
44 2 2 13 0.7773 0.0042 14 0.7773 0.0033
45 6 5 13 0.2225 0.0059 14 0.2229 0.0049
46 8 6 14 1.6662 0.0068 20 1.6114 0.0268
47 3 6 17 0.3840 0.0069 30 NaN 0.0101
48 3 7 64 0.5853 0.0260 7 0.5886 0.0018
49 3 3 12 0.8750 0.0044 14 0.8750 0.0038
50 2 7 64 0.6310 0.0458 7 0.6367 0.0017
95
51 7 7 12 1.7618 0.0047 13 1.7618 0.0036
52 2 7 64 0.1590 0.0244 7 0.1647 0.0018
53 4 5 15 3.6371 0.0059 16 3.6371 0.0455
54 9 3 42 0.4919 0.0216 41 0.7683 0.0158
55 7 1 45 NaN 0.0833 28 NaN 0.0224
56 5 4 20 1.4544 0.0331 14 1.6696 0.0063
57 2 1 45 NaN 0.0891 45 NaN 0.0296
58 7 5 41 1.6545 0.0388 33 0.8089 0.0497
59 1 7 64 0.1189 0.0272 8 0.1203 0.0020
60 9 9 12 3.7354 0.0071 15 3.7354 0.0049
61 6 8 16 1.3473 0.0067 17 1.3473 0.0051
62 2 8 65 0.1011 0.0256 7 0.1045 0.0018
63 2 6 16 0.5000 0.0055 19 0.5000 0.0048
64 2 6 62 0.4073 0.0233 6 0.4123 0.0014
65 4 6 15 1.0340 0.0374 19 1.0340 0.0054
66 3 9 66 0.9751 0.0292 8 0.9772 0.0021
67 5 5 13 0.9247 0.0050 14 0.9247 0.0039
68 6 6 13 1.6887 0.0051 14 1.6887 0.0452
69 2 2 13 0.1492 0.0045 15 0.1492 0.0036
70 3 2 171 NaN 0.1243 54 2.9584 0.0724
71 7 2 201 0.2807 0.2557 66 0.0886 0.0771
72 5 2 201 0.6778 0.4181 75 1.0557 0.1723
73 2 9 66 0.0872 0.0264 7 0.0909 0.0018
74 5 7 14 1.4273 0.0056 15 1.4273 0.0040
75 9 9 12 4.2331 0.0055 14 4.2331 0.0048
76 4 4 13 0.7073 0.0047 14 0.7073 0.0040
77 2 9 66 0.4177 0.0272 7 0.4248 0.0019
96
78 7 9 15 2.3273 0.0066 16 2.3273 0.0050
79 7 5 15 0.6883 0.0382 13 0.8011 0.0524
80 3 5 17 0.7023 0.0066 19 0.7023 0.0053
81 8 5 17 1.6375 0.0082 87 0.5440 0.1686
82 5 6 17 0.5266 0.0070 22 0.5266 0.0065
83 6 9 16 1.1541 0.0073 19 1.1541 0.0059
84 1 2 52 0.9460 0.0199 5 0.9489 0.0012
85 3 9 20 1.4204 0.0087 39 NaN 0.0153
86 9 4 29 0.1871 0.0378 32 0.6971 0.0615
87 9 5 37 0.9494 0.0764 32 1.3586 0.0558
88 4 9 18 0.5192 0.0078 23 0.5192 0.0411
89 9 5 41 0.5489 0.0475 18 1.5730 0.0528
90 6 4 58 NaN 0.1482 58 NaN 0.1727
91 3 5 15 0.6959 0.0058 20 0.6959 0.0055
92 7 5 14 2.9338 0.0065 14 2.3208 0.0053
93 3 5 60 0.4861 0.0749 7 0.4866 0.0020
94 3 2 196 4.5525 0.3504 146 1.8557 0.3399
95 2 8 65 0.6134 0.0258 7 0.6152 0.0018
96 6 8 15 1.6926 0.0062 19 1.6926 0.0058
97 3 6 15 0.8336 0.0057 17 0.8336 0.0047
98 4 3 56 NaN 0.0830 14 1.4102 0.0045
99 7 2 201 0.0365 0.4461 83 1.0238 0.2068
100 6 1 45 NaN 0.1986 45 NaN 0.0328
101 1 4 58 0.6234 0.0448 6 0.6307 0.0014
102 7 5 14 1.1884 0.0070 38 0.4592 0.1011
103 2 5 60 0.5702 0.0634 6 0.5733 0.0014
104 8 5 38 1.2510 0.0198 19 1.3811 0.0390
97
105 5 7 16 0.7932 0.0068 19 0.7932 0.0267
106 2 3 56 0.6463 0.0198 5 0.6492 0.0012
107 7 9 18 1.2119 0.0081 24 1.2119 0.0078
108 4 2 116 0.4112 0.1974 116 0.3480 0.2755
109 2 4 58 0.3450 0.0323 6 0.3478 0.0014
110 8 6 47 0.8233 0.0480 25 0.8203 0.0931
111 1 5 60 0.0042 0.0245 7 0.0052 0.0017
112 5 7 19 2.2536 0.0077 25 2.2536 0.0074
113 7 7 13 1.4697 0.0053 14 1.4697 0.0044
114 9 2 120 0.4624 0.2295 201 0.8555 0.4208
115 9 8 12 1.9521 0.0063 13 1.9450 0.0872
116 1 1 45 0.4377 0.0147 5 0.4378 0.0011
117 6 3 40 1.3974 0.0493 60 0.0616 0.1541
118 2 5 60 0.0523 0.0223 6 0.0577 0.0014
119 4 5 16 1.3757 0.0060 17 1.3757 0.0046
120 9 8 13 2.8652 0.0067 15 3.0675 0.0062
121 8 8 11 5.0975 0.0048 15 5.0975 0.0047
122 2 2 13 0.3116 0.0042 39 NaN 0.0116
123 9 5 18 0.4133 0.0092 21 0.5460 0.0381
124 5 4 17 1.3629 0.0079 18 1.3382 0.0401
125 9 5 17 1.4464 0.0085 16 1.9442 0.0707
126 7 6 15 2.1528 0.0073 13 1.5332 0.0388
127 1 2 52 0.4030 0.0202 5 0.4060 0.0012
128 9 8 17 1.5577 0.0091 16 1.3553 0.0158
129 7 1 45 NaN 0.1823 36 NaN 0.0295
130 8 9 17 2.5650 0.0080 21 2.5650 0.0070
131 3 4 15 0.3709 0.0055 17 0.3709 0.0062
98
132 8 7 13 1.5864 0.0063 16 2.2049 0.0061
133 9 9 12 1.4834 0.0053 14 1.4834 0.0045
134 8 4 88 0.1456 0.1307 58 0.4504 0.1475
135 6 2 27 0.0678 0.0600 16 NaN 0.0094
136 2 9 66 0.4524 0.0270 8 0.4538 0.0019
137 1 7 64 0.1021 0.0279 7 0.1043 0.0017
138 1 6 62 0.0028 0.0263 7 0.0051 0.0017
139 7 7 12 3.4885 0.0052 13 3.4885 0.0037
140 3 4 15 0.2784 0.0056 17 0.2784 0.0048
141 7 9 14 3.1083 0.0062 15 3.1083 0.0047
142 1 9 66 0.0150 0.0301 7 0.0228 0.0018
143 8 8 12 3.1648 0.0050 13 3.1648 0.0041
144 3 8 16 0.6641 0.0651 17 0.6641 0.0048
145 6 2 127 NaN 0.1905 29 0.0231 0.0666
146 7 7 11 3.2350 0.0046 13 3.2350 0.0043
147 2 4 58 0.7658 0.0200 6 0.7670 0.0013
148 8 8 12 2.1261 0.0050 13 2.1261 0.0038
149 2 5 60 0.2748 0.0218 6 0.2799 0.0526
150 1 3 56 0.1017 0.0223 6 0.1030 0.0014
151 8 9 15 2.2372 0.0071 17 2.2372 0.0058
152 7 3 195 0.4054 0.3624 48 0.3538 0.1150
153 3 9 17 0.7720 0.0073 20 0.7720 0.0058
154 1 5 60 0.0085 0.0624 7 0.0096 0.0017
155 5 7 16 0.6305 0.0068 16 0.6305 0.0046
156 7 8 16 1.9168 0.0070 25 1.9168 0.0083
157 4 5 14 0.7501 0.0052 15 0.7501 0.0045
158 7 1 45 NaN 0.1248 16 NaN 0.0114
99
159 3 3 11 0.9672 0.0264 12 0.9672 0.0031
160 1 1 45 0.4010 0.0133 4 0.4008 0.0008
161 4 8 18 0.6044 0.0076 19 0.6044 0.0053
162 6 5 14 1.2959 0.0066 16 1.2533 0.0058
163 9 6 15 1.5190 0.0076 23 1.2027 0.0529
164 9 9 12 2.4810 0.0054 13 2.4810 0.0041
165 7 2 54 3.0844 0.0905 26 1.1829 0.0092
166 9 5 11 0.6774 0.0054 20 0.8400 0.0390
167 3 9 66 0.1178 0.0305 7 0.1240 0.0019
168 2 3 56 1.2827 0.0201 5 1.2850 0.0012
169 8 5 14 3.1274 0.0066 21 2.3811 0.0387
170 2 4 58 0.2706 0.0206 6 0.2715 0.0014
171 4 3 79 1.8914 0.1462 159 0.6958 0.3458
172 1 5 60 0.0769 0.0245 6 0.0821 0.0015
173 1 3 56 0.1561 0.0220 5 0.1636 0.0012
174 7 4 55 0.3958 0.0958 31 0.8488 0.0474
175 7 6 22 0.4917 0.0108 23 0.5037 0.0401
176 1 4 58 0.0114 0.0541 6 0.0139 0.0014
177 6 7 15 1.2088 0.0063 19 1.2088 0.0057
178 8 7 22 2.3937 0.0109 31 3.0084 0.1104
179 9 8 12 2.3249 0.0063 23 2.0587 0.0430
180 1 8 65 0.0225 0.0277 8 0.0236 0.0329
181 8 3 21 0.2064 0.0111 52 0.1950 0.0961
182 8 1 45 NaN 0.1603 45 NaN 0.0903
183 3 1 45 NaN 0.0892 45 NaN 0.0170
184 4 9 16 0.7594 0.0070 25 0.7594 0.0080
185 7 6 12 1.7165 0.0058 16 1.2626 0.0795
100
186 9 9 12 4.2814 0.0057 13 4.2814 0.0042
187 7 9 15 1.0139 0.0067 17 1.0139 0.0051
188 2 3 56 0.0452 0.0192 5 0.0469 0.0012
189 8 4 48 0.4489 0.0365 14 1.0610 0.0049
190 1 2 52 0.0018 0.0198 5 0.0047 0.0012
191 4 9 18 1.5071 0.0082 23 1.5071 0.0282
192 4 3 15 0.5991 0.0068 16 0.6997 0.0056
193 6 3 123 0.0110 0.2378 67 0.0438 0.1039
194 1 6 62 0.1704 0.0566 7 0.1724 0.0017
195 3 2 53 NaN 0.0634 17 NaN 0.0057
196 2 4 58 0.0721 0.0200 6 0.0768 0.0015
197 3 8 16 2.0542 0.0065 20 2.0542 0.0056
198 5 1 45 NaN 0.1366 45 NaN 0.0839
199 4 5 13 1.3755 0.0054 15 1.3755 0.0041
200 3 3 14 1.8529 0.0051 15 1.8529 0.0039
----------------------------------------------------------------
New method Classical method
dim(m*n) iteration time iteration time
average 4.9 5.4 40.7 0.0448 24.6 0.0362
----------------------------------------------------------------
MatLab code forTable 5.3
solve the random system with dimention 200*300
m=200;n=300;
A=rand(m,n);
b=rand(m,1);
c=rand(n,1);
epsilon=0.01;
101
in new mehtod, p=0.8,q=1.2
New method Classical method
idx iteration Opt time iteration Opt time
-----------------------------------------------------------
34 31 13.3783 2.1870 34 13.3783 2.4084
35 33 13.5477 2.3660 38 13.5477 2.7975
36 35 12.2685 2.4543 47 12.2685 3.6504
37 34 12.1387 2.4273 42 12.1387 3.1664
38 31 14.3202 2.1804 42 14.3202 3.1886
39 33 11.7495 2.4815 30 11.7495 2.0452
40 30 15.0938 2.1312 39 15.0938 2.8654
41 40 15.4534 3.0469 44 15.4534 3.3165
42 38 12.4233 2.8703 44 12.4233 3.2338
43 66 12.7004 5.6847 40 12.7022 2.8478
44 39 12.4386 2.9631 40 12.4386 2.8733
45 33 13.5479 2.3851 42 13.5479 3.0937
46 36 13.7716 2.6777 43 13.7716 3.1809
47 33 12.7877 2.5047 35 12.7877 2.5006
48 36 16.7568 2.7086 36 16.7568 2.5545
49 38 15.3569 2.9702 34 15.3569 2.4201
-----------------------------------------------------------
average
New method Classical method
iteration time iteration Opt time
-----------------------------------------------------------
35.8100 2.6626 40.0000 2.8812
102
New method Classical method
idx iteration Opt time iteration Opt time
-----------------------------------------------------------
1 33 14.1325 2.2903 36 14.1325 2.4028
2 34 11.4602 2.3497 40 11.4602 2.6396
3 34 14.8578 2.3979 43 14.8578 3.4298
4 32 14.0615 2.3182 42 14.0615 2.9494
5 37 13.8361 2.6571 40 13.8361 2.7531
6 27 12.5830 1.8774 31 12.5830 2.0912
7 35 15.0891 2.4443 46 15.0891 3.1579
8 42 15.4758 3.0902 51 15.4758 3.5747
9 33 13.9757 2.3580 36 13.9755 2.3991
10 33 14.4968 2.3421 34 14.4968 2.3027
11 31 11.4472 2.1374 65 11.4295 4.6683
12 29 13.3044 1.9933 36 13.3044 2.4292
13 34 13.6009 2.3862 50 13.6009 3.4991
14 34 11.4376 2.4010 41 11.4376 2.8593
15 31 14.8721 2.1535 35 14.8721 2.3473
16 38 14.0051 2.7457 35 14.0051 2.3498
17 29 13.8349 2.0148 35 13.8349 2.3490
18 35 12.8241 2.4992 42 12.8241 2.9333
19 33 13.0498 2.3489 42 13.0499 2.9874
20 33 14.5358 2.3046 35 14.5358 2.3323
21 33 13.7607 2.3485 38 13.7607 2.6101
22 31 13.3368 2.1436 36 13.3368 2.4539
23 31 12.6427 2.1378 44 12.6427 3.0703
24 33 13.1727 2.3204 47 13.1727 3.3550
103
25 38 12.4437 2.7154 45 12.4437 3.1244
26 30 16.4101 2.0944 39 16.4101 2.6710
27 43 14.5286 3.1413 39 14.5286 2.6813
28 32 14.2258 2.2594 31 14.2258 2.1112
29 34 11.9038 2.4134 51 11.9038 3.6221
30 29 12.9792 1.9981 33 12.9792 2.1890
31 30 13.5566 2.0404 39 13.5566 2.6850
32 30 13.9748 2.1090 31 13.9748 2.0360
33 38 13.9658 2.6906 41 13.9658 2.7484
34 31 13.3783 2.1870 34 13.3783 2.4084
35 33 13.5477 2.3660 38 13.5477 2.7975
36 35 12.2685 2.4543 47 12.2685 3.6504
37 34 12.1387 2.4273 42 12.1387 3.1664
38 31 14.3202 2.1804 42 14.3202 3.1886
39 33 11.7495 2.4815 30 11.7495 2.0452
40 30 15.0938 2.1312 39 15.0938 2.8654
41 40 15.4534 3.0469 44 15.4534 3.3165
42 38 12.4233 2.8703 44 12.4233 3.2338
43 66 12.7004 5.6847 40 12.7022 2.8478
44 39 12.4386 2.9631 40 12.4386 2.8733
45 33 13.5479 2.3851 42 13.5479 3.0937
46 36 13.7716 2.6777 43 13.7716 3.1809
47 33 12.7877 2.5047 35 12.7877 2.5006
48 36 16.7568 2.7086 36 16.7568 2.5545
49 38 15.3569 2.9702 34 15.3569 2.4201
50 35 10.1950 2.5621 41 10.1950 2.9928
51 39 14.4130 3.0114 47 14.4130 3.6163
104
52 35 14.5251 2.6399 40 14.5251 2.9441
53 72 14.9484 6.2311 45 14.9489 3.2947
54 35 16.3867 2.5885 40 16.3867 2.8813
55 38 12.7772 2.9128 42 12.7772 3.1070
56 34 13.0523 2.5302 38 13.0523 2.8084
57 35 13.0530 2.5921 43 13.0530 3.2378
58 34 13.0528 2.5316 35 13.0528 2.4582
59 31 15.8548 2.2426 35 15.8548 2.5212
60 29 14.2370 2.0127 39 14.2370 2.8138
61 43 13.5499 3.3413 38 13.5499 2.7139
62 35 12.8535 2.5790 35 12.8535 2.4133
63 33 12.9314 2.4403 37 12.9314 2.6749
64 95 13.7239 9.0412 44 13.7796 3.2587
65 31 15.8651 2.2830 34 15.8651 2.3814
66 37 13.8844 2.7863 38 13.8844 2.6984
67 38 14.4416 2.9258 46 14.4415 3.5383
68 35 13.3879 2.5361 44 13.3879 3.1893
69 37 14.7301 2.6624 47 14.7301 3.5063
70 34 14.0756 2.5296 40 14.0755 2.8794
71 37 13.8434 2.7015 41 13.8434 2.9964
72 32 14.9581 2.3169 32 14.9581 2.1915
73 31 13.0970 2.2517 40 13.0970 2.9671
74 32 12.1322 2.3486 42 12.1322 3.1600
75 33 14.5811 2.4676 36 14.5811 2.5803
76 36 15.1297 2.7490 40 15.1297 2.9157
77 28 16.3501 2.0110 37 16.3501 2.6951
78 37 11.7357 2.7682 38 11.7357 2.6857
105
79 47 13.4661 3.8289 38 13.4661 2.7929
80 30 14.1476 2.1423 38 14.1476 2.7840
81 33 14.0924 2.4596 38 14.0924 2.7350
82 37 14.7447 2.7495 39 14.7447 2.7690
83 40 13.4680 2.9901 46 13.4680 3.4051
84 37 11.5136 2.7620 42 11.5136 3.0184
85 36 14.5218 2.7568 36 14.5218 2.6255
86 54 13.8599 3.9193 36 13.8574 2.6522
87 32 12.6705 2.4086 41 12.6705 3.1666
88 34 12.5076 2.6738 38 12.5076 2.8091
89 40 14.6875 3.2020 34 14.6875 2.5030
90 30 11.3641 2.2120 33 11.3641 2.3333
91 34 14.8216 2.5382 40 14.8216 2.9601
92 34 16.0217 2.5202 48 16.0217 3.8445
93 33 14.0469 2.4813 41 14.0469 3.1175
94 40 15.8152 3.0568 41 15.8152 2.9229
95 30 14.3876 2.2207 34 14.3876 2.4590
96 30 15.8316 2.2985 40 15.8316 3.0683
97 33 14.2434 2.4493 38 14.2434 2.8031
98 37 13.7590 2.8397 46 13.7590 3.5587
99 39 15.5669 2.8991 52 15.5669 3.9323
100 35 15.0051 2.6977 48 15.0051 3.7811
-----------------------------------------------------------
average
New method Classical method
iteration time iteration Opt time
35.8100 2.6626 40.0000 2.8812
106
MatLab code for Table 5.4
solve the random system with dimention 400*700
m=400;n=700;
A=rand(m,n);
b=rand(m,1);
c=rand(n,1);
epsilon=0.01;
in Dr. Lesaja's mehtod, p=0.8,q=1.2
New method Classical method
idx iteration Opt time iteration Opt time
-----------------------------------------------------------
1 41 16.8015 25.4030 57 16.8018 35.8048
-----------------------------------------------------------
MatLab code for Table 5.4
impNew
p=1-lambda;
q=1+lambda;
average of 200 iterations
107
----------------------------------------------------------------
0.1 0.2
dim(m*n) iteration time iteration time
average 5.0 5.1 35.9 0.0526 37.2 0.0603
----------------------------------------------------------------
----------------------------------------------------------------
0.3 0.4
dim(m*n) iteration time iteration time
average 4.8 5.1 40.6 0.0673 41.6 0.0648
----------------------------------------------------------------
MatLab code for Table 5.6
20 iterations
----------------------------------------------------------------
0.3 0.4
dim(m*n) iteration time iteration time
average 200.0 300.0 33.8 2.5982 37.8 2.9957
----------------------------------------------------------------
----------------------------------------------------------------
0.1 0.2
dim(m*n) iteration time iteration time
average 200.0 300.0 36.6 2.8611 34.7 2.6867
----------------------------------------------------------------
108
----------------------------------------------------------------
0.1 0.2
idx dim(m*n) iteration Opt time iteration Opt time
----------------------------------------------------------------
1 200300 36 15.3765 2.7472 35 15.3765 2.6816
2 200300 33 13.6721 2.5727 33 13.6721 2.5643
3 200300 39 13.1033 3.0115 41 13.1033 3.2237
4 200300 38 14.9661 3.0094 40 14.9661 3.2188
5 200300 32 12.9225 2.4362 28 12.9225 2.0493
6 200300 32 13.3005 2.3987 34 13.3005 2.5932
7 200300 38 11.4939 2.9336 36 11.4939 2.8237
8 200300 32 13.6853 2.4423 37 13.6853 2.9811
9 200300 33 12.6659 2.5796 28 12.6659 2.0762
10 200300 79 13.1793 6.9372 43 13.1802 3.4659
11 200300 33 12.4495 2.4903 33 12.4495 2.4935
12 200300 33 14.7143 2.5612 31 14.7143 2.3323
13 200300 32 13.0591 2.3790 34 13.0591 2.6057
14 200300 41 14.7339 3.2123 33 14.7339 2.4766
15 200300 34 15.4971 2.6664 33 15.4971 2.6206
16 200300 37 12.8786 2.9451 40 12.8786 3.0927
17 200300 36 12.6252 2.7991 37 12.6252 2.9537
18 200300 33 14.2547 2.5594 32 14.2547 2.4313
19 200300 32 15.6560 2.4247 32 15.6560 2.4553
20 200300 29 14.5694 2.1154 34 14.5694 2.5937
----------------------------------------------------------------