Synergetic Control Theory Approach for Solving Systems of Nonlinear Equations

Anton BEZUGLOV

Earth Sciences Resources Institute, University of South Carolina Columbia, SC, 29208

Anatoliy KOLESNIKOV

Synergetics and Controlling Processes Dept, Taganrog State University of RadioEngineering, Taganrog, Russian Federation

Igor KONDRATIEV

Electrical Engineering Dept, University of South Carolina, Columbia, SC, 29208

Juan VARGAS

Computer Science & Engineering Dept, University of South Carolina, Columbia, SC, 29208

ABSTRACT

This paper presents Synergetic Control Theory (SCT) and discusses how it can be used for solving systems of nonlinear equations. SCT is a new methodology for solving systems of nonlinear equations. The main advantage of SCT is that it maps the original system of equations to a dynamical system such that (1) any trajectory in the state space of the system ends in an attracting point; (2) the attracting point is located at the solution of the original system and (3) the rate at which the dynamical system moves towards the attracting point is controllable. An algorithm based on SCT is discussed. The algorithm has the following advantages: (1) If solutions exist, the algorithm finds one at a controllable rate, independently of initial guesses, (2) If no solutions exist, the algorithm makes such determination quickly, using stability analysis.

Keywords: Nonlinear equations, Systems of nonlinear equations, dynamical systems, stability analysis, Synergetic Control Theory, attracting regions, attractors.

1. INTRODUCTION

The purpose of this work is to describe a synergetic method for obtaining solutions to systems of nonlinear equations. Similar to the traditional Newton-Raphson and Broyden’s methods, the synergetic method transforms the equations to a dynamical system that performs iterations. The synergetic dynamical system can be considered superior to other systems, because: (1) It is linear and (2) It has attractors only where they are needed, i.e. at the solutions.

This paper is organized as follows. Section 2 discusses the Newton-Raphson and Broyden’s methods. In section 3 the basics of SCT are given and the

synergetic algorithm is explained. Section 4 provides examples of finding roots using the synergetic algorithm and compares the convergence rate versus the Newton-Raphson method.

2. BACKGROUND

Iterative methods for finding roots of nonlinear systems of equations are used, because mathematics does not have a sufficiently general analytical method. In contrast to analytical methods, iterative methods consider the process of solving equations or systems of equations as a dynamical process, which controls the convergence from an initial guess to a solution. The dynamical systems that guide these processes are a by product of the iterative methods. Therefore, the properties of the methods, such as convergence rate, robustness to initial guesses, robustness to local maxima and minima, are mostly determined by the properties of the dynamical systems.

A typical approach is the expansion of the Taylor’s series, which is applied in a family of methods, including the Newton-Raphson, the Broyden’s, and their modifications [1,2]. Let us review approaches used in these methods. Suppose that a problem is formulated as follows:

( ) 0=xFi (1)

where MiFi ≤≤1, is a set of equations that need to be zeroed, x is a vector of variables. The vector of functions can be approximated as a part of the Taylor’s series as:

F

( ) ( ) xJxFxxF ∆⋅+=∆+ (2)

where is the Jacobian. The iterative formulae can be J

derived by setting ( ) 0=∆+ xxF and solving (2) for x∆ :

The Broyden’s method The Broyden’s method [4] does not require an exact

Jacobian, this may save significant computational time as compared to the Newton-Raphson method. The Broyden’s method is a secant method, in which the Jacobian is approximated by the following formula:

( )xFJx ⋅−=∆ −1 (3)

( ) ( ) ( ) ( )11 −− −≈−⋅ kkkkk xFxFxxxJ (5) The Newton-Raphson method

The Newton-Raphson method uses the dynamical system based on equation (3). The solution is approached from an initial guess 0x by the iterative formula:

The Broyden’s method is computationally more

efficient, and its efficiency can be improved by using Sherman-Morison matrix inversion formula [5] for finding ( )⋅−1J . However, the global convergence properties of this method are not better than those of the Newton-Raphson method.

xxx kk ∆+=+1 (4)

where is an iteration number. The converging process stops when the distance iterations become less than some threshold:

k

0,1 ><−+ εεkk xx .

Note that in both methods, the dynamical systems linearize the nonlinear function ( )xF . This linearization works when the nonlinear nature of the function is not dominant or when successful initial guesses are made. However, the dynamical systems are developed so that their convergence properties remain unknown. Apart from converging to the roots, they may converge to local extrema, diverge, or be unstable, causing numerical overflow.

The main aim of this work is to suggest a different approach to the process of finding roots. The synergetic iterative algorithm, which is described in the following section, creates a dynamical system with known properties. Perhaps the single most important feature of this algorithm is that the generated dynamical system will be linear according to the solutions of the system of equations. Therefore, the global convergence of the dynamical system (and of the method itself) is its unique property.

Figure 1. Failures of the Newton-Raphson method

The Newton-Raphson method is known to have a quadratic convergence rate, provided that the initial guesses happen to be near the roots. If the initial guesses are not near the roots, the method may fail to converge. Figure 1 (taken from [3]) shows several types of unsuccessful choices of initial guesses. In the top left, the Newton-Raphson will infinitely loop at the local minimum. The top right and bottom right demonstrate a numerical overflow if the guess is close to a point where the first derivative of is close to zero. In the bottom left graph, the divergence is demonstrated.

( )⋅F

3. THE SYNERGETIC CONTROL THEORY APPROACH (SCT)

In general, SCT [6,7] provides methods for designing optimal controllers for dynamical systems, where the controllers are coordinated with internal expectations of the systems. The resulting dynamical systems with their controllers have areas of attraction that correspond to the control goals. Introducing such areas of attraction, or attractors, to dynamical systems, is one of the main concepts of the SCT. In this work, the attractors are created at the roots of nonlinear systems of equations. This allows the Synergetic Algorithm (which is described later in this section) to converge and find the roots of the systems.

The insufficient global convergence and instability at the local extrema are limitations of the Newton-Raphson method, which can be improved. For instance, the iteration formula can be modified

xxx kk ∆⋅+=+ α1 , where 10 ≤<α . The parameter

α is chosen so that it minimizes ( 12

+kxF ). This modification, which is supposed to enhance the convergence, is a heuristic, and as such, does not guarantee a global convergence.

An attractor is a region in the state space of a dynamical system that pulls the trajectories from nearby areas of the state space. Depending on the dimensionality of the state space, attractors can be points, contours, tori or regions of fractal dimensionality. The attractors represent the internal ‘wishes’ of the dynamical system. Whatever the initial conditions are, the system moves towards one of the attractors and remains there infinitely.

If the requirements that the controllers provide to the system cannot be fulfilled, the attractors cannot be created, and the dynamical system becomes unstable. This situation can be identified by stability analysis.

20 40 60 80 100Time

-0.05

0.05

0.1

X

If no attractors are present, the system is unstable and it will not converge. This situation can be diagnosed by analyzing stability using the Lyapunov Stability Theory [8, 9] or a potential function of the system [10].

The SCT suggests creating attractors in dynamical systems at those areas of state spaces that correspond to the control purposes. The control purposes are formulated as aggregated macrovariables iψ that need to be zeroed. The aggregated macrovariables are functions of system variables x and control signals u : ( )uxii ,ψψ ≡ . Hence the attractors need to be introduced where all the aggregated macrovariables are equal to zero.

Figure 2. Convergence of ψ from different initial

conditions to attractor at 0=ψ

The parameter T in (8), as it can be seen from the equation, determines the rate of convergence. By choosing smaller T , the rate of the transition processes can be increased.

The SCT gives an equation which can be used for creating dynamical systems with attractors at 0=iψ .

( ) 0=+⋅ ψϕψ&T (6) Now, let us apply the concepts of the SCT for

creating the Synergetic Algorithm for finding roots of the systems of equations.

where T determines the rate of convergence to the attractor, ψ& is the derivative of the aggregated

macrovariable by time; and ( )⋅ϕ - some function that influences the approaching to the attractor. One of possible expressions for ( )⋅ϕ is: ( ) ψψϕ = . In this case, the equation is rewritten as:

The Synergetic Algorithm Suppose that a system of 1 nonlinear

equations is given as a set of: Mj ≤≤

( ) 0,,, 21 =Nj xxxf K (9)

This set of equations represents the purpose of the

control, which is expressed by the aggregated macrovariables. The aggregated macrovariables in this case are equal to the equations: ( )⋅≡ jj fψ . This assures that the attractors will be created at the solutions of the system of equations.

0=+⋅ ψψ&T (7)

The solution of this differential equation gives the following function for ψ :

( ) Tt

et−

⋅= 0ψψ (8) The dynamical system is created using equation (7): ( ) ( ) ( ) 0,,

,,,,1

11

1

1 =+

⋅

∂

∂+⋅

∂

∂⋅ NjN

N

NjNj xxfxx

xxfx

xxxf

T K&K

&K

(10) where stands for time. As it is depicted in Figure 2, t( )tψ is attracted to 0=ψ from any initial conditions

0ψ .

In order to model the dynamical system, its equations need to be solved for time derivatives. According to time derivatives, these differential equations are linear, thus the solution can be found using Gauss Elimination, which has a complexity of ( )3nO , or other methods.

The resulting dynamical system:

( )Txxxgx Nii ,,,, 21 K& = (11)

can be iterated using any iterative method, like Runge-Kutta, etc.

A specific result can be obtained for a one dimensional case:

20 40 60 80 100

Time

-1

-0.5

0.5

1

1.5

X,Y

Figure 3. Convergence of the system (a= 2, b=1) from different initial values

( )( )xfTxf

xx ′⋅−=& (12)

An important property of the dynamical system

given by (11), is that its attractors are located where the system of equations has solutions. Besides, although

are nonlinear, the dynamical system is linear

according to

( )⋅ig

jψ . This guarantees convergence to the solutions only, provided that they exist.

If no roots exist, iterating over the dynamical system is not needed; instead, stability analysis can be used for checking the system’s consistency. An example of such a system is given in Section 4.

Figure 3 demonstrates the convergence of the dynamical system for X and Y to the root from several initial guesses. For cases when there are several roots, several

attractors are created in the dynamical system. The initial guess determines the root to which the system converges.

The convergence is finished after . The obtained solution matches the root of the system, found analytically: for

60>t

2=a and , it is 1=b 38.0=x and 62.0−=y .

By using SCT concepts for solving systems of nonlinear equations, the synergetic algorithm achieves two goals: (1) It checks if the system has roots prior to iterating and (2) It converges to the roots globally, i.e. independently to initial guesses. Inconsistency in equations

Let us now discuss the case when the equations have no solution. Without loss of generality and for illustrative purposes only, a one dimensional case is considered here.

4. EXPERIMENTS AND DISCUSSION

Examples illustrating the synergetic algorithm are given in this section. The algorithm is discussed in relation to: (1) A simple system of equations. (2) A system that may be inconsistent, (3) A comparison of the the convergence rate versus the Newton-Raphson method.

Suppose that the equation is:

02 =+ ax (15)

Depending on the value of , the equation can have one or two roots, or no roots at all. After applying the procedure to the equation, the dynamical system that converges to the root(s) is:

aA simple system of equations

Suppose that we need to find the roots of the system:

Txaxx

⋅⋅−−

=2

2

& (16)

=−=+byxaee yx

(13)

By introducing a potential function ( )xV as The application of the synergetic algorithm in this case will consist of the following steps. First, the dynamical system based on the equation (10) is constructed:

( ) xxxV

&−=∂

∂ (17)

( )

( )

=−−+−⋅=−++⋅+⋅⋅

00

byxyxTaeeyexeT yxyx

&&

&& (14)

we can represent the ‘energy’ of the dynamical system and use it for stability analysis. The set of maxima of the potential function correspond to unstable states (analogous to maxima of ‘potential energy’), minima – to stable states.

Then, the system is solved for and to obtain the dynamical system equations. Once that is done, the dynamical system is iterated over to obtain the solution.

x& y& The potential function for the obtained dynamical system is:

( ) ( )xT

aT

xxV ln24

2

⋅⋅

+⋅

= (18)

-4 -2 2 4

x

1

2

3

4 V(x)

Figure 6. Potential function V for ( )x 0=a

Now consider the three possible situations, based on parameter . a

Case 1. . 0>a

-4 -2 2 4

x

-6

-4

-2

2

4

V(x)

Figure 4. Potential function V when a ( )x 0>

When 0=a , the dynamical system is stable, it has a single attractor and globally converges to the solution as depicted in Figure 7:

2 4 6 8 10

Time

-0.05

-0.025

0.025

0.05

0.075

0.1X

Figure 7. Stable system with one attractor when 0=a

Thus, the potential function has no minima and the system is unstable (Figure 5):

2 4 6 8 10

Time

-1200

-1000

-800

-600

-400

-200

X

Figure 5. Unstable system when 0>a

Case 3. 0<a . In this case the potential function has two minima as

depicted in

-4 -2 2 4

x1

2

3

4

5

6 V(x)

Figure 8. Potential function V when ( )x 0<a

As it is depicted in Figure 5, the choice of close initial guesses causes significant deviations of the trajectories. There is no conversion and the system stops due to numerical overflow.

Case 2. . 0=aIn this case, the logarithm is eliminated from the

potential function, and the potential function becomes a parabola with a focus at as depicted in Figure 6. 0=x

As in the previous case, the dynamical system is globally stable (except for ), however now it has two attractors corresponding to the roots. The choice of any of the attractors depends on the initial guesses. In this case, the axes is a potential boundary between the two attracting regions. A dynamical system cannot intersect this line by itself. Therefore, the choice of the root is determined by the initial guesses, as it is illustrated in Figure 9.

0=x

y

5. CONCLUSIONS In terms of the SCT, this dynamical system has a bifurcation point at , where any deviation from zero causes significant changes in the system.

0=x This paper discusses the application of the Synergetic Control Theory for solving systems of nonlinear equations. A Synergetic algorithm is presented and its characteristics are discussed. The algorithm has the the following advantages vis-a-vis existing iterative algorithms: (1) The nature of the algorihm is such ahta it makes possible to directly assess the consistency of the systems of equations, (2) The global convergence to the roots of systems is guaranteed and (3) The algorithm exhibits controllable rate of the convergence, which can be made faster/slower than the rate of the other algorithms.

-3

2 4 6 8 10

Time

-2

-1

1

2

3

4 X

Figure 9. Stable system with two attractors when 0<a

BIBLIOGRAPHY

[1] Robert J. Schilling, Sandra L. Harris, Applied Numerical Methods for Engineers Using Matlab and C, Brooks/Cole, 2000, pp. 228-231

[2] Acton, F.S. 1970, Numerical Methods That Work; 1990, corrected edition (Washington: Mathematical Association of America), Chapter 14, 1970

Convergence rate The convergence rate of the dynamical system and

hence the algorithm is determined by the value of the parameter T in (7) and (10). The smaller the T value, the higher the convergence rate. Since T can be any positive number, the convergence rate can be chosen so that the algorithm outperforms the traditional iteration methods. For instance, the comparison of the convergence rates between the Synergetic and the Newton-Raphson methods is illustrated in Figure 10. The figure demonstrates that the convergence rate can be controlled.

[3] Stefan Jahn, et al., Qacs: Technical papers, http://qucs.sourceforge.net/tech/node30.html, 2000

[4] C.G. Broyden, A class of methods for solving nonlinear simultaneous equations, 1965, Mathematics of Computation, vol. 19, pp. 577–593

[5] G.J. Lastman, N.K. Sinha, Microcomputer-based numerical methods for science and engineering, 1989, New York: Saunders

[6] A.A. Kolesnikov, The Basis of the Synergetic Control Theory, Moscow, Ispo-Servis, 2002 (in Russian)

There is a restriction on the choice of T that comes from (10). If T is small and iterations for modeling the dynamical system are large, then large values of and

are possible during the transition process. The

increase of

ix&

ixT helps to keep the transition process under

control.

[7] A.A. Kolesnikov, Synergetic Control Theory, Moscow, Energoatomizdat, 1994 (in Russian)

[8] W. Hahn, Theory and Application of Liapunov's Direct Method, Englewood Cliffs, NJ: Prentice-Hall, 1963.

[9] R.E. Kalman, J.E. Bertram, Control System Analysis and Design Via the 'Second Method' of Liapunov, I. Continuous-Time Systems, J. Basic Energ. Trans. ASME 82, 371-393, 1960

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Iterations

X va

lue

Synergetic, T=0.8Synergetic, T=1.1Newton-Raphson

[10] A.A. Kolesnikov, et al., The Modern Applied Control Theory, Part 2, pp. 43-47, Taganrog State University of Radioengineering, 2000 (in Russian)

Figure 10. Convergence rates of the methods