[ieee 2012 ieee 13th international symposium on computational intelligence and informatics (cinti) -...

Progressive Bacterial Algorithm

László Gál1, Rita Lovassy2, László T. Kóczy3,4 1Dept. of Technology, Informatics and Economy

University of West Hungary, Szombathely, Hungary 2Inst. of Microelectronics and Technology

Kandó Kálmán Faculty of Electrical Engineering Óbuda University Budapest, Hungary

3Dept. of Automation, Faculty of Engineering Sciences Széchenyi István University Győr, Hungary

4Dept. of Telecommunications and Media Informatics Budapest University of Technology and Economics, Budapest, Hungary

e-mail: [email protected], [email protected], [email protected]

Abstract—The purpose of this paper is to present a new version of the Bacterial Algorithms used for fuzzy rule base extraction called Progressive Bacterial Algorithm. In order to explore high quality models with very good speed of convergence towards the optimal rule base, we develop an improved version of the Bacterial Evolutionary and former Bacterial Memetic Algorithms. It is shown, in case of multidimensional reference problems, by comparing with existing methods, that an efficient and fast convergent tool is obtained.

I. INTRODUCTION Special applications of fuzzy systems, the fuzzy

controllers are present in every day applications. The design of fuzzy controllers is concerned with the calculus of fuzzy rules [17]. The construction of fuzzy rules, mathematically, sets of fuzzy relations, is one of the key problems of fuzzy reasoning and control. An important task in fuzzy rule extraction is how to select a set of important fuzzy rules from a given rule base. The application of the Bacterial Memetic Algorithm (BMA) for fuzzy rule base identification (FRBI) was proposed in [3, 4].

This combination of evolutionary and gradient based algorithms was used rather successful in global optimization approaches, by optimizing parameter values, and to improve its function approximation performance. A series of new bacterial algorithms: Bacterial Evolutionary Algorithm (BEA), [17], Improved Bacterial Memetic Algorithm (IBMA) [8], and the Modified Bacterial Memetic Algorithm (MBMA) [9] were compared from the required rule base size, function approximation capabilities of the identified rule base system, furthermore from the convergence speed point of view [12].

In our previous paper [10] the Levenberg-Marquardt (LM) method [15] and a special combination of LM algorithm with the BEA [17], had been applied to fuzzy flip-flop based neural network (FNN) parameter optimization and training. This method is called Bacterial Memetic Algorithm with Modified Operator Execution Order (BMAM) The function approximation properties of various FNNs built up from various J-K and D fuzzy flip-flop neurons have been investigated.

In [11] the function approximation performance of fuzzy neural networks built up from fuzzy J-K flip-flop neurons a new learning algorithm, the Three Step

Bacterial Memetic Algorithm was proposed. In that particular approach the bacterial memetic algorithm in combination with various kind of gradient methods with different convergence speed had been proposed. Fast algorithms with less complexity were proposed to be applied for each clone. The Quasi-Newton algorithm, Conjugate Gradient algorithm, and two Backpropagation training algorithms: Gradient Descent and Gradient Descent with Adaptive Learning Rate and Momentum were nested in the bacterial mutation.

In our previous papers [12] we had examined how using different t-norms instead of the min fuzzy operator affect the learning capability, the convergence speed of the system, and how accurately input-output data samples could be reproduced by using fuzzy rule bases obtained by an automatic rule identification process. The extensive investigations showed that non-parametric t-norms like algebraic and the (novel) trigonometric t-norms [10], furthermore Hamacher product (such as Hamacher’s (parameter = 0) or Dombi’s (parameter = 0) operators) using IBMA or MBMA training algorithms, definitely improved the system learning capabilities, resulting higher convergence speed and overall lower mean squared error (MSE) values.

Now we propose a special type of bacterial algorithm, a combination of bacterial evolutionary and bacterial memetic algorithms. The new Progressive Bacterial Algortihm (PBA) exploits the above mentioned algorithms favorable properties (fast convergence and finding the global optimum) during the whole course of the optimization.

The outline of this paper is as follows. After the Introduction, in Section II we give an overview of the BEA, BMA, IBMA, BMAM and MBMA algorithms. Section III lists the proposed new algorithm main steps. Finally, Section IV provides simulation results of fuzzy rule base identification processes using multidimensional reference test functions highlighting the efficiency of the new algorithm, followed by Conclusions and References.

CINTI 2012 • 13th IEEE International Symposium on Computational Intelligence and Informatics • 20–22 November, 2012 • Budapest, Hungary

317978-1-4673-5206-2/12/$31.00 ©2012 IEEE

II. BACTERIAL EVOLUTIONARY AND MEMETIC ALGORITHMS

A. Bacterial Evolutionary Algorithm (BEA) The Bacterial Evolutionary Algorithm starts with the

creation of the initial population. A number of individuals are randomly created and evaluated. Each individual contains a number of fuzzy rules encoded in the chromosome. The next steps are the bacterial mutation, applied for each individual [6], furthermore the gene transfer operation [18]. This genetic operation establishes relationship between the individuals of the population.

The gene transfer operation first sorts the population according to the fitness values and split them in two groups. The superior half contains the better individuals and the inferior half the rest. As a second step a “source chromosome” from the superior half, and a so called “destination chromosome” from the inferior half has been selected. The next step is to transfer a part from the source chromosome to the destination chromosome. The part is selected randomly or by a predefined criterion. Finally, the above steps are repeated according to the number of “infections” times.

B. Bacterial Memetic Algorithm (BMA) Bacterial Memetic Algorithm combines evolutionary

and local search algorithms [16], in particular the BEA and LM methods. The algorithm main steps are after the creation of the initial population the bacterial mutation to each individual, followed by a few iterations of LM method, and finally the gene transfer operation applied per generation a number of infection times. The above listed steps are repeated from the bacterial mutation until a certain stopping criterion is satisfied. Applying this method in case of trapezoidal shaped fuzzy membership functions it often happens that the trapezoid breakpoints not satisfy a certain relationship [6], the membership function defined by the four breakpoints cannot be interpreted as a fuzzy membership function. In this case (knot order violation, KOV) an update vector reduction factor is applied in the LM method [2].

C. Improved Bacterial Memetic Algorithm (IBMA) Gál et all. proposed in [7] the so called Improved

Bacterial Memetic Algorithm for handling knot order violation occurred in the bacterial memetic algorithm used for fuzzy rule base extraction. This method performs slightly better than the method used before. The “merge of the violating knots into a single knot”, and “swap of the knots that are in the wrong order” methods are introduced, which are easy to implement and to integrate in the BMA.

D. Bacterial Memetic Algorithm with the Modified Operation Execution Order (BMAM)

Another improvement of the BMA is the BMAM algorithm [8] which exploits the LM method more efficiently. In particular the LM is nested into the BEA. In this way the local search is applied for every global search cycle. In this approach after each mutation step (of every bacterial mutation iteration) several LM iterations are done. Several tests have shown that it is enough to run just 3-5 LM iterations per mutations to improve the performance of the whole algorithm [6]. The IBMA and BMAM perform better than the original BMA algorithm,

increasing the model convergence speed. In particular, the IBMA shows better results in the early phase of the optimization process, in case of more complex fuzzy rule base, while BMAM performs better after the initial period of the optimization process, when we have less complex fuzzy rule base.

E. Modified Bacterial Memetic Algorithm (MBMA) The advantages of IBMA and BMAM algorithms are

combined in the Modified Bacterial Memetic Algorithm [9]. The original Bacterial Memetic Algorithm is modified in the knot order violation handling (affecting the LM method incorporated in the BMA), and in the operator execution order. In our previous papers [13] we compared the above mentioned algorithms from several criteria: model convergence speed, fuzzy rule base complexity, the new generation apparition way and time (iteration length). To obtain a high quality model with a good convergence speed, in our previous papers [5], we proposed the MBMA algorithm. The simulation results proved that this method is superior to IBMA and BMA algorithms. We observed that the model convergence speed depends not only from the complexity of the fuzzy rule base, but varies in different phases of the optimization process. For example, we measured less convergence speed in the first 10% of the optimization process, case of a 2 dimensional test function, and in the 20-30% of the optimization process, case of a 6 dimensional test function [13].

In the various algorithms the time measured between two generations (one iteration length) is very different. The BEA is the fastest (the cycle includes bacterial mutation and gene transfer), followed by the IBMA (applying the bacterial mutation, LM cycles and gene transfer). A single iteration of the MBMA requires the longest time, including in each cycle modified bacterial mutation (with LM iterations), LM method and gene transfer. Comparing the BEA – IBMA (BMA) resp. the IBMA – MBMA methods we concluded that BEA shows the highest convergence speed in the initial phase of the optimization process. As a possible new approach, an adaptive scheduling of optimization algorithms (Fast Greedy Scheduler – FGS) was introduced in [1].

III. PROGRESSIVE BACTERIAL ALGORITHM (PBA) The novel algorithm combines the improvements of

BEA, IBMA and MBMA algorithms. In case of unknown applications the main questions are:

What should be the proper sequence of these methods? How can they be combined to obtain a high quality model overall of the optimization process with a very good convergence speed?

The main goal is to obtain low model error values during the whole optimization process. We propose to use BEA in the early stage of the optimization, followed by IBMA, and using MBMA as the last step. In order to determine when to change from one algorithm to another we apply concurrently two algorithms for different individuals in the population. The individuals were monitored in terms of time. For all those, whom the simplest algorithm provides no more better model errors we switch the individuals training algorithm to the other one. In this way in each stage of the optimization process favorable overall MSE values and training characteristics can be obtained.

L. Gál et al. • Progressive Bacterial Algorithm

318

Figure 1. Flowchart of the PBA

The PBA flowchart can be seen in Figure 1. The

different algorithms we denoted by: M1 = BEA, M2 = IBMA and M3 = MBMA. The first step is to create the initial population, after that we apply different learning algorithm for each individual, for example: at the beginning of the optimization process for the main part of the population we choose the method M1, while for the rest (for example: 1 + int(NPopulation / 5) ) M2. Our

proposed approach starts in parallel with two versions, and we chose the best one by calculating and comparing the models MSE value. In order to have comparable MSE data we have to ensure same optimization time for each individual. Their learning time are measured (e.g. by the function QueryPerformanceCounter) and stored for each bacterium along with the current training method. In this novel approach, regarding to one PBA generation, any


319

version (individual’s) generation iteration time corresponds to the more complicated methods generation iteration time.

The following step of the PBA is to create the individuals next generation. First we try for a number of individuals the “complicated” IBMA (M2 iterations without gene transfer). Then the BEA method (M1 iterations without gene transfer) is applied for a few times for the corresponding remainder individuals, while each individual’s optimization total time (TM1) is less than a medium total time per individual (TM2) reached with the M2 algorithm.

Then the gene transfer operation within method groups is made (and if a certain termination criterion is satisfied the optimization process is over). During the gene transfer operation a model parameter sequence is transferred from the better individuals into individuals with less fitness values. For them, in many cases, the infections result in temporary high MSE values. That is the reason why the gene transfer operation, in case of sufficient number of individuals, is applied just among individuals which belongs to the same current training method. If we have in the population, for example four individuals trained with BEA and one trained with IBMA: application of the gene transfer operation to individuals trained with various methods, it probably happens that the gene transfer (which causes repeated temporary deterioration) will always affect that one trained with IBMA method, thus reducing the model efficiency.

When the minimum MSE value of individuals trained by M2 is higher than the corresponding maximum MSE value of individuals trained by M1, the procedure is repeated from the M2 – M1 training steps until a stopping criterion is satisfied (e.g. maximum number of generations). For individuals where the inequation MSEi < min(MSEM2) is not satisfied the M2 method is assigned.

After all the population’s individuals are switched to M2 algorithm, we assign the method M2 to M1 furthermore method M3 to M2, and then we start from the beginning, with reassigning methods M1 and M2 for individuals (this method can be extended for more than three suitable training algorithms).

In contrast to the former algorithms (e.g. BEA, IBMA, MBMA), in this new algorithm (PBA), the course of the simulation is not replicable due to the granularity in processing time measurement. Not even for the same initial conditions and pseudo-random number sequences.

IV. SIMULATION RESULTS In the next we will compare the BEA, IBMA, MBMA

and PBA evolutionary and bacterial optimization approaches from the model accuracy and speed of convergence point of views. We will study how depends the model MSE values from the beginning and from the end phase of the optimization process. The time values reflect just the bacterial mutation and the LM iteration periods. There are not included the very short time intervals of the gene transfers. The gene transfer operations appear just for relatievely few times (and occur especially in the BEA). The tests were performed for a two and six variable test function, as follows:

• Test function with 2 input variables (2iv); 200 samples 5

1 1 20 5 0 7 = ⋅ ⋅ ⋅f ( x ) sin ( . x ) cos( . x )

[ ] [ ]1 2where 0 3 0 3∈ ∈x ... , x ...π π (1) • Test function with 6 input variables (6iv); 500 samples

[18] ( )5 620 5

2 1 2 3 4 2 ⋅ −= + + ⋅ + ⋅ x x.f ( x ) x x x x e

[ ] [ ] [ ][ ] [ ] [ ]

1 2 3

4 5 6

where 1 5 1 5 0 4

0 0 6 0 1 0 1 2

∈ ∈ ∈

∈ ∈ ∈

x ... , x ... , x ... ,

x ... . , x ... , x ... . (2)

The algorithms parameters are: Fuzzy rules: 5; Population size: 7; Clones: 7; Gene transfers: 3 per generation; MSE values (2 iv): 30 runs average; MSE values (6 iv): one tipical simulation; Mamdani inference system aggregation operator: min and LM iterations per memetic bacterial mutation: 5.

Figures 2 and 3 present the graphs of the simulations in case of test function with 2 and 6 input variables comparing the characteristics for BEA, IBMA, MBMA and PBA algorithms. The graphs show the relation between the simulation time (horizontal axis) and the model MSE values (vertical axis).

Comparing the simulation results, in case of both test functions, it is shown that the new PBA method provides the favorable behavior: low model error compared to the other algorithms from the first stage and high model accuracy in the final stage of the optimization. However, in the initial phase of the optimization the BEA still provides higher convergence speed. The BEA is fast because the main steps of the algorithm are the bacterial mutation and the gene transfer operation. In the PBA, gene transfer operations are not applied until every single (M1) individual’s processing time exceeds the method M2 (IBMA) iteration time. During the optimization process the BEA is followed by the PBA, IBMA and finally by the MBMA algorithms. In the middle and in the final phase of the process, when the BEA efficiency decreased, the PBA algorithm MSE values are the lowest, highlighting the method main advantage.

V. CONCLUSIONS In this paper we proposed a special combination of

bacterial evolutionary and memetic algorithms. The novel PBA method speed of convergence is comparable with the listed approaches; nevertheless it can be used as a useful tool by finding a good compromise between the accuracy and the complexity of the model. The low MSE values overall the optimization process gives a reliable and stabile behavior. In the future we intend to prove the PBA algorithm efficiency by using multidimensional benchmark data sets as in [14] and find solutions for applying gene transfer operations during the M1-loop (BEA).

ACKNOWLEDGMENT This work is partially supported by Óbuda University

Grants, the project TÁMOP 421B, a Széchenyi István University Main Research Direction Grant, further the National Scientific Research Fund Grants OTKA K 75711, and K 105529.


320

PBA performance - 2iv

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 200 400 600 800 1000 1200

M time units

MSE

* 10

0

BEAIBMAMBMAPBA

Figure 2. Simulation results 2 iv (30 runs avg.), BEA, IBMA, MBMA, PBA

PBA preformance - 6iv

0

20

40

60

80

100

120

140

160

180

200

0 2000 4000 6000 8000 10000 12000M time units

MS

E * 1

00

BEAIBMAMBMAPBA

Figure 3. Simulation results 6 iv, BEA, IBMA, MBMA, PBA

References

[1] Balázs, K., Koczy, L. T.: A Remark on Adaptive Scheduling of Optimization Algorithms, IPMU 2, Vol. 81, 2010, pp. 719-728.

[2] Botzheim, J., Kóczy, L. T., Ruano,A. E.: Extension of the Levenberg-Marquardt algorithm for the extraction of trapezoidal and general piecewise linear fuzzy rules, IEEE World Congress on Computational Intelligence, Honolulu, 2002, pp. 815-819.

[3] Botzheim, J., Cabrita, C., Kóczy, L. T., Ruano, A. E.: Estimating Fuzzy Membership Functions Parameters by the Levenberg-Marquardt Algorithm, FUZZ-IEEE 2004, Budapest, Hungary pp. 1667-1672.

[4] Botzheim, J., Cabrita, C., Kóczy, L. T., Ruano, A. E.: Fuzzy Rule Extraction by Bacterial Memtic Algorithm, IFSA 2005, Beijing, China, pp.1563-1568.

[5] Botzheim, J., Gál, L., Kóczy L. T.: Fuzzy Rule Base Model Identification by Bacterial Memetic Algorithms, In: E Rakus-Andersson, R R Yager, N Ichalkaranje, L C Jain (szerk.), Recent Advances in Decision Making, Berlin; Heidelberg; New York: Springer-Verlag, 2009. pp. 21-43 (Studies in Computational Intelligence; 222.)

[6] Gál L., Kóczy, L. T.: Advanced Bacterial Memetic Algorithm, Acta Technica Jaurinensis, Series Intelligentia Computatorica, Vol. 1, No. 3, Győr, 2008, pp. 481-498.


321

[7] Gál, L., Botzheim, J., Kóczy, L. T., Ruano,A. E.: Fuzzy Rule Base Extraction by the Improved Bacterial Memetic Algorithm, 6th international Symposium on Applied Machine Intelligence and Informatics, Herlany, Slovakia, 2008, pp. 49-53.

[8] Gál, L., Botzheim, J., Kóczy, L. T.: Improvements to the Bacterial Memetic Algorithm used for Fuzzy Rule Base Extraction, Computational Intelligence for Measurement Systems and Applications, CIMSA 2008, Istanbul, Turkey, pp. 38-43.

[9] Gál, L., Botzheim, J., Kóczy, L. T.: Modified Bacterial Memetic Algorithm used for Fuzzy Rule Base Extraction, 5th International Conference on Soft Computing as Transdisciplinary Science and Technology, CSTST 2008, Paris, France, pp. 425-431.

[10] Gál, L., Lovassy, R., Kóczy, L. T.: Function Approximation Performance of Fuzzy Neural Networks Based on Frequently Used Fuzzy Operations and a Pair of New Trigonometric Norms, WCCI 2010 IEEE World Congress on Computational Intelligence, FUZZ-IEEE 2010, Barcelona, Spain, pp. 1514-1521.

[11] Gál L., Kóczy, L. T. and Lovassy R.: Three Step Bacterial Memetic Algorithm, Proc. of 14th IEEE International Conference on Intelligent Engineering Systems INES 2010, Las Palmas of Gran Canaria, Spain, 2010, pp. 31-36.

[12] Gál, L., Kóczy, L. T.: Fuzzy Rule Base Extraction by Bacterial Type Algorithms using selected T-norms, Acta Technica Jaurinensis 4, Series Intelligentia Computatorica, Vol. 3, Győr, Hungary, 2011, pp. 157-175.

[13] Gál, L., Lovassy, R.,Kóczy, L.T.: IBMA and MBMA Performance Analysis, In: The Fourth Győr Symposium on Computational Intelligence, 2012, pp. 43-45.

[14] Kulczycki, P. Kowalski, P.A.: Bayes classification of imprecise information of interval type, Control and Cybernetics, vol. 40, nr 1, 2011, pp. 101-123.

[15] Marquardt, D.: An Algorithm for Least-Squares Estimation of Nonlinear Parameters, SIAM J. Appl. Math., 11, 1963, pp. 431-441.

[16] Moscato, P.: On evolution, search, optimization, genetic algorithms and martial arts: Towards memetic algorithm, Technical Report Caltech Cocurrent Computational Program, Report.826, California Institute of Technology, Pasadena, California, USA, 1989.

[17] Nawa, N. E., Hashiyama, T., Furuhashi, T., Uchikawa, Y.: A Study on Fuzzy Rules Discovery Using Pseudo-Bacterial Genetic Algorithm with Adaptive Operator, in Proc. of IEEE Int. Conf. on Evolutionary Computation, ICEC’97, 1997, pp.589-593.

[18] Nawa, N. E., T., Furuhashi, T: Fuzzy Systems Parameters Discovery by Bacterial Evolutionary Algorithms, IEEE Transactions on Fuzzy Systems 7, 1999, pp. 608-616.


322

[ieee 2012 ieee 13th international symposium on computational intelligence and informatics (cinti) -...

Documents