2007_splicing system based genetic algorithms for developing rbf networks models

8/3/2019 2007_Splicing System Based Genetic Algorithms for Developing RBF Networks Models

1/7


2/7

241plicing System Based Genetic Algorithms forDevelopingRBF Networks Modelsis based on a self-organizingclustering approach, suchas k-means clustering[151, the nearest neighbor clus-tering method[l6]. The application of the above algo-rithms requires the transcendental knowledge of anappropriate clustering degree which is difficult to de-termine, and i t considers only the input data. The pro-posed approach in this study does not require thetranscendental knowledge of the plant; moreover, thestructure and RBF centers can be synchronously opti-mized by utilizing the input output data.GA is an optimization algorithm on the basis ofDarwinism, which is very flexible. Depending on thefeatures of the problem's solution space, there is awide range of choices of fitness functions, the codingmethod, and the genetic operations, and all these fac-tors affect the efficiency of genetic algorithm. Thisstudy is focused on the optimization of RBF networkusing the splicing system based GA .2.1 Coding methodThere are totally n,xn real number parameters tobe optimized in the RBF network, which means onechromosome should be able to give n,xn real numbervalues, where n, is the number of hidden nodes, n isthe number of input nodes. Hence, binary codingchromosome will become very complex, and decimalcoding chromosome is used. The structure of the Zthchromosome is shown below

[ O O O O iwhere 1= 1,2, *.-, L , L is the size of the population,n,is randomly produced between 1 and D, is themaximum number of hidden nodes, the rows below n,are set to zeros and do not correspond to the center.The elements of C, are computed using the follow-ing equation:

cI/ = X j , m i n + r ( X j , m a x - j ,min )l


3/7

Chin. J. Ch . E. (Vol. 15,No.2)422.3 Operators in splicing system based GA

Li et a l . [13 ]summarized all possible operationsof the DN A splicing systems, such as elongation op-eration, deletion operation, absent operation, insertionoperation, translocation operation, transformation op-eration permutation operation, etc. Above operationsactually include three basic operations: selectlon,crossover, and mutation. Other operations adopted bystandard genetic algorithm (SGA) fo r special prob-lems may improve the performance of SGA.2.3.1 Selection operatorA set of individuals from the previous populationmust be selected for reproduction. This selection de-pends on their fitness values. Individuals with goodfitness values will most probably survive. There existdifferent types of selection operators, and in this studyroulette wheel method is applied. The probability ofthe selected individual. P(C1). s given by:

W , ) = f ( C , ) C f ( C , ) ( 5 )L:,whereflCJ is the fitness function of the individual C1,whch is obtained by I/J(Cl, wi) .The roulette wheel isplaced with L equally spaced pointers. A single spin ofthe roulette wheel will simultaneously pick all themembers of the next population.As the computationalcomplexity in Eq.(4) is O(Nz) , the complexity of se-lection operator in one generation is 0 ( L N 2 ) .2.3.2 Crossover operator

The crossover operator is applied after selectionwith a probability @,I, which produces novel indi-viduals, i.e., the novel structure and centers of RBFnetworks. It is executed between the currently selectedindividual Cl and its subsequent individual C I + ~ ,ndyields the offspring chromosomes C;, i+ l .As thenumber of input nodes (n) s fixed during the wholeGA optimization process, the crossover point is cho-sen between 1 and n. The procedure is demonstratedin Fig.1, which represents the single-point crossover.As the computational complexity of the plus operatorfor two D-by-n matrices is O ( n D ) , in the worst case(all chromosome pairs execute crossover operator),the crossover operator in one generation requiresO(LnD/2)computations.2.3.3 Mutation operator

To effectively explore the search space, mutationis carried out. When the element of an individual is

mutated with a probability (P,,,), it is replaced by anovel generated element in terms of Eq.(2). Similarcomplexity can be obtained as in the case of crossoveroperator, which is O(LnD).2.3.4 Splicing operators

As shown in the description of the crossover op-eration, different structures of RBF networks can beproduced using t h s genetic operator. However, theoperator may yield unreasonable RBF structure asshown in Fig.1, where most of the centers in ro w 5and row 6 of C; re zeros. Moreover, the crossoveroperator does not always modify the structure of theparent chromosomes. Therefore, enlightened by theDNA splicing system[131, two more genetic operators,i .e. , elongation operator and deletion operator are in-troduced. The elongation operator is used to add anovel node, and a random nonzero vector is created asdescribed in Eq.(2), whereas the deletion operator isutilized to replace the existing unreasonable nodec i ( [ C , ! ~ , C : ~ ; . . , C : ~ ] ) ith a zero vector. Suppose r is arandom number between 0 and 1, when p e > r , theelongation operator is executed. In the worst case (thenumber of hidden nodes in each chromosome add upto D ) , the complexity is O ( z L=1 ( D - n i ) n ) . Thedeletion operator is executed in the place of existingunreasonable node, which is determined by the num-ber of zeros in the node centers. If the number of zerosgoes beyond 2, the node is considered as an unrea-sonable one and will be replaced by deletion operator.In the worst case (all nodes are deleted), the computa-tional complexity is O (C"=1 n(n)2.4 The procedure of splicing system based G AThe whole processes of the optimization of RBFnetworks are described in the following steps.Step 1: Generate the code for L chromosomesrandomly in the search space.

Step 2: Calculate the corresponding L weightvectors using RLS method and compute the perform-ance index f for each individual.Step 3: Select the chromosomes for the genera-tion of new chromosomes of the next generation ac-cording to the selection operator.

c

Figure 1 Schematic diagram of the crossover operationApril, 2007


4/7

243plicing System Based Genetic Algorithmsfor Developing RBF Networks ModelsStep 4: Choose a point randomly in therange[I,n], and exchange the codes of the pairs ofchromosomes reproduced in Step 3. Repeat this for allthe pcL/2 pairs of parents.Step 5: Implement m utation, replace the element

of the current chromosome with the novel elementgenerated by Eq.(2).Step 6: Execute the splicing operators, when theconditions are met.Step 7: Repeat Steps 2 to 6 until a terminationcriterion is met. This can be the set of maximumnumber of evolutions, or the set of minimum im-provement of the best performance in successive gen-erations. Moreover, Elitism, the inclusion of the bestcurrent set in the next population, is used througho ut.Considering the complexity of o ne generation inthe whole procedure, the basic operations of one gen-eration being performed and the worst case complexi-ties associated w ith it, are as follows:(1) L RLS solutions of weight vector is

(2) Selection operator is O(LN2);(3) Crossover operator is O(LnD/2),(4)Mutation operator is U(LnD);( 5 )Elongation operator is O (cf=,D-n:)n), an d(6) Deletion operator is 0(c,=,tn) .As can be seen, the overall complexity of the

above algorithm is O ( x , , I N , ( n j ) 2 ) Once thenumber of nodes ( n i ) n the hidden layer increases,the computation significantly increases. Moreover, asthe elitism strategy is adopted throughout the wholeevolutionary procedure, the proposed GA can be com-pletely converg ed[ 181.3 SIMULATION RESULTSThe performance of the proposed m ethodology isevaluated by applying it on two different systems: anonlinear benchmark problem described by a discreteinput output model and a nonlinear continuous stirredtank reactor.3.1 Simulationtestsonadiscrete input output modelThe discrete input output model is described asfollows[ 191.y(k) =y(k -l)y(k - )y(k - )u(k - )[y(k - 3) -11 + u(k -1)

1+ y(k - 2)2 + y(k - )(6)

where lo.*(k) = 27ck25 0sin(-),sin(-) + 0.27ck250 27ck25sin(-), k 500

The objective of this application is to utilize th eproposed methodology to obtain suitable RBF con-figuration for modeling the aforementioned system.The input of RBF model consists of two previousvalues of u and three previous value s of y.x (k )= [u (k - l )u (k -2 ) Y(k -1 ) ~ ( k - 2 ) (k-3) ] (7)

The proposed methodology is compared withnearest neighbor clustering method and k-means clus-tering method, which are used to train the centers ofRBF network. 1000 data points are produced accord-ing to Eq.(6), where the first 500 data points are usedto optimize the RBF networks, and the other 500 datapoints are used to test the performance of the givenRBF networks. The operational parameters used bythe proposed algorithm can be seen in Table 1 .Table 1 GA param eters used in the d iscrete model andCSTR examples

Algorithm parameters Discrete example CSTR examplenumber of chromosomes L 30 30maximum of hidden nodesD 40 150number of generations G 150 15 0probability of crossover pc 0.8 0.8probability of mutation pm 0.1 0.1probability of elongation p e 0.01 0.0

The simulation results obtained using the abovetwo methods and the proposed GA are shown inFigs.2-5. Fig.2 shows the fitting curve of predictionvalues and real values using nearest neighbor cluster-ing method, and the plots of the estimation error aredepicted in Figs.3-5. As the nearest neighbor clus-tering method was used for calculating the number ofGaussian functions and their corresponding functioncenters, o should be set in advance. In Fig.3, CJ is se-lected as 0.2 using trial and error method. In Fig.4, thestructure of RBF network is obtained using k-meansclustering method with 20 clusters in terms of nearestneighbor clustering method, and CJ is optimized usingSGA. The number of hidden nodes in RBF networkand the sum of absolute values of modeling error ( S )using the above 3 methods are listed in Table 2. Bycomparing the results in Table 2 and the plots inFigs.3-5, it can be observed that the proposed ap-proach minimizes the error to a small extent using analmost equivalent network structure. Moreover, it isneedless to set the netwo rk parameter, such as G o 0.2and clusters to 20.

Table 2 Simulation results using the 3 different methodsMethods Number of hidden nodes S-~~nearest neighbor clustering 20 5.2876

k-means clustering & SGA 20 1.9928proposed GA 22 0.2765

Chin. J. Ch. E. 15(2)240 (2007)


5/7

244 Chin. J. Ch. E. (Vol. 15, No.2)0.6 I . ._ I

h h-1.0 L ! I0 100 200 300 400 500time

Figure 2 RB F network predictions using nearestneighbor clustering method- - - -neural predictions; -real values

-0.04-0.05 I0 100 200 300 400 500timeFigure 3 Estimation of error using nearestneighbor clustering method

-0 010-100 200 300 400 50 0timeFigure 4 Estimation of errorusingk-meansclustering method and SG A2. 5 1 I

-2.0-1 0 200 300 400 500timeFigure 5 Estimation errorusing proposed method3.2 Simulation tests on a continuous stirred tankreactor

In this study, a nonisothermal CSTR process isconsidered, which is characterized using the followingdynamic equations[20]:

where x1 and x2 represent the dimensionless reactantconcentration and reactant temperature, respectively,the physical parameters in the CSTR model equationsare: Du, 9, B and S,which correspond to the Damok-hler number, the activated energy, heat of reaction,and heat transfer coefficient, respectively. The valuesof the system parameters can be obtained as: Da=0.072, p=20, B = 8 , 6=0.3. The control input, u, isthe dimensionless temperature of the coolant. Becauseof the hard input constraints, u lies between -5 and 5.The objective is to build discrete dynamic modelsfor predicting the state variable XI and x2 using theinput output data. The input vector of RBF networkmodel is selected as follows.

x ( k ) = [ u ( k-1)u ( k - 2) u(k -3 ) y ( k ) y ( k - ) 1 (9)The proposed method is applied to optimize boththe structure and the centers of RBF networks. As theproblem is relatively complex, the number of hiddennodes obtained using nearest neighbor clustering is169 corresponding to ~ ~ 0 . 3 ,hich is too compli-cated to be adopted. Hence, the results of the proposedalgorithm are compared with another RBF identifica-tion scheme, i .e. , the improved S&A GA in Ref.[ll].The same radial basis function is adopted and themaximum number of hidden nodes is chosen as 150.For all simulations that follow, a set of 700 inputoutput data points are created by randomly selecting thevalues of the input variable within the space [ -5 , 51.The first 300 data are used in the training procedure tocalculate the connection weights; the second 200 dataare also used during the training process to evaluatethe RBF network in each generation, whereas the re-maining 200 data are used to test the efficiency of theultimate RBF network. To test the generalization per-formance of RBF networks, the values of the outputvariables are modified by adding random noise chosenfrom a uniform distribution at the interval [ 3%, +3%]of the maximum values of the given variable.To evaluate the approximation capability and gen-eralization performance, both the training error andtesting error are compared as shown in Figs.6-9. Oncethe best structure and the centers of RBF network areoptimized using GA according to the first and seconddata sets, the weights between the hidden layer and theoutput layer are updated using RLS method accordingto the second 200 data, the training errors are thus ob-tained, and then, the weights are fixed and the testingerrors are calculated in terms of the third 200 data.Figs.6 and 8 show the responses of RBF network pre-dictor optimized using S& A GA with l U S method be-cause the LS method in Ref.[1 I] could not always ob-tain the inverse of the matrix. By comparing with Figs.7and 9 through RBF networks using S& A GA , trainingerrors are obtained, which are smaller than that by theuse of proposed GA, i .e., the sum of absolute values ofthe training error (S1) using S& A GA is smaller thanthat by the use of proposed GA , and they demand muchmore hidden nodes regardless of the complexity of the

April,2007


6/7

Splicing System Based Genetic Algorithms for Developing RFSF Networks Models 2450.0150.010 Ig 0.005

.3 0Ql

C

' 'I' 'II 10.010/1 ''g 0.005-0.01 5 ' , 1 I J0 50 100 150 200time(4

0.200.15 I0.1088 0.05.? 0

3 -0.05-0.10-0.15' I I0 50 100 150 200

time(b )Figure 6 Training and testing error forXIusing S&A GA in RBF network

0.06 I 10.04

8 0.028.- O$ -0.02-

-0.04 ' . 1 'I I I I0 50 100 150 200

time(4

-0.06 '

0.06 I . I0.040.02

E og -0.02.-% -0.04* -0.06

-0.08-0.10 1 ' I t 4 I0 50 100 150 200time

(b)Figure 7 na ining and testing error forx1using proposed GA in RBF network

network structure. Moreover, the sum of the absolutevalue of the testing error (S2) are quite similar, andsome points of testing error in F igs.6 and 8 are smallerthan that in Figs.7 and 9, which can also be testifiedusing the maximum testing error (&-). This may becaused by the over-fitting of RBF network with severalhidden nodes. Table 3 shows that, as the number ofhidden nodes using S&A GA is much more than that bythe use of proposed G A, the runtime has m arkedly in-

6 1 . n I

-6 ' I I 1 I0 50 100 150 200time(4

0.100.080.061 0.04

d o- -0.02-0.04; .02.*

-0.06 , I0 50 100 150 200time(b)

Figure 8 Training and testing error forx2 usingS&A GA in RBF network

0.060.04

L 0.02g oM.s -0.02.-

-0.060.04I-0.080 50 100 150 200

time(a)

-0.08 1)-0.10 I I0 50 100 150 200

time(b)

Figure 9 Training and testing error forx2using proposed GA in RBF network

creased, which is consistent with the analysis of com-putational complexity. All algorithms are programmedby MATLAE37.01 using the com puter with C eleron(R)CPU 2.4GHz and RAM 256MB.4 CONCLUSIONSThis article presents a splicing system based GAfor the optimization of both structure and centers ofthe RBF network model. This algorithm is based on

Chin. J. Ch. E.U(2)240 (2007)


7/7

246 Chin. J. Ch . E. (Vol. 15, No.2)Table 3 The simulation results using tw o improved GA

.xi X ?

Number of nodcs S , S. El,,, Kuntirne. s Number of nodes S , S2 Runtime, sMethodsS&A GA 146 0.5654 3.8694 0.1752 4.5582X 10 136 0.3206 2.2326 0.0973 4.5076X lo 3

proposed GA 47 1.7059 3.9323 0.094s 872.7500 45 2.1 181 2.7235 0.0983 699.4690

input-output data, and its objective function considersboth approximation capability and generalization per-formance of the RBF network. In this manner. thenetwork simultaneously retains a reasonable size andeffectively describes thc whole system. Differentsimulations of benchmark problem and typicalnonlinear dynamic CSTR system ar e performed t oillustrate the effectiveness of thc proposed method.The results show that this method can produce highlyaccurate prediction and keep a relatively simple net-work structure.NOMENCLATUREheat of the reactionthe Ith n-by-L) chromosome matrixthe ith node center vectormaximum num ber of the hidden nodesDamiikhler numberniaximum o f the testing crrorthe fitness functionmaximum number ol generationsthe objective function

I( dimension assistant Lector in the Ith network i nRLS algorithmsize of the populationrnaxirnunr iterative timenumber of the input nodcsnumber of hidden nodes of the Ith networkni -bj-n: assistant matrix in the ith ncrwork i n RL Salgorithmprobability of c r o s o v c rprobability of elongationprobability of mutationrandom number between 0 an d 1su m of the absolute valuc of the modeling errorsum of the absolute value of the training enorsum of the absolute value of the testing errorcontrol input oi the actual systcinweight vector of thc Ith R BF networkri i dimension output vector of the hidden laycr i nRLS algorithmN , n-dimension input vector5N2 n-dimension input vectorsinput vector of the RB F networkreactant concentrationreactor temperatureNl output valucs of the systemN 2 output values of the systemoutput value of the actual systemheat transfer coefficientwidth of the Gaussian functionactivated energy

REFERENCES1 Park, J. . Sandberg. I. . Ciniversal approxim ation usingradial basis function networks. Neitrul Cornpr., 3(2),246-25 7( 1991 .

7-3

15

6

7

X

0

10

1 1

1213

11IS

16

17

1819

20

Prechelt, L.. Automatic early stopping using cross vali-dation: quantifying the criteria. Neural Network., 11(4),76 -767( 1998).Alexandridis. A., Sarimveis, H., Bafas, G., A new algo-I-ithm for online structure and parameter adaptation ofR B F networks, Ncicrul Nelwork., 16(7), 1003-1017(2003).Holland. J.H.. Adaptation in Natural and Artificial Sys-tems, MIT Prcss. Cambridge, MA, USA (1992).Wang. Y.. Yao. P., Sirnul& on and opti&zation for ther-mally coupled distillation using artificial neural networkand genetic algorithm, Chin. J . Chem. Eng., 11(3),307-3 1 (2003 ).Yang. T.. Lin, H.C.. Chen, M.L., Metamodeling ap-proach in solving the machine parameters optimizationproblem using neural network and genetic algorithms: Acase study. Robot Cornput. Integrated Matiuf., 22(4),322-33 l(2006).Blanco, A,. Delgado, M., Pegalajar, M.C., Real-codedgenetic algorithm for training recurrent neural networks.Nrrrrtrl Network. , 14( ) , 93--105(2001).1)clgado. M., Pegalajar. M.C., A rnultiobjective geneticalgorithm for obtaining the optimal size of a recurrentneural network for grammatical inference, PatternRecogni t . , 38(9), 1444-1456(2005).Vesin. J.M., Gruter, R., Model selection using a simplexreproduction genetic algorithm, Signal Process., 78(3),32 1-327(1999).Esposito. A, . Marinaro, M., Oricchio, D., Scarpetta, S.,Approximation of continuous and discontinuous map-pings by a growing neural RBF-based algorithm, NeuralNe t wo r k . , 13(6 ), 65 1-665(2000).Sarimveis. H., Alcxandridis, A., Ma zarakis, S., Bafas, G.,A new algorithm for developing dynamic radial basisfunction neural network models based on genetic algo-rithms. Cornput. Chem. Eng., 28(1), 209-217(2004).Chen. S.. Cowan, C.F.N., Grant, P.M., Orthogonal leastsquares learning algorithm fo r radial basis function net-works. IEEE Trans. Neurul N e w . , 2(2), 302-309( 1991 ).Li. S.C.. Xu , J ., Pan, L.Q., Operational rules for digitalcoding of RNA sequences based on DNA computing inhigh dimensional space, Bull. Sci. Tech. Soc., 18(6),388-398(2003).Jonoska, N. , Seeman, N.C., DNA Computing,Springer-Verlag, New York (2002).Pham. D.T.. Dimov, S.S., Nguyen, C.D., Selection of Kin k-means clustering, Proc. Inst. Me c h. Eng. C . Mech.E n g . Sci.. 219( ), 103-119(2005).Zhang. X.. Song, J. , RBF neural networks based on dy-namic nearest neighbor-clustering algorithm and its ap-plication in prediction of MH-Ni battery capacity,Trunsactions of China Elecfrotechnicul Society, 20( 1I),84-87(2005).Hou. Y.B., Wang M., Wang, L.Q., System Identificationan d Its MATLAB Simulation, Science Press, Beijing(2004).Li. M.Q., Kou, J.S.. Basic Theories and Applications ofGenetic Algorithm, Science Press, Beijing (2002).Liu, G.P., Kadirkamanathan, V., Billings, S.A., On-lineidentification of nonlinear systems using Volterra poly-nomial basis function neural networks, Neural Netw. ,11(9), 1645--1657( 1998).Chen. C.T., Peng, S.T., Learning control of process sys-tems with hard input constraints, J . Process Control,9(2) , 151- 160(1999).

April, 2007

2007_splicing system based genetic algorithms for developing rbf networks models

Documents