Investigation of Behavior of Death of Neuron andNeurogenesis in Multi-Layer Perceptron

Yuta Yokoyama, Chihiro Ikuta, Yoko Uwate and Yoshifumi NishioDepartment of Electrical and Electronic Engineering, Tokushima University

2-1 Minami-Josanjima, Tokushima-shi, Tokushima, 770-8506, JapanEmail: {yuta, ikuta, uwate, nishio}@ee.tokushima-u.ac.jp

Abstract— It is said that there are about 100 billion neurons inthe human’s brain. However, neurons had been considered to belost with age until several years ago. Death of neurons are canbe occurred by various things. And, the neurogenesis improvesability to solve problems. In the previous study, we have proposedartificial neural network which was applied the neurogenesis.

In this study, we investigate the influences of the death ofneuron and neurogenesis in more detail. We propose the deathof neuron and neurogenesis during the MLP learning process.Then, we focus on the connection weights of the neuron from theinput layer to the hidden layer. We choose the neuron betweenthe input and hidden layer. After that, the neuron is deleted fromthe network and the new neuron is generated. We compare thelearning performance of each MLP by some MLPs.

I. I NTRODUCTION

The network of the human’s brain is formed by connectingof more than one neuron. It is said that there are about 100billion neurons in the human’s brain. However, neurons hadbeen considered to be lost with age until several years ago.Namely, some neurons are generated in the brain at the infantstage. In recent studies, some researchers reported that newneurons are generated in the dentate gyrus of hippocumpus[1] - [3]. This process is called “neurogenesis”. By utilizingthe neurogenesis, some brain cells increase and the networkof within is substantial. It is known that the neurogenesisimproves ability to solve problems by combining new neurons.We focus on the characteristic of the neurogenesis. In theprevious study, we have proposed artificial neural networkwhich was applied the neurogenesis [4] - [6]. Therefore, wehave researched the performance of the proposed network.

In this study, we focus on the characteristics of the death ofneuron and neurogenesis. We apply the behavior of the deathof neuron and neurogenesis to the MLP learning process. Wechoose the neuron which is the smallest action between theinput and hidden layer. Then, the neuron is deleted from thenetwork and the new neuron is generated at there. And, thenew neuron is generated in the bottom of the hidden layer.Therefore, we investigate the influences of the death of neuronand neurogenesis in more detail.

II. N ETWORK APPLYING NEUROGENESIS

A. Death of neuron and Neurogenesis

Before, neurons had been considered to be lost with age.Some neurons die in the adult human’s brain. Death of neuronsare can be occurred by various things. However, there is no

neural stem cell which makes a neuron in the brain of anadult. It was impossible to generate the new neuron. Therefore,the neurogenesis had been considered to generate for periodof growth. However, some researchers reported that newneurons are generated in the adult brain. This process is called“neurogenesis”. The neurogenesis in the hippocumpus of thehuman brain was discovered in the late 1990s by Erickson etal [1] - [3]. The neurogenesis is included in the existing neuralcircuit by given the learning and new neurons are generated inthe human brain. It is known that the neurogenesis improvesmemory, learning, thinking ability, and so on. We focus onthe behavior of the death of neuron and neurogenesis to thelearning performance during the learning.

B. Neurons Updating

We use a Multi-Layer Perceptron (MLP) which is compar-atively easy network for artificial neural network and one of afeed-forward neural network. This network is used for patternrecognition, time series prediction, noise reduction, motioncontrol, and other tasks. The MLP is composed some layers ofneuron, it has input layer, hidden layer, and output layer. Thisnetwork learns to the tasks by changing the weight parameters.Generally, the performance of the MLP is changed by thenumber of neurons. Moreover, we used the Back Propagation(BP) which is one of the MLP’s learning method.

A Back Propagation (BP) is used to the MLP’s learningalgorithm. The BP was introduced by D.E. Rumelhart in 1986[7] - [9]. In this algorithm, the network calculate the errorfrom the output and teaching signal.

The neuron has the multi inputs and a single output. In thisstudy, we use the method of MLP and BP. The updating ruleof neuron is described by Eq. (1).

xi(t+ 1) = f

n∑j

wij(t)xj(t)− θ

, (1)

where x is the input or output andw is the connectionweight parameter andθ is threshold. In this equation, theweight of connection and threshold of neuron are learnedby BP algorithm. We used the sigmoid function for theoutput function. The error of MLP propagates backward inthe feed-forward neural network. BP algorithm changes valueof weights to obtain smaller error than before. The total error

- 65 -

IEEE Workshop on Nonlinear Circuit NetworksDecember 13-14, 2013

E of the network is described by Eq. (2).

E =1

2

p∑p=1

n∑i=1

(tpi − opi)2, (2)

whereE is the error value,p is the number of the input data,n is the number of the neurons in the output layer,tpi is thevalue of the desired target data for thepth input data, andopi is the value of the output data for thepth input data. Thefunction of the connection weight is described by Eq. (3).

∆pwk−1,kij = ηkpjo

k−1pi = −η

∂Ep

∂wk−1,ki,j

, (3)

wherewk−1,ki,j is the weight between theith neuron of the

layer k − 1 and thej the neuron of the layerk, andη is theproportionality factor known as the learning rate.

C. Proposed Network

In this study, we investigate in more detail the influences ofneurogenesis. We apply the behavior of the death of neuronand neurogenesis to MLP. We use that the MLP is composed ofthree layers (one input, hidden, and output layer). We proposethe death of neuron and neurogenesis during the MLP learningprocess. Then, we consider characteristics of the extinction,death of neuron and generated neurons.

(a) Conventional network.(b) Death of neuron and replaced

neurogenesis.

(c) Death of neuron and additional neurogenesis.Fig. 1. Network models.

We explain how to generated neurogenesis and death ofneurons. We focus on the connection weights of the neuronfrom the input layer to the hidden layer. Then, we considertwo kind of the death of neuron and neurogenesis. Oneis the death of neuron and replaced neurogenesis. In thisnetwork, we choose the neuron which is the smallest actionbetween the input and hidden layer. After that, the neuronof the smallest action is deleted to the network and the newneuron is generated at there. This process is called “death of

neuron and replaced neurogenesis”. The other is the death ofneuron and additional neurogenesis. This network is carriedout in the similar method of the death of neuron and replacedneurogenesis. However, new neuron is generated in the bottomof the hidden layer after the neuron of the smallest action isdeleted and generated during the learning process. Namely,this net work is additional neurogenesis while the neuronis deleted and generated. Then, all connection weights tothe generated neurons are newly set small random values.In this study, we assume the process to generated neuronsand connection to “neurogenesis”. After that, the connectionweights are newly calculated. Figure 1 (b), (c) show a structureof the proposed network.

III. S IMULATIONS

In this study, we apply the learning of the time series toMLP learning process. We use the chaotic time series producedby the logistic map. We compare the learning performance ofthe each MLPs. Figure 2 shows the three input data.

(a) Input data 1. (b) Input data 2.

(c) Input data 3.Fig. 2. Three input data.

We set that the conventional network and the proposednetwork are composed of three layers. The number of neuronsin the input layer and the output layer are 50 neurons. Then,we set that the conventional MLP has 5 neurons in the hiddenlayer. In the MLP with death of neuron and additional neuroge-nesis, we set to 5 neurons in the hidden layer at the start of thelearning process. Therefore, the proposed MLP is set that thenumber of neurons in the hidden layer increases. And, we setthe maximum number of the neurons in the hidden layer. TheMLPs learn 10000 times during the one trial. The learning rateof the existing neuron isη = 0.005 and the generating neuronis 0.05. We set the timing that we choose the smallest action ofneuron at every 750 iterations. The initial value of the weightsof existing neuron are given between -1.0 and 1.0 Similarly,the generating neuron are given between -0.5 and 0.5 atrandom. Because, we consider that the neurogenesis is possibleto many learning. We compare the learning performance of thefollowing three kinds of MLPs:

- 66 -

1) The conventional MLP2) The MLP with death of neuron and replaced neurogen-

esis (MLP - DR)3) The MLP with death of neuron and additional neuroge-

nesis (MLP - DRA)We use a Mean Square Error (MSE) as the measure of theirperformances. MSE is defined by Eq. (4).

MSE =1

N

N∑n=1

(tn − on)2. (4)

We make a comparison between the three MLPs. Weinvestigate the learning in some approaches.

A. Example of the learning performance

In this section, we compare the example of the learningperformance of the three different MLPs. We show an exampleof the learning performance of the MLPs in Figs. 3 and 4.In this example, we set the neurons in the hidden layer thatthe conventional MLP and the MLP - DR is set to 5 and 15neurons. Therefore, the MLP - DRA is set the neuron in thehidden layer that the number of neurons increases until 15neurons during the learning process. We show these results ofthe learning performance when we set that same initial valueof the weight connections.

Fig. 3. Example of learning performance (5 neurons in the hidden layer).

Fig. 4. Example of learning performance (15 neurons in the hidden layer).

In Fig. 3, we show the example of the learning performancewhen we set the conventional MLP and the MLP - DR with 5neurons in the hidden layer. In the MLP - DRA, we set to 5neurons in the hidden layer at the start of learning. During thelearning process, neurons in the hidden layer are deleted andgenerated until 15 neurons. Similarly, we show the exampleof the learning performance when we set the the conventional

MLP and the MLP - DR with 15 neurons in the hidden layerin Fig. 4.

From Figs. 3 and 4, the error of each MLP decreases. Then,we focus on the value of the error at the last of learning time.The learning performance of the MLP - DRA is better thanthe conventional MLP. We considered that the behavior ofthe death of neuron and neurogenesis can be applied well.However, we can say that the learning performance of theMLP - DR obtains negative result in Fig. 3. Therefore, weconsider that the death of neuron and neurogenesis haveadverse effects on the small number of neurons in the hiddenlayer.

B. Average of the learning performance

In this section, we compare the average of the learningperformance of the three different MLPs. Moreover, we carryout 100 trials with different initial weights of connections. Weused the conventional MLP and the MLP - DR with 5 and15 neurons in the hidden layer. In the MLP - DRA, we setto 5 neurons at the start of the learning process. We show theaverage of the learning performance of the MLPs in Table I.

TABLE I

AVERAGE OF THE LEARNING PERFORMANCE

(1) The conventional MLP.Neurons Average Minimum Maximum Standard variation

5 0.00841 0.00301 0.01793 0.0037415 0.00135 0.00075 0.00322 0.00044

(2) The MLP - DR.Neurons Average Minimum Maximum Standard variation

5 0.00935 0.00020 0.02896 0.0076415 0.00101 0.00004 0.00754 0.00136

(3) The MLP - DRA.Neurons Average Minimum Maximum Standard variation

5→ 15 0.00094 0.00005 0.00522 0.00126

In Table I (1), we show the average of the learning perfor-mance of the conventional MLP. We consider that the learningperformance of the 15 neurons in the hidden layer is betterthan 5 neurons in the hidden layer. And, we show the averageof the learning performance of the MLP - DR in Table I (2).Then, we compare the conventional MLP and the MLP -DRwith 5 neurons in the hidden layer. In this comparison, we cansee that the average of the learning performance of the MLP -DR obtains negative result from Table I (1), (2). Similarly, wecompare the conventional MLP and the MLP - DR with 15neurons in the hidden layer. In this comparison, we can seethat the average of the learning performance of the MLP -DR is better than the conventional MLP from Table I (1), (2).Namely, we can show the minimum error of the MLP - DR issmaller than the the conventional MLP. From these results, weconsider that the death of neuron and replaced neurogenesishave adverse effects on the small number of neurons in thehidden layer.

In Table I (3), we show the average of the learning perfor-mance of the MLP - DRA. We compare the conventional MLP

- 67 -

with 15 neurons, the MLP - DR with 15 neurons in the hiddenlayer and the MLP - DRA. In this comparison, we can showthat the learning performance of the MLP - DRA is the bestresult of all MLPs. Moreover, we can see that the minimumerror of the MLP - DRA is the smallest value. Because, weconsider that the parameter of the connecting weight changedby the behavior of the death of neuron and neurogenesis. Thus,we consider that the death of neuron and neurogenesis have apositive effect on the this network and the learning process.

C. Output data of the learning process

In this section, we compare the output data of the learningprocess of the conventional MLP with 15 neurons in the hiddenlayer and the MLP - DRA. We give the same initial value inthe each MLP. We compare the target data when we set thesecond pattern of the input data 2. Then, we show the errorvalue and the processing time in Table II at these results. Atthe same moment, we show the example of the target data andoutput data of the each MLP in Figs. 5 and 6 when we obtainthe error value in Table II.

TABLE II

LEARNING PERFORMANCE

(1) The conventional MLP. (2) The MLP - DRA.

Network model Neurons Error value Processing time

(1) 15 0.00167 0.86(2) 5→ 15 0.00072 0.76

Fig. 5. Comparison of the target data and the output data of the conventionalMLP (Target data 2).

Fig. 6. Comparison of the target data and the output data of the MLP - DRA(Target data 2).

In Fig. 5, we show the example of the target data and outputdata when we set that the conventional MLP with 15 neuronsin the hidden layer. Similarly, we show the example of the

target data and output data when we set that the MLP - DRAin Fig. 6.

From Figs. 5 and 6, we compare the output data of theeach MLP. We can say that the each MLP has approximatedthe target data. However, we can show that the MLP - DRAis more approximate than the conventional MLP to the targetdata. Namely, we are able to obtain the good performanceby the proposed network. Because, we consider that the errorvalue of the MLP - DRA is smaller than the conventional MLPfrom Table II. At the same moment, we are able to obtain thatthe processing time of the MLP - DRA is a few faster thanthe conventional MLP. Because we used the small number ofneurons in the hidden layer at the start of learning process.

IV. CONCLUSIONS

In this study, we proposed the MLP - DR and MLP - DRAfor the death of neuron and neurogenesis during the MLPlearning process. In the proposed network, we chosen theneuron between the input and hidden layer. After that, theneuron was deleted from the network and the new neuronis generated. We compared the learning performance of eachMLP by some MLPs.

In some simulations, we were able to obtain the goodperformance of the proposed network. We can see that thedeath of neuron and neurogenesis have a positive effect on thethis network and the learning. Because, we considered that theparameter of the connection weight changed by the behaviorof the death of neuron and neurogenesis. Therefore, we cansay that the death of neuron and neurogenesis have a positiveeffect on the this network and learning process.

REFERENCES

[1] S. Becker, J. M. Wojtowicz, “A Model of Hippocampal Neurogenesisin Memory and Mood Disorders,” Cognitive Sciences, vol. 11, no. 2,pp. 70-76, 2007.

[2] R. A. Chambers, M. N. Potenza, R. E. Hoffman, W. Miranker, “Simu-lated Apotosis/Neurogenesis Regulates Learning and Memory Capabili-ties of Adaptive Neural Networks,” Neuropsychopharmacology, pp. 747-758, 2004.

[3] H. Satoi, H. Tomimoto, R. Ohtani, T. Kondo, M. Watanabe, N. Oka,I. Akiguchi, S. Furuta, Y. Hirabayashi and T. Okazaki, “AstroglialExpression of Ceramide in Alzheimer’s Disease Brains: A Role DuringNeuronal Apoptosis,” Neuroscience, vol. 130, pp. 657-666, 2005.

[4] Y. Yokoyama, T. Shima, C. Ikuta, Y. Uwate and Y. Nishio, “Improvementof Learning Performance of Neural Network Using Neurogenesis,”Proceedings of RISP International Workshop on Nonlinear Circuits andSignal Processing (NCSP’12), pp. 365-368, Mar. 2012.

[5] Y. Yokoyama, C. Ikuta, Y. Uwate and Y. Nishio, “Performance of Multi-Layer Perceptron with Neurogenesis,” Proceedings of InternationalSymposium on Nonlinear Theory and its Applications (NOLTA’12),pp.715 718, Oct. 2012.

[6] Y. Yokoyama, C. Ikuta, Y. Uwate and Y. Nishio, “Investigation ofInfluences of Neurogenesis in Multi-Layer Perceptron,” Proceedingsof International Symposium on Nonlinear Theory and its Applications(NOLTA’13), pp.382 385, Sep. 2013.

[7] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Represen-tations by Back Propagation Error,” Nature, vol. 323-9, pp. 533-536,1986.

[8] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning InternalRepresentations by Error Propagation,” Parallel Distributed Processing,vol. 1, pp. 318-362, 1986.

[9] D.E. Rumelhart, J.L. McClelland, and the PDP Research Group, “Par-allel distributed processing,” MIT Press, 1986.

- 68 -