NORIIt- ItOLLAND

On the Behavior of K-out-of-N Hopfield Networks

SRIDHAR NARAYAN

Department of Mathematical Sciences, University of North Carolina at Wilmington, Wilmington, North Carolina 28403

I n t e l l i g e n t S y s t e m s

Communicated by Subhash C. Kak

ABSTRACT

Hopfield neural networks have been successfully applied to problems in areas such as combinatorial optimization, pattern recognition, and vision. Despite the successes, achieving consistency in performance remains an elusive goal. Numerous network parameters such as the initial state of the network dictate network performance, and tuning these parameters has proven to be a complex task. In this paper, we focus on a problem known as the K-out-of-N problem, and analytically explain the variations observed in the time required to achieve a solution for a fixed N as K varies between 1 and N. ©Elsevier Science Inc. 1997

1. I N T R O D U C T I O N

Mathemat ica l models tha t mimic behavior exhibited by biological neu- rons are known as artificial neurons. Collect ions o f artificial neurons, known as artificial neura l networks, have been successfully applied in a variety o f tasks such as pa t te rn recognit ion, classification, and opt imiza- tion. Refe rences [1] and [2] provide excellent surveys o f the field o f artificial neura l networks. T he class o f networks known as Hopf ie ld net- works, which fo rm the focus o f this paper , is briefly reviewed here.

The Hopf ie ld neural ne twork mode l [3, 4] employs an a r r angemen t o f artificial neurons as il lustrated in Figure 1. Neurons are represen ted by the circles, and the arrows represent weighted links tha t cor respond to synapses in biological neura l systems. A l though a H o p field ne twork may be com- pletely connec ted as shown in Figure 1, neural ne twork applications do no t necessarily require comple te interconnectivity; in particular, neurons typi- cally do no t employ self-connections.

INFORMATION SCIENCES 96, 183-191 (1997) © Elsevier Science Inc. 1997 0020-0255/97/$17.00 655 Avenue of the Americas, New York, NY 10010 PII S0020-0255(96)00189-2

184 S. NARAYAN

~ I I

~ i I2

. / V

i _

v 1

O v 2

V N

Fig. 1. Hopfield network.

In the Hopfield model, the behavior of a neuron is characterized by its activation level u i which is governed by the equation

d u i - u i ~-~ d t = T + - - T i j V j + I i (1)

y=l

where ( - u i / r ) is a passive decay term, Tq is the strength of the connec- tion between neuron j and neuron i , V 1 is the output of neuron j, and I i is an external input to neuron i.

The activation level is a continuous quantity that is analogous to the membrane potential in biological neural systems. In the absence of feed- back from other neurons and external stimulation, the passive decay term causes the activation level to gradually decay toward 0. The neuron's output V/=g(ui) is a nondecreasing function of the activation level. Most implementations of Hopfield networks employ a sigmoidal activation func- tion such as the tanh function or a piecewise linear approximation to a sigmoid. With a sigmoidal activation function, when a neuron's activation level becomes sufficiently large, the output becomes saturated and V/= 1. In this case, the neuron is said to be "on." Likewise, when the activation level is sufficiently small, V/= 0 and the neuron is said to be "off."

B E H A V I O R OF K-OUT-OF-N H O P F I E L D NETWORKS 185

Hopfield [3] showed that, when the gain of the activation function is sufficiently high, a network of neurons will equilibrate in a state corre- sponding to the minima of the quantity

E = - ½ E E T~jV~Vj - E V~Ij (2) i j i

if the interconnection matrix T is symmetric (T/y = Tji) with 0s along its diagonal (T/i=0). Eq. (2), which is often referred to as the network's "energy function," can be exploited in computations. Since a network always evolves in a manner that tends to minimize E, Hopfield networks are capable of optimization. Hopfield and Tank [4] demonstrated that networks configured in the manner of Figure 1 could be employed to obtain solutions to the traveling salesman problem. Other researchers have achieved success in applying Hopfield networks to solve problems such as stereo image matching ~[5], object recognition [6], and weapon-to- target assignment [7]. A survey of the applications of the Hopfield neural network model to problems in combinatorial optimization can be found in [8].

Although successful in a variety of domains, the quality of solutions yielded by Hopfield networks can depend heavily on network parameters. Tuning network parameters to achieve consistently good results is a complex task. One measure of performance for I-Iopfield networks is the time taken (measured by the number of iterations of the network) to arrive at a solution for a given problem. Most implementations of Hopfield networks start the solution process from a random initial state of the network followed by a randomized update sequence for the neurons in the network. Therefore, the time required to reach a solution for a given problem depends on the initial state of the network and the update sequence followed during the solution process. In this paper, we focus our attention on using the Hopfield network to solve a problem known as the K-out-of-N problem, and analytically explain the variations observed in the time required to achieve a solution for a fixed N as K varies between 1 and N.

2. T H E K-OUT-OF-N P R O B L E M

The K-out-of-N problem can be stated as follows, Given N neurons, determine the interconnection strengths between the neurons, and the external inputs to the neurons, such that exactly K of the neurons are

186 S. NARAYAN

"on" and ( N - K ) neurons are "off" when the system equilibrates. Page and Tagliarini [9] showed that the K-out-of-N rule could be enforced on any fully interconnected set of N neurons provided each interconnection T/j was equal to - 2 and each neuron received an external input of (2K-1) . Furthermore, their work showed that repeated applications of the K-out-of-N rule could be used to develop a methodology for determin- ing interconnection strengths and external inputs in Hopfield networks that represented more complex optimization problems.

As might be expected, the number of iterations required to arrive at a solution in a K-out-of-N network, for a fixed N, depends on the value of K. Contrary to expectations, however, this dependence is not linear. Figure 2 shows the average number of iterations required (over 50 solution attempts) for a K-out-of-N network to converge for different values of K and N. Our simulations employed a piecewise linear approximation to a sigmoidal activation function given by

'1, if u i> ~

~= ui+l 7 T ~" 2 ' i f - - ~ < u i < - ~

T 0, if u i < 2

(3)

with ~- set equal to 10. The initial values of u i were chosen at random within the interval (-(7/4),(~-/4)), and updates were performed asyn- chronously using a time step of 0.5. For clarity, Figure 2 only shows data for values of N between 5 and 50 that are multiples of 5; for each value of N, K varies between 1 and N. As can be seen from the figure, for a given

# of iterations 60 50

30 20 10 0

5

Fig. 2. Average convergence time for K-out-of-N networks.

BEHAVIOR OF K-OUT-OF-N HOPFIELD NETWORKS 187

value of N, the time taken by a network to converge increases as K approaches (N /2 ) and is roughly symmetric about K = (N/2) . One possi- ble explanation for this behavior is that when K = 1, a K-out-of-N network need only choose between N feasible network configurations. When K = 2, the number of feasible network configurations increases to (N). As K approaches (N/2) , the number of feasible states approaches a maximum of ((N~2)). The following section offers a more analytical explanation for the observed behavior.

3. ANALYZING THE BEHAVIOR OF A K-OUT-OF-N SYSTEM

Consider a system of N neurons, of which exactly K should be on when the system equilibrates. The Hopfield energy equation for such a system is given by

E = - 1 E E T~jV~Vj - E Vfli (4) i j i

where T/.= Tji= T= - 2 Vi , j , i-~j, T/i=0 and I i = I = ( 2 K - 1 ) Vi. Rewrit- ing Eq. ~4), we get

T Ev Evj+T Evi2-IEV. E = - . . ( 5 )

t j i i

Eq. (5) can be rewritten as

N 2 T ~ i Vi ~ j V] T E#V~ E = 2 N ---N- + 2 ~ Viz - N I • (6)

i

Denoting (E iV i /N) by V, we obtain

N2T:=2 T ,--, E-- -~--- V + ~- 2.., V/2 -N/IT.

i

(7)

Substituting T-- - 2 and I = 2 K - 1 results in

E ---N2V "2 - ~, V~ 2 - N ( 2 K - 1)V. (8) i

188 S. NARAYAN

In the following equation, the subscript s refers to the value of the corresponding parameter at system initialization. Therefore, E s, the system energy at initialization, is given by

Es =N2Vs 2 - E V/2 -N(2K- 1)V,. (9) i

When the system equilibrates, exactly K neurons are on and ( N - K ) neurons are off. Using the subscript f to signify the system at equilibrium, the average activation at equilibrium, Vf, is equal to ( K / N ) and EiV/r 2 =K. The system energy at equilibrium, Ey, is given by

E f = N 2 ( - ~ ) 2 - K - N ( 2 K - 1 ) ( - ~ ) . (10)

Simplifying Eq. (10), we obtain

El= - K z. (11)

Now consider the difference in energy levels between system initialization and system equilibrium:

E~ - E f = N 2 ~ 2 - ~.,V. 2 - N ( 2 K - 1)~ + K 2. l s i

(12)

We proceed to show that E , - E y is maximum when K=(N/2) . Differen- tiating Eq. (12) with respect to K,

O(Es-Ef ) - 2NI~ + 2K. (13) OK

The right-hand side of Eq. (13) is zero when ( K / N ) = ~ . Typically, to avoid predisposing neurons to their final states, the initial values of ui are chosen in the neighborhood of 0, so that ~ = 0.5, and therefore K = (N/2 ) is a maxima for E s - E l .

The change of the system energy with respect to time is given by

dE dg~ OE (14) dt = ~ dt OV i " i

BEHAVIOR OF K-OUT-OF-N HOPFIELD NETWORKS 189

The equations of motion for the system can be written as

dE v~ ae at ~ OV~" (15)

Substituting for ( ,gE/SV~) in Eq. (14) from Eq. (15), we get

dE dV~ [ U~ dUi ) (16) -d-{ = ~. dt ~ 7 -d-[ "

l

Rewriting Eq. (16),

dE x-~ dV~ dU i dt = ~i d U~ dt r at (17)

dE dV/( dU/ U/ ( dU/~21 (18) dt =~i - ~ i dt r ~--di-] ]"

Consider the system at initialization, that is, in the neighborhood of U/= 0. In this neighborhood, ( d V i / d U i) is a nonzero positive constant, denoted by C in the following equation. Therefore, at initialization,

dt = - ~i C " (19)

From the equation of motion for the system, we have

dUi Ui + ~7~TijVj+I i. (20) dt r

J

Substituting T~j = T = - 2 and I i = 2 K - 1, we get

dU/_ U/ _ 2 • Vj + 2V/+ ( 2 K - 1) (21) dt r

J

dUi E 2 N - - ~ + 2V/+ ( 2 K - 1) (22) dt r

dU. E - - ' - 2NIP~ + 2V/÷ ( 2 K - 1) (23) dt r

190 S. NARAYAN

At initialization, in the neighborhood of U~=0, ~ ~-0.5 and F~ =0.5. Therefore,

dU~ --d-/- = - 0 - N + 1 + ( 2 K - 1) (24)

and

at = 2 K - N . (25)

Substituting this value for (dUffdt) in Eq. (19), we have

dE - • C ( 2 K - N ) 2. (26) dt =

i

From Eq. (26), it can be seen that the rate of change of the system energy is governed by the product of C (= (dFi/dUi)) and the second term in Eq. (26). ( dF i /dU i) has the largest value in the neighborhood of U/=0 and decreases as U~ moves away from zero. This is especially true when a high gain is employed for the neuron activation function. This implies that the system energy is likely to change most rapidly during the initial phases. During later phases, when the system moves away from the neighborhood of U/= 0, ( d F i / d U i) ~ O, and therefore the system energy is not substan- tially changed. However, from Eq. (26), it can be seen that around system initialization, when the potential for change is significant, ( d E / d t ) is least when K = (N/2). This observation, coupled with the fact that the energy differential between the start and end states, E s - E r, is maximum when K = (N/2), explains the behavior observed in Figure 2.

4. SUMMARY

Although successful in a variety of domains, achieving consistency in performance remains an elusive goal for applications of Hopfield net- works. Numerous network parameters such as the initial state of the network dictate network performance, and tuning these parameters has proven to be a complex task. In this paper, we focused on a problem known as the K-out-of-N problem, and analytically explained the varia- tions observed in the time required to achieve a solution for a fixed N as K varies between 1 and N. Since the K-out-of-N problem underlies a systematic methodology for solving other combinatorial optimization prob-

B E H A V I O R OF K - O U T - O F - N H O P F I E L D N E T W O R K S 191

lems using Hopfield networks, it is anticipated that the work presented in this paper can be extended to explain, and perhaps improve, the perfor- mance of Hopfield networks in more complex problems. Another interest- ing problem relates to how the behavior depicted in Figure 2 would be affected if the network were initialized in a different manner. For instance, Hopfield and Tank [4] suggest that the network used to solve the TSP problem be initialized in a manner designed to distribute the net neuron activity desired at equilibrium equally among all the neurons in the network. Using this approach, each neuron in a K-out-of-N network would be initialized so that V~,---(K/N). Exploring this aspect provides a likely focus for future work.

R E F E R E N C E S

1. D. R. Hush and B. G. Home, Progress in supervised neural networks, IEEE Signal Processing Magazine 8-39 (Jan. 1993).

2. R. P. Lippmann, An introduction to computing with neural nets, IEEE Acoustics, Speech, and Signal Processing Magazine 2(4):4-22 (1987).

3. J. J. Hopfield, Neurons with graded response have collective computational proper- ties like those of two-state neurons, Proc. National Academy of Science USA 3088-3092 (1984).

4. J. J. Hopfield and D. W. Tank, Neural composition of decision optimization prob- lems, Biological Cybernetics 141"152 (1985).

5. M. S. Mousavi and R. J. Schalkoff, ANN implementation of stereo vision using a multi-layer feedback architecture, IEEE Trans. Syst., Man Cybernet., (8):1220-1238 (1994).

6. G. Shumaker, G. Gindi, E. Mjolsness, and P. Anandan, Stickville: A neural net for object recognition via graph matching, Tech. Rep. 8908, Center for Systems Science, New Haven, CT, Apr. 1989.

7. G. A. Tagliarini, J. F. Christ, and E. W. Page, Optimization using neural networks, IEEE Trans. Comput. (20):1347-1358 (1991).

8. J. Ramanujam and P. Sadayappan, Mapping combinatorial optimization problems onto neural networks, Inform. Sci. 82(3/4):239-255 (1995).

9. E. W. Page and G. A. Tagliarini, Algorithm development for neural networks, in: Proc. SPIE Syrup. on Innovative Science and Technology, 1988, pp. 11-18.

Received 25 April 1996; revised 20 May 1996, 18 July 1996