literature survey 2shodhganga.inflibnet.ac.in/bitstream/10603/40078/10/10_chapter2.pdf · design of...

2

Contents

Introduction

Study of Artificial Neural

Network (ANN)

Study of Fuzzy Logic

Neuro-Fuzzy

Hybridization

Conclusion

Solution!

Problems!

Literature Survey

Chapter 2 : Literature Survey

Design of Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic 20

2.1 Introduction

The objective of the study is to design a generic architecture platform for

development of Neuro-Fuzzy system using extended functionalities of type 2 fuzzy

logic. The base for developing the architecture under study lies on two major

technologies of soft computing artificial neural network and fuzzy logic. The fuzzy

logic was later extended to type 2 fuzzy logic to enhance the decision support and

easy attachment while hybridizing with artificial neural network. Figure 2.1 describes

the major areas in which literature survey is done.

Figure 2.1: Areas of Literature Survey

Artificial Neural Network

To explore various existing artificial neural networks like Feed Forward, Radial Basis, SOM, Learning Vector Quantization, Recurrent, etc.

To study and analyze various algorithms, methods & properties (static & dynamic) for the considered artificial neural networks.

To clearly understand advantages and disadvantages of various standard methods for respective domain applicability.

To study the improvement and research done by other researchers in the field, and also find the future scope for the practical applicability of subject under research.

Fuzzy Logic (Type2)

To study the mechanism of fuzzy inference engine to understand working of fuzzy systems.

To analyze various existing fuzzy inference models like Mamdani, Sugeno and Tsukamoto.

To analyze the advantages offered by type 2 fuzzy logic and its practical applicability.

To study various types of membership functions for fuzzy inference systems.

To study defuzzification methods and their practical applicability.

To study how to generate a knowledge base for given domain area.

Neuro-Fuzzy Approach

To analyze various hybridization methods with their parameter for hybridizing artificial neural network with fuzzy logic.

To bridge the gap between crisp data of artificial neural network and fuzzy logic to generate an efficient interface for generating various neuro-fuzzy advisory systems under research.

To study existing neuro-fuzzy systems with their practical applicability and development time required.

Neuro-Fuzzy Advisory System Using Type 2 Fuzzy Logic



2.2 Study of Artificial Neural Network (ANN)

An artificial neural network (ANN) is an interconnected group of artificial neurons that

uses a computational model for processing information based on a connectionist

approach to computation. An artificial neural network is a parallel system, capable of

resolving paradigms that linear computing cannot. ANN is powerful tool for modeling

intelligent systems when underlying relationship is unknown. ANN has ability to

identify and learn the correlation between the input values and the required output

target values. For this purpose ANN have to be trained through various existing

learninig approaches so that they can predict correct output as required by the

parallel systems. The most important feature of ANN is that they learn by example

(training) which replaces traditional programming approach towards problem solving.

Due to this feature, ANN are increasingly being used in intelligent systems where

one has little or incomplete understanding of the problem under research but training

data of domain expert is readily available.

In 1943 McCulloch – Pitts[14] proposed a model for computing elements which

performed weighted sum of inputs on computing elements and then applied

threshold logic on them. This model has drawback, as the weights were fixed, it was

not able to learn by example. In 1949, Hebbian[6] proposed a model for learning

scheme by adjusting the weight of the connection on pre and post synaptic values of

the variables. Later this became the famous Hebbs law for learning rule in neural

network. In 1958 Rosenblatt[19] proposed a model based on perceptron which also

had adjustable weights based on percptron learning rule. Widrows and Hoff in

1960[27] proposed adaptive linear element for computing elements and least mean

square algorithm for learning. In 1982 Hopefield[7] made first move to develop

feedback neural networks, in his model with symmetrical weights the analysis has

shown stable equilibrium states. Taking a step further to this study in 1986

Rumelhart et al.[20] showed that it possible to adjust weights of neuron in feed

forward multi layer neural network in a systematic manner to map implicit learning of

input and output patterns in form of pairs using training sets. The method was called

as generalized delta rule or error back propagation method. From here onwards the

real life applicability of artificial neural network started. In 1998 Zhang et al [30]



provided summary of working of ANN in forecasting, further provided guidelines for

forecasting, paradigms and other issues of ANN.

Developing Artificial Neural Networks

Artificial neural networks are constructed with layers of neurons; hence the term

„Multilayer‟ is used with such artificial neural networks. A Multilayer consists of three

kind of layer namely input layer, output layer and hidden layer. There can be more

than one hidden layer. A multi layered neural network is shown in Figure 2.2. The

process of mapping of input training pattern to that of output training pattern is

actually carried out by back propagating error in hidden layers, which in turn adjust

the weights of neurons to match the required output pattern. Generally, multiple

hidden layers are designed in ANN as per requirement of the problem domain. The

smoothness of learning increases with more number of hidden layers but the

computational complexity also rises. In ANN each neuron of the preceding layer is

connected to every other neurons of the following layer. There are two functions that

steer the behavior of a neuron in particular layer, which is also applied to every other

neuron of the same neural network. They are Input function and the output function

(Activation function). Input into a node is weighted sum of output connected to it. The

threshold sign function is defined as � = 1, ℎ 0 � � = 0, ℎ < 0

And its differentiable form is represented with the help of sigmoid function

�� =1

1 + −

The activation functions are applied under particular learning algorithm. One of the

common learning algorithms for artificial neural network is back propagation. Beside

these other algorithms are radial base, self organizing map, learning vector

quantization, and reinforcement.

Back Propagation Algorithm

Back propagation algorithm was initially introduced by Werbos in 1974 [26] and further

developed by Rumelhart and McClelland in 1986 [20]. Multi Layer Perceptrons (MLP)

networks are neural networks with one or more hidden layers. Cybenko in 1989 [5] and



Funahashi in 1989 [10] have proved that the MLP network is a general function approximator

and that one hidden layer networks will always be sufficient to approximate any continuous

function up to certain accuracy. A MLP network with two hidden layers is shown in Figure

2.2. The input layer acts as an input data holder that distributes the input to the first hidden

layer. The outputs from the first hidden layer then become the inputs to the second layer and

so on. The last layer acts as the network output layer.

A hidden neuron performs two functions that are the combining function and the

activation function. The output of the j-th neuron of the k-th hidden layer is given by

1

1

1kn

i

k

j

k

i

k

ij

k

j btvwFtv ; for 0 knj (1)

and if the m-th layer is the output layer then the output of the l-th neuron yl of the

output layer is given by

1

1

1ˆmn

i

m

i

m

ijl tvwty ; for 0 onl (2)

where nk, no w‟s, b‟s and F(.) are the number of neurons in k-th layer, number of

neurons in output layer, weights, thresholds and an activation function respectively.

Figure 2.2 represent multi layered perceptron network.

Figure 2.2: Multilayered Perceptron Network

Consider a network with a single output node in output layer and a single hidden

layer is used, i.e m = 2 and no = 1. With these simplifications the network output is:

rn

j

jjij

n

i

ii

n

i

i btvwFwtvwty1

11

1

21

1

211

ˆ (3)

Where nr is the number of nodes in the input layer.

V1

V2

Vn

y1

Yn

Hidden Layers

Input Layer Output Layer

Wi Weight



The activation function F(.) is selected to be

tve

tvF

1

1 (4)

The weights wi and threshold bj are unknown and should be selected to minimise the

prediction errors defined as

tytyt ˆ (5)

Where y (t) is the actual output and y t is the network output.

Back propagation is the steepest decent type algorithm where the weight connection

between the j-th neuron of the (k-1)-th layer and the i-th neuron of the k-th layer are

respectively updated according to

tbtbtb

twtwtw

k

i

k

i

k

i

k

ij

k

ij

k

ij

1

1 (6)

with the increment w tijk and b ti

k given by

1

11

tbttb

twtvttw

k

ib

k

ib

k

i

k

ijw

k

j

k

iw

k

ij

(7)

where the subscripts w and b represent the weight and threshold respectively, w

and b are momentum constants which determine the influence of the past

parameter changes on the current direction of movement in the parameter space, w

and b represent the learning rates and ik

t is the error signal of the i-th neuron of

the k-th layer which is back propagated in the network.

Since the activation function of the output neuron is linear, the error signal at the

output node is

tytytm ˆ (8)

and for the neurons in the hidden layer

j

k

ji

k

j

k

i

k

i twttvFt 111 k = m-1, ... , 2, 1 (9)

where F v tik is the first derivative of F v ti

k with respect to v tik .



Since back propagation algorithm is a steepest decent type algorithm, the algorithm

suffers from a slow convergence rate. The search for the global minima may become

trapped at local minima and the algorithm can be sensitive to the user selectable

parameters.

Topologies for Artificial Neural Network

Feed Forward- where the data flow from input to output units is strictly

feedforward. The data processing can extend over multiple (layers of) units,

but no feedback connections are present, that is, connections extending from

outputs of units to inputs of units in the same layer or previous layers.

Recurrent- that do contain feedback connections. Contrary to feed-forward

networks, the dynamical properties of the network are important. In some

cases, the activation values of the units undergo a relaxation process such

that the neural network will evolve to a stable state in which these activations

do not change anymore. In other applications, the changes of the activation

values of the output neurons are significant, such that the dynamical behavior

constitutes the output of the neural network as shown by Pearlmutter [17].

Radial Basis- is an artificial neural network that uses radial basis functions as

activation functions. It is a linear combination of radial basis functions. They

are used in function approximation, time series prediction, and control. Radial

basis function (RBF) networks typically have three layers: an input layer, a

hidden layer with a non-linear RBF activation function and a linear output

layer. In the basic form all inputs are connected to each hidden neuron. The

norm is typically taken to be the Euclidean distance (though distance appears

to perform better in general) and the basis function is taken to be gaussian

function .i.e. changing parameters of one neuron has only a small effect for

input values that are far away from the center of that neuron. RBF networks

are universal approximators. This means that a RBF network with enough

hidden neurons can approximate any continuous function with arbitrary

precision.

Kohenen Self Organizing Map (SOM)- is a type of artificial neural network

that is trained using unsupervised learning to produce a low-dimensional



(typically two-dimensional), discrete representation of the input space of the

training samples, called a map. Self-organizing maps are different from other

artificial neural networks in the sense that they use a neighborhood function to

preserve the topological properties of the input space. This makes SOMs

useful for visualizing low-dimensional views of high-dimensional data.

A self-organizing map consists of components called nodes or neurons.

Associated with each node is a weight vector of the same dimension as the

input data vectors and a position in the map space. The usual arrangement of

nodes is a regular spacing in a hexagonal or rectangular grid. The self-

organizing map describes a mapping from a higher dimensional input space

to a lower dimensional map space. The procedure for placing a vector from

data space onto the map is to first find the node with the closest weight vector

to the vector taken from data space. Once the closest node is located it is

assigned the values from the vector taken from the data space.

While it is typical to consider this type of network structure as related to feed

forward networks where the nodes are visualized as being attached, this type

of architecture is fundamentally different in arrangement and motivation.

Large SOMs display properties which are emergent. In maps consisting of

thousands of nodes, it is possible to perform cluster operations on the map

itself.

Learning Vector Quantization- LVQ was invented by Teuvo Kohonen [23]

and it can be understood as a special case of an artificial neural network,

more precisely, it uses winner-take-all Hebbian learning-based approach. An

LVQ system is represented by prototypes W= (w(i),..., w(n)) which are defined

in the feature space of observed data. In winner-take-all training algorithms

one determines, for each data point, the prototype which is closest to the input

according to a given distance measure. The position of this so-called winner

prototype is then adapted, i.e. the winner is moved closer if it correctly

classifies the data point or moved away if it classifies the data point

incorrectly. Advantage of LVQ is that it creates prototypes that are easy to

interpret for experts in the respective application domain. LVQ systems can



be applied to multi-class classification problems in a natural way. It is used in

a variety of practical applications.

A key issue in LVQ is the choice of an appropriate measure of distance or

similarity for training and classification. Recently, techniques have been

developed which adapt a parameterized distance measure in the course of

training the system as shown by Schneider, Biehl, and Hammer [16].

Modular Networks- is a neural network characterized by a series of

independent neural networks moderated by some intermediary. Each

independent neural network serves as a module and operates on separate

inputs to accomplish some subtask of the task that the network hopes to

perform. The intermediary takes the outputs of each module and processes

them to produce the output of the network as a whole. The intermediary only

accepts the modules‟ outputs; the modules do not interact with each other.

Training of Artificial Neural Network

Once a network has been structured for a particular application, that network is

ready to be trained. To start this process the initial weights are chosen randomly.

Then, the training, or learning, begins. There are two basic approaches to training -

supervised and unsupervised. Supervised training involves a mechanism of

providing the network with the desired output either by manually "grading" the

network's performance or by providing the desired outputs with the inputs.

Unsupervised training is where the network has to make sense of the inputs without

outside help. The vast bulk of networks utilize supervised training. Unsupervised

training is used to perform some initial characterization on inputs. The third approach

for training ANN is reinforcement learning where a punishment strategy is used to

train the neural network.

Supervised Learning- we are given a set of example pairs and

the aim is to find a function in the allowed class of functions that

matches the examples. In other words, we wish to infer the mapping implied

by the data; the cost function is related to the mismatch between our mapping

and the data and it implicitly contains prior knowledge about the problem

domain. A commonly used cost is the mean-squared error, which tries to



minimize the average squared error between the network's output, f(x), and

the target value y over all the example pairs.

When one tries to minimize this cost using gradient descent for the class of

neural networks called multilayer perceptrons, one obtains the common and

well-known back propagation algorithm for training neural networks. Some

popular tasks that fall within the paradigm of supervised learning are pattern

recognition (also known as classification) and regression (also known as

function approximation). The supervised learning paradigm is also applicable

to sequential data (e.g., for speech and gesture recognition).

Un-supervised Learning- some data is given and a cost function is to be

minimized, that can be any function of the data and the network's output, .

The cost function is dependent on the task (what we are trying to model, the

implicit properties of our model, its parameters and the observed variables).

As a trivial example, consider the model , where is a constant and

the cost . Minimizing this cost will give us a value of that is

equal to the mean of the data. The cost function may be much more

complicated depending on the application: for example, in compression it

could be related to the mutual information between and , whereas in

statistical modeling, it could be related to the posterior probability of the model

given the data.

Reinforcement Learning- Here data are usually not given, but generated

by an agent's interactions with the environment. At each point in time t, the

agent performs an action and the environment generates an observation

and an instantaneous cost , according to some (usually unknown)

dynamics. The aim is to discover a policy for selecting actions that minimizes

some measure of a long-term cost; i.e., the expected cumulative cost. The

environment's dynamics and the long-term cost for each policy are usually

unknown, but can be estimated. More formally, the environment is modeled

as a Markov decision process (MDP). Dynamic programming has been

coupled with ANNs (Neuro dynamic programming) by Bertsekas and

Tsitsiklis[3] and applied to multi-dimensional nonlinear problems such as

vehicle routing and natural resources management.



2.3 Study of Fuzzy Logic

In 1965 prof. lofti zadeh[29] introduced fuzzy logic. A fuzzy subset A of a (crisp)

set X is characterized by assigning to each element x of X the degree of

membership of x in A (e.g., X is a group of people, A is the fuzzy set of old people

in X). Now if X is a set of propositions then its elements may be assigned

their degree of truth, which may be “absolutely true,” “absolutely false” or some

intermediate truth degree: a proposition may be more appropriate than another

proposition. Fuzzy variables can be used in IF-THEN-ELSE logic. They may be

understood as partial imprecise knowledge on some crisp function and have (in the

simplest case) the form IF x is Ai THEN y is Bi. In a wider sense fuzzy logic is

synonymous with the theory of fuzzy sets, a theory which relates to classes of

objects with smooth boundaries in which membership is a matter of degree. There

exists various membership functions used to classify the applicability of fuzzy

linguistic variable. A fuzzy linguistic variable are non numeric data that represents

rules or fact. A membership function can be designed in three ways: (1) through the

knowledge acquisition process like interview from experts, those who are familiar

with the underlying concept and later adjust it based on a tuning strategy; (2)

construct it automatically from data; and (3) learn it based on feedback from the

system performance. The first method was commonly used by researchers until the

late 1980s and is still a useful way if there is sufficient a priori knowledge about the

control system. Because of the poor systematic tuning strategy, most fuzzy systems

are tuned using a trial and error process. This has become one of the points of

criticism in fuzzy logic technology. Fortunately, some techniques have become

available for developing the second two methods since the late 80s, for example the

statistical techniques. The most commonly used membership functions are linear,

triangular and trapezoidal. Other membership functions like Gaussian, S-Shaped,

Generalized bell shaped etc are application specific and many other membership

functions have been developed by researcher for their application specific use.

To construct a fuzzy logic based systems it is necessary to obtain knowledge from

human domain experts in form of fuzzy IF-THEN rule analysis. These rules are

combined together and applied on the given problem domain. The fuzzy logic

systems constitutes of four basic components, i.e. Rule Base, Inference Engine,



Fuzzifier and De-Fuzzifier. Type 2 fuzzy system uses membership functions that

constitutes of type 2 fuzzy sets. For type 2 fuzzy logic same components are used

along with additional type reducer that helps in type reduction from type 2 fuzzy logic

to simple fuzzy logic. The type reducer generalizes type 2 fuzzy sets to type 1 fuzzy

sets using type 2 membership functions. Figure 2.3 represents the flow of type 2

fuzzy logic system. The detailed mathematical equation for fuzzy logic components

can be found in paper written by klir & Yuan in 1995[12] and interval type method

was proposed by Qilian Liang & Jerry Mendel in 2000 [18] for type 2 fuzzy logic.

Figure 2.3: Flow of Fuzzy Logic (Type 2) Systems

Inference Engine

The basic function of the inference engine is to compute level(s) of belief in output

fuzzy sets from the levels of belief in the input fuzzy sets. The output is a single

belief value for each output fuzzy set. In this stage, the fuzzy operator is applied in

order to gain a single number that represents the result of the antecedent for that

rule. The inference engine is mainly based on “rules”. Rules determine the closed-

loop behavior of the system. The rules are based on expert opinion, operator

experience, and system knowledge. The basic function of the rule base is to

represent in a structured way the control policy of an experienced process operator

and/or control engineer in the form of a set of production rules such as if (process

state) and then (control output).



Fuzzy Inference Operators

The process of mapping the given input to output using fuzzy logic is carried out

using fuzzy inference operators. The fuzzy inference operators evaluates fuzzy

membership functions using specific output. The following are few of the fuzzy

inference operators used with a specified method,

Dienes – Rescher Implication – μR(x, y) = max (1 − μA(x), μB(y)) (1)

Zadeh Implication – μR(x, y) = max ( min ( μA(x), μB(y) ), 1 − μA(x)) (2)

Lukasiewicz Implication – μR(x, y) = min (1, 1 − μA(x) + μB(y)) (3)

Godel Implication – μR(x, y) = � �� ( ) � � � � (4)

Minimum Implication – μR(x, y) = min (μA(x), μB(y)) (5)

Product Implication – μR(x, y) = μA(x) ・ μB(y) (6)

All the operators mentioned above are individual rule based inference.

Composition Based Inference

The ways in which rules are combined depend on the interpretation or meaning of

the rule in general sense. When rules are viewed as independent conditional

statements, then a reasonable mechanism for aggregating nR individual rules Ri

(fuzzy relation) is the union:

≐ ��=1

= � 1 �, ,… , � �,

(8)

On the other hand, if rules are seen as strongly coupled conditional statements, their

combination should also have facility of intersection operator:

≐ ��=1

= � 1 �, , … , � �, (9)



Based on the fuzzy inference operator mentioned in equations (1) to (6) and their

generalized form as shown in equation (8) and (9) on applying composition based

inference respective inference engines namely Dienes – Rescher, Zadeh,

Lukasiewicz, Godel, Minimum, Product, etc. are obtained.

Fuzzy Inference Models

To deduce fuzzy inference from mathematical equation of fuzzy logic there was a

need of generalized fuzzy model. Following are the major model for inference in

fuzzy inference systems (FIS).

Mamdani Model: Mamdani Fuzzy Inference System (FIS) [21] was initially proposed

for controlling a steam engine and boiler combination by synthesizing a set of

linguistic control rules obtained from experienced human operators. Following steps

are used to compute output based on Mamdani inference logic.

Determine a set of fuzzy rules.

Fuzzify the inputs using the input membership functions.

Combine these fuzzified inputs with fuzzy rules to compute strength of fuzzy

rule.

Obtain the consequence of the rule by hybridizing the rule strength with

output membership functions.

Combine the obtained consequence to get output distribution.

Defuzzify these outputs (This step is required only when the result is required

in crisp data form).

Sugeno Model: The Sugeno fuzzy model also known as TSK fuzzy model was

proposed by Takagi, Sugeno and Kang [24], [21] to develop a systematic approach

to generating fuzzy rules from specified input-output data set. The main difference

between Mamdani and Sugeno model is that the output consequence is not

computed by clipping an output membership function at the strength of the specified

rule. Actually Sugeno model does not use output membership function, instead the

output is a crisp number computed by multiplying each input by a constant and then

adding up the result. “Rule Strength” is reoffered as “degree of applicability” and the



outputs are called as actions. Also there is no output distribution. There is only a

“resulting action” which is a mathematical combination of rule strengths and outputs.

Tsukamoto Model: In this model the consequent of each fuzzy IF-THEN rule is

represented by a fuzzy set with a monotonical membership function, hence as a

result the inferred output of every rule is defined as a crisp value induced by the

rules firing strength [21]. A output of each rule is a crisp value, the Tsukamoto fuzzy

model aggregates the output of each rule by weighted average and thus saves the

time consumed by the process of Defuzzification. However this model is not

practically feasible as it is not as transparent compared to Mamdani and Sugeno

model.

Type 2 Fuzzy Based Inference System

The last decade saw how type 2 fuzzy logic based system replaced traditional fuzzy

inference systems. A group of researchers led by Jerry Mendel made way to

practically implement the idea of type 2 fuzzy logic which was initially proposed by

prof. Zadeh. The basis of type 2 fuzzy logic is type 2 fuzzy sets. Type 2 fuzzy sets

incorporate uncertainty as extra third dimension which gives much clear and logical

information about the problem under research. A type 2 fuzzy sets À is characterized

by type 2 fuzzy membership function:

µÀ (x,u) where x ∈ X and u ∈ Jx ⊆ [0,1]

À = , � , �À , � ∀ ∈ �, ∀ � ∈ � ⊆ 0,1 where, 0 �À , � 1.

Also computation of the system is significantly reduced when, �À , � = 1.

Example of type 2 fuzzy membership functions are Gaussian membership functions

with uncertain mean or uncertain standard deviation. The membership function of

type 2 fuzzy sets is also a three dimensional, where the third dimension is the value

of the membership function at each point on its two dimension domain; this is called

as footprint of uncertainty (FOU). The footprint of uncertainty actually represents the

vagueness of type 1 membership functions and it is completely described by its two



bounding functions lower membership function (LMF) and upper membership

function (UMF) both of which are basically of the type 1 fuzzy sets. Hence it is much

easier to shift from type 1 fuzzy set to type 2 fuzzy sets without wasting much time in

understanding it. Prof. Mendel and his students introduced interval type 2 fuzzy

systems which actually help to represent footprint of uncertainty in more precise

manner. This in turn led to added advantage of representing unclear ideas or the

degree of relativity in much better and efficient way. The differential functions offered

by type 2 fuzzy logic are much easy to interpret, as they are non-zero equations.

The Defuzzification Methodologies

The defuzzificatation is the process of conversion of fuzzy logic values, obtained

from fuzzy inference engine by applying fuzzy membership functions, back to crisp

values. The process of defuzzification is required only when output is required in

crisp values. There are different defuzzification methods which enables to convert

inferred fuzzy rules back to crisp values, some of the standard defuzzification

methods are;

Centroid – It returns the center of area under the curve, when thinking of a

plain surface, the centroid can be defined as the point along the x – axis,

about which this shape would balance.

Bisector – is a vertical line that divides a region into two sub region of equal

area, gives values approximately similar to that of Centroid function.

Middle, Smallest & Largest of Maximum – these are three different methods

with a similar approach, these three methods key off the maximum values

assumed by the aggregate membership functions, If the aggregate function

has unique maximum value, then all the three i.e. Middle, Smallest & Largest

of Maximum take the same value.

Generally the centroid method is the most commonly used method when the

situation is not known, however at a later stage it is possible to change

defuzzification method to select the most appropriate methods. Some other methods

used for defuzzification also includes AI (adaptive integration), BADD (basic

defuzzification distributions), CDD (constraint decision defuzzification),



COA (center of area), COG (center of gravity), ECOA (extended center of area),

EQM (extended quality method), FCD (fuzzy clustering defuzzification), FM (fuzzy

mean), FOM (first of maximum), GLSD (generalized level set defuzzification), ICOG

(indexed center of gravity), QM (quality method), RCOM (random choice of

maximum), SLIDE (semi-linear defuzzification), and WFM (weighted fuzzy mean),

however these methods are application specific. Wu and Mendal[28] have shown

working of these methods for interval type 2 fuzzy systems.

Generation of Knowledge Base

When a database of rules is processed by inference engine knowledge base is

generated. The knowledge base generated through fuzzy inference is domain

specific according to rules fired by the inference engines which are stored inside

database.

Figure 2.4: Knowledge Base

The raw data regarding a specific domain under study is stored inside database in

an organized manner. The data is retrieved as per operational requirement of the

data. On the selected set of data a set of rules are to be applied. These rules are in

the form of logical condition‟s like IF-THEN ELSE statements and structured in

formed of decision trees, which decides the appropriate rule to be fired on

encountering certain conditions, and finally rule applicability is inferred by judging the

degree of truthiness of the rule under execution. Figure 2.4 represents knowledge

base with its parameters/entities. Table 2.1 describes these parameters/entities in

brief.

Operating Parameters (knowledge Tuning)

Connecting Parameters (Behavior Learning)

Structural Parameters (Structure Learning)

Logical Parameters (System Design)

Database

Rule base

Inference

Engine

Fuzzifier De-Fuzzifier

Knowledge Base



Class Parameters/Entities Components

Logic

Reasoning Mechanism Inference Engine

Fuzzy Operators

Type of Membership Functions Fuzzifier and

Defuzzifier

Defuzzification Method Defuzzifier

Structure

Relevant Variables

Knowledge Base Number of membership functions

Number of Rules

Connection

Antecedent of Rule

Rule base Consequent of Rule

Rule Weight

Operation Membership function value Database

Table 2.1: Parameters of Fuzzy Systems

Hence the fuzzy system goes under various phases of execution; the final decision

obtained will represent the knowledge base for the fuzzy system in a given domain.

Knowledge base serves as a backbone for fuzzy inference based expert systems.

2.4 Neuro-Fuzzy Hybridization

On the basis of the study made on artificial neural network and fuzzy logic, it is

obvious that both artificial neural network and fuzzy logic have limitations. To briefly

summarize, following are the advantages and limitations for artificial neural network

and fuzzy logic:

Advantages of Fuzzy Logic

Mimic human decision making to handle vague or uncertain condition.

Rapid computation due to parallel processing attribute.

Ability to deal with imprecise or imperfect information.

Resolving conflicts by applying underlying information of linguistic variable

with help of collaboration, aggregation and propagation.



Improved knowledge representation and modeling of complex, non-linear

problems.

It also helps in documentation of knowledge in rule form.

Natural language processing capability with the use of linguistic variable for

better reasoning.

Limitations of Fuzzy Logic

No self learning capability.

Needs help of domain experts to identify rule for data relationship.

It is abstract form of crisp logic and presents heuristic information.

Advantages of Artificial Neural Network

Mimics the working of biological neural network (Human brain).

No need to know about data relationship as the network can be self

organized.

It has self learning capability.

It has self tune itself adaptively.

Applicable to different models on various systems.

Limitations of Artificial Neural Network

Unable to process linguistic information.

Unable to manage imprecise or vague information, hence it is unable to

resolve conflicts based on numeric data combined with linguistic and logical

data.

Rely on trial and error mechanism to determine hidden layers and nodes.

Hence the objective is to remove the limitations and use the advantages offered by

both the fields - artificial neural network and fuzzy logic. Hybridizing these two

paradigms will remove their limitations and create better decision support and import,

maximum degree of intelligence.

In 2007 Ajith Abraham summarized state of art modeling techniques for neuro-fuzzy

systems [1], [2]. The architectures are categorized in following different categories:



Cooperative Model: In cooperative model artificial neural network processes data

before it is applied to fuzzy inference system. Here a pre-processing phase is carried

out, where artificial neural network first determines some components of the fuzzy

system. The output of the artificial neural network determines the membership

function or in simpler words fuzzy rules for inference engine of fuzzy inference

system. Once the input parameter for fuzzy inference systems are made available

the artificial neural network goes in background and lets membership function

decides the output for the given neuro-fuzzy system.

Concurrent Model: In the concurrent systems the neural network and the fuzzy

system work together continuously. Hence, the neural networks pre-process the

inputs or post-process the outputs of fuzzy system. The major difference between

the cooperative model and concurrent model is that, the artificial neural network

keeps on churning the data for fuzzy inference system continuously. This helps in

situation where input variables of fuzzy inference system cannot be controlled

directly. Hence sometimes it might happen that the output of fuzzy inference system

might be redirected to artificial neural network for further processing to determine the

fuzzy input control variable.

Hybrid (Fused) Model: In hybrid neuro-fuzzy architecture the artificial neural

network determines the variables for fuzzy inference systems, which enables them to

share data structure and knowledge representations. This process is carried out in

iterative fashion. The easiest way to apply learning algorithm to fuzzy logic is to

generate a special ANN like hybrid structures. However the theory does not fit to

practical applicability as conventional learning algorithms are gradient decent and

the function used for fuzzy inference are non-differentiable. Hence to overcome this

problem either standard learning algorithm for artificial neural network should be

followed or differentiable function should be used to deduce fuzzy inference. The

significant work in this area are GARIC[4], FALCON[13], ANFIS[8], NEFCON[15],

FUN[22], SOFIN[9], FINEST[25], EFuNN[11], dmEFuNN[11], evolutionary design of

neuro-fuzzy systems[2] and many other researcher have contributed in this area.

(i) FALCON: Fuzzy Adaptive Learning Control Network (FALCON)[13] uses a

five layered architecture, two linguistic nodes are determined for each output



variable. One of them is for training the data (or desired output) and the

second one represents the actual output. Fuzzification of each input variable

is carried out by the first hidden layer; the second hidden layer specifies the

conditions that are to be followed by the consequent of the rule in third hidden

layer. An unsupervised hybrid learning algorithm is used to locate

membership functions or rule base and a gradient decent approach is

followed to optimally adjust the parameter of membership function to produce

desired output.

(ii) ANFIS: Adaptive Neuro-Fuzzy Inference System (ANFIS)[8] implements

Takagi-Sugeno based fuzzy inference system. It also has a five layered

architecture. The first hidden layer is for Fuzzification of input variables,

second hidden layer uses T-norm operator to compute the antecedent part,

the third hidden layer normalizes the rule strength followed by the fourth

hidden layer which determines the consequent part of the rule. The learning

procedure is carried out in two parts; the first part propagates the input pattern

and the optimal consequent which is estimated by the least mean square

method, here the premise parameter are fixed for the current cycle through

the training set, in the second part the patterns are propagated again using

the backpropagation algorithm which in turn modifies the premise parameter

while the consequent parameters are fixed. This procedure is then iterated.

(iii) GARIC: Generalized Approximated Reasoning based Intelligent Control

(GARIC)[4] generates a neuro-fuzzy controller using two artificial neural

networks, the Action Selection Network (ASN) and Action State Evaluation

Network (AEN). The AEN is an adaptive network that evaluates the actions

generated by ASN. ASN used in GARIC is a feed forward network with five

layers, here the connections are not weighted connection. The first hidden

layer stores linguistic values of all input variables, the second layer represents

fuzzy rule nodes, the third hidden layer represents the linguistic values of the

output control variable. GARIC uses a mixture of gradient descent and

reinforcement learning to fine tune the node parameters.



(iv) NEFCON: Neuro-fuzzy Control (NEFCON)[15] was designed to implement

the Mamdani fuzzy inference system. Connection in NEFCON are weighted

with fuzzy sets and rules, the same antecedent used shared weight, which are

represented by ellipse drawn around the connections. They ensure the

integrity of the rule base. The fuzzification is carried out through input unit; the

propagation function represents the inference logic and the output unit is the

defuzzification interface. The learning process is a mixture of back

propagation and reinforcement learning. It can be used to learn initial rule

base where no prior knowledge is available.

(v) FINEST: Fuzzy Inference and Neural Network in Fuzzy Inference Software

(FINEST)[25] is capable of two kind of tuning processes, fuzzy predicate

tuning and combination of functions and tuning of implication functions. It uses

backpropagation algorithm to fine tune its parameters and thus provides a

framework to tune any parameter, which appears in the node of the network

representing the calculation processes of the fuzzy data, if the derivative

function with respect to its parameter is given.

(vi) FUN: Fuzzy Net (FUN)[22] performs fuzzification in first hidden layer with the

help of membership functions and in second hidden layer it performs the

conjunction operations. Membership functions of the output variable are

stored in third hidden layer. The network is initialized with fuzzy rule base and

it uses stochastic learning techniques that randomly change the parameters of

the membership function and connection within the network. The learning

process is driven by cost function and if the modified results are better than

previous result they are retained otherwise the process is repeated.

(vii) EFuNN: Evolving Fuzzy Neural Network (EFuNN)[11] creates nodes

dynamically during learning process. The input layer passes data to the

second layer; the second layer calculates the fuzzy membership degree to

which input values belong to predefined fuzzy membership functions. The

third layer consists of nodes containing fuzzy rules that determine the

input/output data. Every rule node is defined by 2 vectors of connection

weights, which are adjusted through hybrid learning techniques. The fourth



layer calculates the applicability of the degree of membership function and the

fifth layer performs Defuzzification. A variant of EFuNN is Dynamic EFuNN

(DEFuNN)[11] was proposed with idea that not only the wining rule node to be

propagated but a group of rule node is dynamically selected with every new

input vector and their activation values are used to calculate dynamic

parameters of the output function. Also the major difference is that EFuNN

implements Mamdani model while DEFuNN implements Takagi-Sugeno

model.

(viii) SONFIN: Self Constructing Neural Fuzzy Inference Network (SONFIN)[9]

implements Takagi-Sugeno model for fuzzy inference system with

modification. In the phase of structure identification the input space is

portioned in a flexible way according to aligned cluster based algorithm. With

the help of projection based inference measure some selected additional

variable are added to consequent part incrementally as learning proceeds. To

identify parameter, least mean square or recursive least square method is

used to fine tune consequent parameter and to fine tune precondition

parameter backpropagation algorithm is used.

(ix) Evolutionary Design of Neuro-Fuzzy System: A five-tier hierarchical

evolutionary search procedure is used. The evolving neuro-fuzzy system can

adapt to Mamdani or Takagi-Sugeno based fuzzy inference system. The basic

architecture defines only the layers not procedure. The evolutionary search

process actually decides the optimal type and quantity of nodes and

connection between layers. The Fuzzification layer and the rule antecedent

layer behaves similarly to any other neuro-fuzzy model and the consequent

part of the rule will be determined according to the inference system

depending upon the type of the problem, which will be adapted accordingly to

evolutionary search mechanism. Defuzzification, aggregation operators will

also be adapted according to the fuzzy inference system chosen by the

evolutionary algorithm.



Besides the above models, there are dedicated software systems available for

artificial neural network and fuzzy logic. Some example of these systems are as

follows.

For development of artificial neural network based systems various commercial tools

like Forecaster/XL[http://www.alyuda.com/neural-network-software.htm], Neuro

Intelligence[http://www.alyuda.com/neural-network-software.htm],

Neurosignal/XL[http://www.alyuda.com/neural-network-software.htm],

predictor[http://www.alyuda.com/neural-network-software.htm], Neural Planner

Plus[http://www.easynn.com/], Easy NN plus[http://www.easynn.com/], Neural

Power[http://www.neuroshell.com/], Neural Shell

Predictor[http://www.neuroshell.com/], and many more. Public domain tools like

brainbox[http://brainbox.sourceforge.net/], tiberirusXL[www.tiberius.biz/],

JNNS[http://www.ra.cs.uni-tuebingen.de/downloads/JavaNNS/], and

SNNS[http://www.ra.cs.uni-tuebingen.de/SNNS/] are available but either, they are in

terms with license or the output generated is not modifiable.

For development of fuzzy logic based systems Various fuzzy editors or tools or

programs like

FuzzyCOPE[https://www.aut.ac.nz/resources/research/research_institutes/kedri/dow

nloads/pdf/fuzzycope.pdf], Fuzzy Logic Inference Engine, Fuzzy

Clips[http://www.graco.unb.br/alvares/DOUTORADO/omega.enm.unb.br/pub/doutor

ado/disco2/ai.iit.nrc.ca/IR_public/fuzzy/fuzzyClips/fuzzyCLIPSFiles.html],

NEFCLASS[http://fuzzy.cs.uni-magdeburg.de/nefclass/], Fuzzy logic development

environment for embedded systems, fuzzy logic inferencing

toolkit[http://www.mathworks.in/products/fuzzy-

logic/index.html;jsessionid=e17603dbc9ea8bab686ae8b65030],

FuzzyTECH[http://www.fuzzytech.com/] are available on the internet with a limited

functionalities which includes terms and conditions.

Much later software packages like ANFIS and DENFIS were introduced in

MATLAB[http://www.mathworks.in] software. Further these software‟s had fixed set

of activation function and fixed methods of learning, hence to generate new

activation function or learning methods coding had to be done from the beginning.

http://www.alyuda.com/neural-network-software.htm




http://www.easynn.com/

http://www.easynn.com/

http://www.neuroshell.com/

http://www.neuroshell.com/

http://brainbox.sourceforge.net/

http://www.tiberius.biz/

http://www.ra.cs.uni-tuebingen.de/downloads/JavaNNS/

http://www.ra.cs.uni-tuebingen.de/SNNS/

https://www.aut.ac.nz/resources/research/research_institutes/kedri/downloads/pdf/fuzzycope.pdf



http://www.graco.unb.br/alvares/DOUTORADO/omega.enm.unb.br/pub/doutorado/disco2/ai.iit.nrc.ca/IR_public/fuzzy/fuzzyClips/fuzzyCLIPSFiles.html



http://fuzzy.cs.uni-magdeburg.de/nefclass/

http://www.mathworks.in/products/fuzzy-logic/index.html;jsessionid=e17603dbc9ea8bab686ae8b65030

http://www.mathworks.in/products/fuzzy-logic/index.html;jsessionid=e17603dbc9ea8bab686ae8b65030

http://www.fuzzytech.com/

http://www.mathworks.in/



Moreover the MATLAB files are not easy to configure with other programming

languages or to deploy them on web.

The aforementioned models, techniques and software systems are application

specific and not generic. Hence at this point of time, there is no generalized

model/tool available for neuro-fuzzy hybridization. The developer of the application

has to choose which model will best fit in the situation of given problem under study.

2.5 Conclusion

The chapter provides a literature review on various neuro-fuzzy modeling

techniques. Also according to situations different neuro-fuzzy modeling techniques

can be used. Through the in-depth literature survey made here it was possible to

study and design the most prominent neuro-fuzzy modeling techniques in the

research work. Furthermore advantages of using neuro-fuzzy system are also listed

which overcomes the limitations of artificial neural network and fuzzy logic when

used separately. Thus designing of library will become simpler based on the

literature survey and will aid in incorporating the beneficial features of artificial neural

network along with type 2 fuzzy logic in order to generate a generic library for the

neuro-fuzzy system. Further it is observed that, the neuro-fuzzy model helps in

extending interpretability. Thus we can conclude that by combining these two major

components soft computing, i.e. artificial neural network and fuzzy logic one can

generate hybrid intelligent system that takes advantages of both the components

and overcomes their limitations to generate intelligent application which will be most

needed in the coming future to meet daily requirements and expectation of a

common man of modern world.

References

[1] Abraham, Adaptation Neuro Fuzzy Systems: State-of-the-art Modeling

Techniques, http://arxiv.org/ftp/cs/papers/0405/0405011.pdf, retrieved on 26th

December 2012.

http://arxiv.org/ftp/cs/papers/0405/0405011.pdf



[2] Abraham, Adaptation of Fuzzy Inference System Using Neural Learning,

http://www.softcomputing.net/nf_chapter.pdf, retrieved on 26th december

2012.

[3] Bertsekas, D.P., Tsitsiklis, J.N. , Neuro-dynamic programming. Athena

Scientific. pp. 512.,1996.

[4] Bherenji H R and Khedkar P, Learning and Tuning Fuzzy Logic Controllers

through Reinforcements, IEEE Transactions on Neural Networks, Vol .3, pp.

724-740, 1992.

[5] G. Cybenko. Approximation by superposition of a sigmoidal function. Math.

Control Systems Signals, Vol.2, pp.303-314, 1989.

[6] Hebb, D.O. Organization of Behavior: A Neuropsychological Theory (New

York: John Wiley and Sons).,1949.

[7] J. J. Hopfield, "Neural networks and physical systems with emergent

collective computational abilities", Proceedings of the National Academy of

Sciences of the USA, Vol. 79 no. 8 pp. 2554–2558, April 1982.

[8] Jang R, Neuro-Fuzzy Modeling: Architectures, Analyses and Applications,

PhD Thesis, University of California, Berkeley, July 1992.

[9] Juang Chia Feng, Lin Chin Teng, An Online Self Constructing Neural Fuzzy

Inference Network and its Applications, IEEE Transactions on Fuzzy Systems,

Vol 6, No.1, pp.12-32, 1998.

[10] K. Funahashi. On the approximate realization of continuous mappings by

neural networks. Neural Networks, Vol.2, pp-183-192, 1989.

[11] Kasabov N and Qun Song, Dynamic Evolving Fuzzy Neural Networks with 'm-

out-of-n' Activation Nodes for On-line Adaptive Systems, Technical Report

TR99/04, Department of information science, University of Otago, 1999.

[12] Klir, G. and Yuan, B. Fuzzy Sets and Fuzzy Logic, Theory and Applications,

chapter 3, pages 50-96. Prentice Hall, Upper Saddle River, New Jersey,

USA.,1995.

[13] Lin C T & Lee C S G, Neural Network based Fuzzy Logic Control and

Decision System, IEEE Transactions on Comput. Vol.40 No.12, pp.1320-

1336, 1991.

[14] McCulloch, W. and Pitts, W. A logical calculus of the ideas immanent in

nervous activity. Bulletin of Mathematical Biophysics, Vol.7, pp.115 – 133,

1943.

http://www.softcomputing.net/nf_chapter.pdf



[15] Nauck D, Kruse R, Neuro-Fuzzy Systems for Function Approximation, 4th

International Workshop Fuzzy-Neuro Systems, 1997.

[16] P. Schneider, B. Hammer, and M. Biehl. Adaptive Relevance Matrices in

Learning Vector Quantization.Neural Computation 21: 3532–3561, 2009.

[17] Pearlmutter B.A., Dynamic Recurrent Neural Network, CMU,1990.

[18] Q. Liang and J. Mendel, Interval Type-2 Fuzzy Logic Systems: Theory and

Design. IEEE Trans. On Fuzzy Systems, vol. 8, no.5, pp. 535-550, 2000.

[19] Rosenblatt, Frank., The Perceptron: A Probabilistic Model for Information

Storage and Organization in the Brain, Cornell Aeronautical Laboratory,

Psychological Review, v65, No. 6, pp. 386–408.,1958

[20] Rumelhart, D. E., McClelland, J. L., & the PDP research group. Parallel

distributed processing: Explorations in the microstructure of cognition. Vol.1,

MIT Press, Chapter 8, pp.318-362, 1986.

[21] S.N.Sivanandam & S.N.Deepa ,Principles Of Soft Computing, pp.697-710.,

2008.

[22] Sulzberger SM, Tschicholg-Gurman NN, Vestli SJ, FUN: Optimization of

Fuzzy Rule Based Systems Using Neural Networks, In Proceedings of IEEE

Conference on Neural Networks, San Francisco, pp. 312-316, March 1993.

[23] T. Kohonen. Self-Organizing Maps. Springer, Berlin, 1997.

[24] T. Takagi and M. Sugeno, “Fuzzy identification of systems and its applications

to modeling and control,” IEEE Trans. on Syst., Man, Cybern., vol. 15, pp.

116-132, 1985.

[25] Tano S, Oyama T, Arnould T, Deep combination of Fuzzy Inference and

Neural Network in Fuzzy Inference, Fuzzy Sets and Systems, vol.82 no.2 pp.

151-160, 1996.

[26] Werbos, P.J. Beyond Regression: New Tools for Prediction and Analysis in

the Behavioral Sciences. Ph.D. Thesis, Committee on Applied Mathematics,

Harvard U., 1974. Reprinted in its entirety in Werbos, P.J.: The Roots of

Backpropagation: From Ordered Derivatives to Neural Networks and Political

Forecasting. Wiley, New York (1994). Originally published in 1974.

[27] Widrow, B. and Hoff, M.E., Adaptive switching circuits. IREWESCON

Convention Record Part IV pp. 96-104., 1960.



[28] WU, H. AND MENDAL, J.M., Uncertainty bounds and their use in the design

of interval type-2 fuzzy logic system. IEEE Transactions on fuzzy systems,

Vol.10, no.5, pp. 622-639., 2002.

[29] Zadeh, L.A. Fuzzy sets, Information and Control, Vol. 8, 338-353. 1965.

[30] Zhang, G., Patuwo, B.E. and Hu, M.Y., Forecasting with artificial neural

networks: The state of the art. International Journal of Forecasting Vol.14,

pp.35–62., 1998

literature survey 2shodhganga.inflibnet.ac.in/bitstream/10603/40078/10/10_chapter2.pdf · design of...

Documents