neural networks 1.ppt - fitstaff.fit.ac.cy/com.ke/files/acoe402/neural_networks_1.pdf · acoe 402...
TRANSCRIPT
ACOE 402 Neural Networks and Fuzzy LogicACOE 402 Neural Networks and Fuzzy LogicArtificial Neural Networks
Efthyvoulos C. Kyriacou (PhD) Assoc. Prof.
Computer Science and Engineering Department
Resources: Dr. Neocleous Costas, Cyprus Univ. of Technology
Neural Networks, S. Haykin, Prentice hall 1999Fundamentals of Neural Networks, L. Fausett, Prentice hall 1994
Computational Intelligence (CI) is a special case of A ifi i l I lli (AI)
What is intelligence?
Artificial Intelligence (AI)
What is intelligence?
It is difficult to be clearly definedIt is difficult to be clearly defined.
It may not be needed to be defined to be understoodIt may not be needed to be defined to be understood.
Terman* suggests that it is “the capacity for abstract thought”.gg p y g
Most psychologists though, agree that it is “the capacity for effective adaptation to an environment, which is done throughchange in the organism, or change in the environment, or even
ti f i t”creation of a new environment”.__________________________________________________________________________________________________* Terman L. (1916). The uses of intelligence tests. Ιn “The measurement of intelligence”. Boston, Houghton Mifflin. 2
Here is a more practical suggestion:
Intelligence is the general mental ability involved in h l l ti iprocesses such as calculation, reasoning,
classification, learning, using language, d d h lfunderstanding the environment, self‐correcting,
inventing and adjusting to new situations.
Artificial Intelligence then is human‐created, non‐g ,biological intelligence.
Computational Intelligence is intelligence that emerges through some form of computationthrough some form of computation.
3
AI and CI may lead to solutions of difficult yproblems in science and engineering.
They have also instigated a revitalization of fundamental philosophical questionsfundamental philosophical questions.
They also generated new frontiers of explorationsThey also generated new frontiers of explorations and introduced new questions.
4
Fundamental questions in relation to AIFundamental questions in relation to AICan a biological brain generate an artificial brain and i t lli ?intelligence?
C th h b i “ d t d” it f ti i ?Can the human brain “understand” its own functioning?
Will h d t di b b l t ?Will such understanding be absolute?
C b i ith i l ll ti l tCan our brain, either singly or collectively generate a new kind of intelligence?
How is WILL generated?
5
What is consciousness?What is consciousness?
Can universal consciousness exist?Can universal consciousness exist?
How does creativity appear?How does creativity appear?
How are illusions love hate generated?How are illusions, love, hate, … generated?
Can matter generate cognition ? And how?Can matter generate cognition ? And how?
6
Wh t i ?What is memory?
Can there be olle ti e memor (or national) memor ?Can there be collective memory (or national) memory?
How are numbers words meaning codified?How are numbers, words, meaning codified?
How does the brain process logic?How does the brain process logic?
How can the brain handle with ease both crisp and fuzzyHow can the brain handle with ease both crisp and fuzzy logic?
7
Computational Intelligence ToolsComputational Intelligence ToolsExpert systems (ExS)
Artificial neural networks (ANN)
Evolutionary systems (ES)Genetic algorithms (GA)Artificial life (AL)Coevolutionary roboticsEvolvable hardware
Fuzzy systems (FS)…
8
I believe that the Artificial Neural Networks are presently the most promising technology that may lead to non‐programmed, intelligent artificiallead to non programmed, intelligent artificial systems.
ANNs will be presented here in some detailANNs will be presented here in some detail.
9
Artificial IntelligenceArtificial Intelligence
Weak Strongapproach approach
Human knowledge and specific expressions of it can be simulated through
Human knowledge and specific expressions of it, emerge naturally in
computational systems computational systems
Classic AI (symbolic AI)It is expressed through symbolic entitiesthat may be properly coded
Connectionist AIExpressed through non‐symbolic methods such as the ANN. The various cognitive processes
Some important proponents are:Dennet, Newell and Simon, Chomski, Minsky, Fodor and Pylyshyn
emerge naturally from the dynamically connected cells, as they evolve when influenced from an environment.
10
Fodor and PylyshynSome important proponents are:Smolensky and Hameroff.
Brief explanation of the methodsBrief explanation of the methods
E t tExpert systemsAlso known as Knowledge Based Systems
It is a system that describes the behaviour of one or more experts in some field, by using symbols and rules.
Th f l i li d k l d b i iThe set of rules is applied to a knowledge base, aiming at extracting useful information.
There are limitations because of the need to have rules for the rules of the exponentially growing size and the difficultythe rules, of the exponentially growing size, and the difficulty in finding the rules.
11
Main characteristics of Expert Systems:p y
The programming and the conclusions are based on rules.
They are not easily made tolerant to mistakes.
If the knowledge base changes reprogramming may beIf the knowledge base changes, reprogramming may be needed.
Explanation of results is easier to implement with these systems.
12
Brief explanation of Artificial Neural NetworksBrief explanation of Artificial Neural Networks
IntroductionIntroduction
Biological neuronsg
Biological neural networks
Artificial neurons
Artificial neural networks
LearningLearning
Applicationspp
13
Other names for the Artificial Neural Networks
A tifi i l N l S tArtificial Neural Systems
Parallel Distribution Processing Systems
Connectionist Systems
Neurocomputing SystemsNeurocomputing Systems
Adaptive Networks
Associative Networks
Collective Computation SystemsCollective Computation Systems
Neuromorphic Systems
14
The ANNs are based on the effort to mimic the ( )operational properties of the natural (biological) neural
networks.
They are composed of many artificial neuronsconnected in an organized system in which there is direct or indirect communication and interaction among all its members.
There is usually provision for information input and for the desired outputthe desired output.
Groups of neurons may organized into layers or slabsGroups of neurons may organized into layers or slabs.15
A i l b l b h i th h th f
An Artificial Neural Network
Artificial neural network
An emerging global behavior may appear through the use of simple, and local learning and evolution rules.
SYSTEM SYSTEM
Processing elementArtificial neural network
SYSTEM INPUT
SYSTEM OUTPUT
N 3N 2N1
N 5N 4 54
N 8N 7N 6
16Signal flow
To gain a better understanding of their functioning, let us have a brief encyclopaedic look on the biological neural networks.
Typical Αxon of sending neuron
SynapseDendrite of receiving neuron
biological neuron Nucleus
Cell body somaInternal voltage ≈ ‐ 60 to ‐ 80 mV
INITIAL SEGMENT
AXON HILLOCK
MembraneThickness ≈ 5 to 10 nm = 0.000005 to 0.00001 mmCapacitance ≈ 0.944 μF/cm2
Field intensity ≈ 12000000 V/m = 120 kV/cm
NODE OF RANVIER
TRIGGER ZONE
Αxon or nerve fiber
Μyelin sheath
Diameter: 0.5 ‐ 22 μm in vertebrate500 ‐ 1000 μm in the squid giant axon
DENDRITE OF RECEIVING NEURON17
INFORMATION TRANSMISSION(Generation of the Action Potential)(Generation of the Action Potential)
Excitation from the environment
The receptive sensors convert changes in the environment(light pressure chemical
Habituation
Stimulus
(light, pressure, chemical constituency, ...) in Graded Potential.
Sensory receptor
Synapse Graded Potential
Time The graded potential is gradually weakened as it is propagated towards the somaSy apse
mV
Time
propagated towards the soma.
Many graded potentials from different dendrites are
lmV
Neuron
accumulated to create a bigger or smaller one, which, once it reaches a threshold (–40mV), it becomes an Action PotentialAction Potential
Time
Neuron it becomes an Action Potential– (a train of pulsed voltages, nerve impulses or spikes)
18Another neuron
ΣύναψηSynapse ΣύναψηSynapse
Diameter ≈ 1 μm = 0.001mmGap ≈20 to 40 nm
= 0.00002 to 0.00004 mmDiameter/Gap ratio ≈ 100
Delay in the transmission of the pre‐synaptic potential to a post‐synaptic potential ≈ 0.3 to 1.0 ms
Velocity ≈ 0.2 cm/minute
19
The medium for the signal transmission is basically of electrochemical nature.
There exists a great variety of synapses.
Even the position of a synapse makes a difference in signal transmissiontransmission.
The transmission of coded information at the synapses is y pprimarily done with chemical substances known as neurotransmitters [e.g. acetylcholine, norepinephrine, d h d b ddopamine, 5‐hydroxytryptamine serotonin, aminobutyric acidGABA].
These phenomena are mainly occurring in mammal nervoussystems.y
20
ComparisonsComparisonsThere are about 1010 – 1012 neurons in the brain
and about 1013 – 1016 synapses.
Organism leech Worm Fly Cockroach Bee Man
Number of Synapses >104 >105 ~109 <1011 >1011 ~1014
21
Synapses
There also exist electrical synapses, but these occur mainly in lowerThere also exist electrical synapses, but these occur mainly in lower animals.
The systematic use of a synapse is believed to improve its efficacyimprove its efficacy.
learning, memoryg, y
Hebb’s Rule
22
Mathematical models of the biological neuronMathematical models of the biological neuron
The Hodgkin – Huxley model, 1952
C (dV/dt) I g m3h(V V ) g n4(V V ) g (V V )
g y ,
Cm(dV/dt)=Im–gNam3h(V–VNa)–gKn
4(V–VK)–gL(V–VL)dm/dt = αm(1 –m) – βmm
dn/dt = αn(1 – n) – βnn
dh/dt = αh(1 – h) – βhh
23
The Hodgkin – Huxley modelThe Hodgkin – Huxley modelWhere,
Vrest = – 60 mVVna = 50mVVK = – 77 mVV 54 402 VVL = – 54.402 mVgNa = 120 mmho/cm2
gK = 36 mmho/cm2
g = 0 3 mmho/cm2gL = 0.3 mmho/cm2
E = V – Vrest
/ 25 E/10 β E/18αm = 0.125 – E/e 25‐E/10 – 1 , βm = 4e‐E/18
αn = 0.0110 – E/e10‐E/10 – 1, βn = 0.125e
‐E/80
αh = 0.07e‐E/20, βh = 1/1+ e
30‐E / 10
24
The FitzHugh Nagumo modelThe FitzHugh‐Nagumo model
This is a simpler form of the Hodgkin – Huxley model
duj/dt = uj ‐ uj3/3 ‐ zj + xduj/dt uj uj /3 zj x
d /dt k + k + kdzj/dt = ‐ k1j zj + k2j uj + k3j
25
BRAIN – NERVOUS SYSTEM
26
BRAIN – NERVOUS SYSTEM
The biological neural networks are non‐linear computational systems that are characterized by a highcomputational systems that are characterized by a high degree of parallelism, robustness and fault‐tolerance.
They show a capacity for learning, generalization, and handling vagueness.
It i ti t d th t th t t l l th f th b iIt is estimated that the total length of the brain connections are about 109 meters, which is about 25 times the earth perimeter!times the earth perimeter!
27
Even though the brain is only about 2% of the total weight of a person, it needs about 15% of the total blood and about 25% of the inhaled oxygen.
h d hLiving organisms that do not have nervous systems (bacteria, protozoa, invertebrate) can exhibit some form of behavior memory and even learningof behavior, memory and even learning.
For instance, the paramecium (a single‐cell organism)For instance, the paramecium (a single cell organism) can demonstrate learning behavior, even though it does not have brain and synapses.
28
Every neuron communicates locally and directly with many other neurons and indirectly
LAYER
other neurons and indirectly with all others.
LAYER
Groups of neurons form subsystems, layers, slabs, …
LAYER
y , y , ,
LAYER
The information transmission in synapses is done in parallel. Because of this, the frequency of changes is about 1016 per second. Taking into consideration the fact that at each synapse many neurotransmitters are also transferred, the information transmission is impressive.
29
Comparison of biological and artificial neurons and networksand artificial neurons and networks
BIOLOGICAL NEURONS
AND NETWORKS
ARTIFICIAL NEURONS
AND NETWORKS
Dense connections
~ (1012 )(104 )
Few connections
~ (1012 neurons)(104 synapses) =
= 1016 connections
Single neurons are different to one another Mostly similar to one anotherSingle neurons are different to one another
Modular structures
Autonomous local interaction
Mostly similar to one another
Partly modular
Non‐autonomous
Usually supervision is needed
Parallel processing
Very little energy consumption
Non‐mathematical or algorithmic operation
Usually supervision is needed
Mostly serial processing
Much energy consumption
Mostly mathematical or algorithmic description
30
Non mathematical or algorithmic operation y g p
Artificial Neural Networks
•They are flexible non linear systems composed ofmany•They are flexible, non‐linear systems composed of many cooperating processors, that, among others, help in:
•Study of biological neural systems.•Data processing and knowledge mining.M i d l ti t ti•Mapping and relation extraction.
•Classification, Pattern recognition.•Forecasting.Forecasting.•Study of dynamical systems.•Adaptive signal processing.•Automatic control of systems.•…
31
•It is hoped that through an autonomous evolution of•It is hoped that through an autonomous evolution of such connectionist systems, some form of intelligence may emergemay emerge.
•Will it lead to artificial brain?•Will it lead to artificial brain?
•Artificial mind?•Artificial mind?
•Artificial consciousness?Artificial consciousness?
32
Artificial neurons
•They are the basic building blocks of ANNs•They are the basic building blocks of ANNs.•They are also known as:• Units or Processing Elements
Their main characteristics are:
They have multiple inputs – one output (MISO).
They are non‐linear.
They exhibit adaptivity33
They exhibit adaptivity.
They can be implemented through appropriateThey can be implemented through appropriate software or hardware.
The hardware neurons can be:
Electronic, chemical, optical, micromechanical, nanomechanicalnanomechanical, ...
F ti ll th b i l t d ll lFunctionally, they can be simulated as cellularautomata.
Here is a generalized form of an artificial neuron.
34
General form of a single‐neuron modelInput information from the environment or from other neurons
Output to the environment or to other neurons
Learning system that adapts the various parametersp
POST‐ACCUMULATORPROCESSING
(Subsystem of functional
PRE‐ACCUMULATORPROCESSING
(Subsystem of functional and dynamical processors)
(including cross‐
MainAccumulator
TRIBUTO
R
F df d/F db k
and dynamical processors) , p2()
(including crosscorrelations), p1() DIST
Feedforward/Feedback subsystem
h()
35
Some useful properties of the artificial neural networks
•They can generalize in the sense that they conclude about an area of interest in which they hadn’t been previously taughtof interest in which they hadn t been previously taught.
•They show robustness. They continue to behave acceptably well if d l d difi deven if some neurons are deleted, or some parameters modified.
•They are fault tolerant.y
•The rules are expressed in a distributed processing and representationmannerrepresentationmanner.
•They usually learn through appropriate training, which can be id d ( i d) id d ( i d)guided (supervised) or unguided (unsupervised).
36
Structuring an ANNStructuring an ANNProcessing element
Artificial neural network
SYSTEM INPUT
SYSTEM OUTPUT
TN 3TN 2TN1
TN 5TN 4
TN 8TN 7TN 6
37
Signal flow
ANN Topologies, ArchitecturesNETWORK ARCHITECTURES
STATIC ARCHITECTURES DYNAMIC ARCHITECTURES
AUTONOMOUSLY ADAPTED
EXTERNALLY GUIDED
RANDOM CONNECTIONS
ORGANIΖED CONNECTIONS
ORGANIΖED FEEDFORWARD CONNECTIONS
(Without feedback)
SINGLE LAYER
TWO LAYER
THREE LAYER
MULTI‐ LAYER …
ORGANIΖED RECURRENT CONNECTIONS(With feedback)
SINGLE LAYER
TWO LAYER
THREE LAYER
(With feedback)
38
MULTI‐ LAYER …
CELLULAR NETWORKS
Some examples of ANN architecturesMultilayer perceptron (MLP)
x1 Σ
y = sgn(u )
x2 Σ
y1 = sgn(u1)
.
.Σ
. …Σ
y2 = sgn(u2)
xN ΣWeight, w
39For clarity, the rest of the weights are not shown
Multilayer, multi‐slab network
(hidden)
OUTPUTINPUT
(hidden)SLAB 2
SLAB 5SLAB 4
(hidden)
SLAB 5 SLAB 1
(hidden)
SLAB 3
40
Input ReceptiveWeightMatrix
p(1) α(1) WeightMatrix
A(1)
x1.
f1.
putVector
pField
IxK
Matrixw(1a) p1
(1)
.
.
.(1)
α1(1)
.
.
.(1) OxI
MatrixW(1b) A1
(1)
.
.
.(1)
A y...xk
.
.
.fk
IxK pi(1)
.
.
.
p (1)
αi(1) =
αi(1)
.
.
.
α (1)
OxI Ao(1)
.
.
.
A (1) +
A1
.
.
.
y1...
k...
fk...
JxK
pI( )
p1(2)
.
.
αI( )
α1(2)
.
.
OxJ
AO( )
A1(2)
.
.
+Ao
.
.
.
AO
yo...
yOxK fK
JxK
Weight
.
pj(2)
.
.
.
αj(2) = [ pj(2)]
.
αj(2)
.
.
.
OxJ .
Ao(2)
.
.
.Weight
AO yO
f
WeightMatrixw(2a)
pJ(2)
p(2)αJ(2)
α(2)AO
(2)
A(2)
WeightMatrixW(2b)
41
fk= akep(L) = w(La)f A(L) =W(Lb)α(L)
Kohonen architectures
42
Recurrent architecture
43
Cellular neural network (CNN) architecture
……
……
The artificial neurons… … … The artificial neurons communicate only with their peripheral
44
peripheral.
General comments on neural architectures
It is not necessary to code the information processing in the y p gformalism shown in the previous figures/structures.
We could use an algebraic/differential formalism or other kindWe could use an algebraic/differential formalism, or other kind of graphs such as:
INPUT OUTPUTHIDDEN UNITS
45
INPUT OUTPUT
etc
Learning in ANNs
x1(κ) w1(κ)Desired outputWeight adaptation
x2(κ) w2(κ)
d(κ)
+Comparison
x2(κ)
.
w2(κ)
Σ Σ‐
N
1) i i
iu(κ f x w
=
⎛ ⎞= ⎜ ⎟
⎝ ⎠∑.
. e(κ) = d(κ) – u(κ)Error
xN(κ) wN(κ)
The learning algorithm governs the way that the weights adapt so that the error gradually
N N( )
46
The learning algorithm governs the way that the weights adapt so that the error gradually becomes acceptably small. LEARNING ALGORITHM
Learning in ANNs
Haykin (1994):
Learning is a process by which the free parameters (weights) of a neural network are adapted through a continuing process ofneural network are adapted through a continuing process of stimulation by the environment in which the network is embedded.embedded.
A more general proposition is:
Learning is achieved through any change, in any g g y g , ycharacteristic of a network, so that meaningful results are achieved.
47
results are achieved.
This definition leads to the possibility of achieving learning through:learning through:
Synaptic weight modification.Network structure (topology) modifications.By creating or deleting neurons or connectionsBy creating or deleting neurons or connections.Through the use of suitable attractors or other it bl t bl i tsuitable stable points.
Learning through forgetting.Through appropriate choice of activation functions.Through modifying controllable parameters in aThrough modifying controllable parameters in a look‐up table defining an activation transformationtransformation.Combination of above. 48
Learning is usually a process of minimizing an error function or maximizing a benefit function.
In this respect, learning resembles the optimization, where we seek to extremize a desired criterion function (Error, Loss, Penalty, Computational energy Lyapunov ή Hamilton function)energy, Lyapunov, ή Hamilton function)
49
Learning rulesSome of the learning rules are significantly different than others, whilethere are many that are of a minor variation, or are special cases ofmore general ones usually identified by a different namemore general ones, usually identified by a different name.
Some important rules are:p
HebbDelta and its variations such as:
Backpropagation (the most common)AdalineAdalinePerceptron…
Competitive learningHopfieldpKohonenART 50
Applications of artificial neural networksIf the objective is to find a mapping such that the system captures the dynamics relating an input data set to an output captures the dynamics relating an input data set to an output data set, the problem is called function approximation or system identification in control engineering applications.
If the objective is to find a mapping such the system loosely i l i hi i d h bl iinterpolates within an input‐output data set, the problem is called generalization.
If the objective is to find a mapping such that the system generalizes to discrete output classes, the problem is called generalizes to discrete output classes, the problem is called classification. In some fields this problem is given different names such as associative memories, pattern association or
51
pattern recognition.
Applications of artificial neural networksIn Engineering
• Modeling manufacturing processesprocesses.
• Monitoring and diagnosis of machinery failures.
2x=θ&• Robotics and
autonomous vehicles.1x=θ
l
2x=θ
m
• Automatic control systems.
u cm
• Quality control. 52x=x3 0 4x=x&
In Medicine and Health Sciences
• Diagnostic applications in EEG, ECG, EMG and other medical signaturesmedical signatures.
• Artificial vision.
• Artificial hearing.
• Artificial nose• Artificial nose.
• Brain controlled telekinesis.
• Identifying RNA and DNA in proteins.
53
Neuromuscular disease diagnosis (UCY, ING)
54
Quantitative Assessment of the Prognostic Factors in Breast Cancer (UCY, ING)
Manual, semi–quantitative, immunohistochemical score
% of CellsPositive
Score StainingIntensity
Score TotalScore
Diagnostic
Index
0 0 Negative 0 0 0
0 25% 1 Weak 1 1 4 1+0 – 25 % 1 Weak 1 1 – 4 1+
26 – 50 % 2 Moderate 2 5 – 8 2+
51 – 75 % 3 Strong 3 9 – 12 3+
≥ 76 % 4 VeryStrong
4 > 13 4+Strong
55
In Signal Processing
• Classification – pattern recognition.
Si l id ifi i• Signal identification, speech recognition, i t itisignature recognition, image recognition,
k id tifi tispeaker identification.
• Seismic image analysis.
• Identification of explosives and militaryexplosives and military targets.
• Optical character recognition. 56
In Meteorology
• Modeling meteorological processes.
Filli i i i i f ll d (HTI UCY)Filling‐in missing rainfall data (HTI, UCY).
Creation of iso‐hyets.y
In Cognitive studies
Aoccdrnig to a rscheearch at Cmabrigde Uinervtisy it deosn't
In Cognitive studies
Aoccdrnig to a rscheearch at Cmabrigde Uinervtisy, it deosn t mttaer in waht oredr the ltteers in a wrod are, the olny iprmoetnt tihng is taht the frist and lsat ltteer be at the rghitiprmoetnt tihng is taht the frist and lsat ltteer be at the rghit pclae. The rset can be a total mses and you can sitll raed it wouthit porbelm. Tihs is bcuseae the huamn mnid deos not praed ervey lteter by istlef, but the wrod as a wlohe.
57
In Financial Engineering
• In decision‐making.
F i f k i di (UCY HTI)• Forecasting of market indices. (UCY, HTI)
• Stock exchange trends. (HTI, UCY)g ( )
• Bankrupsy prediction. (UCY)
• Optimization. (HTI, UCY)
58
In Prediction and forecasting
• Fraud detection.
• Power plant load prediction. (HTI, UCY)
• Potential mineral exploration sites. p
In Optimization
• In product distribution (traveling salesman problems).
I d i i ki• In decision‐making.
In Synthesis
• Synthesis of music and speech.
S h i f i l• Synthesis of new materials.59
In Data clustering and compression
In Telecommunications, TelemedicineTelemedicine, Geology, Meteorology
60
Possible research areas in ANNs
Basic Researchl h dNew learning methods
New architecturesGenetic algorithm learningGenetic algorithm learningNew single‐neuron models
MedicineDisease diagnosis
Engineeringl d fPower load forecasting
Meteorological applicationsRobotic and other complicated system control
61
Robotic and other complicated system controlShip resistance calculations
Dynamical systemsConflict resolution, conflict avoidance (Fuzzy Cognitive
Maps ?)Dynamic analysis (Fuzzy Cognitive Maps ?)D i i ki d i i th C F d l St tDecision‐making dynamics in the Cyprus Federal State
(Fuzzy Cognitive Maps ?)
MiscellaneousMiscellaneousEarthquake prognosisAthletic capability prognosisAthletic capability prognosis
62
ANN and Statistical methods
•There exists some conflict among neuroscientists and statisticians on who invented first a particularand statisticians on who invented first a particular methodology !Many of the non‐dynamical models used in ANNs haveMany of the non dynamical models used in ANNs have their roots in known statistical methods such as:
Generalized linear models
P l i l iPolynomial regression
Non‐parametric regression
Discriminant analysis
Principal components
63
Cluster analysis
Comparison of terminology
•STATISTICS ANN•••Variables Features•Independent variables Inputs•Predicted values Outputs
d i bl i i l•Dependent variables Targets or training values•Residuals Errors•Estimation Training, learning, adaptation, or self‐organization.•Estimation criterion Error function, Cost function, or Lyapunov function•Observations Patterns or training pairs•Parameter estimates (Synaptic) weights•Interactions Higher‐order neurons•Transformations Functional links•Regression and discriminant analysis Supervised learning or heteroassociation•Data reduction Unsupervised learning, encoding, or autoassociation•Cluster analysis Competitive learning or adaptive vector q anti ationquantization•Interpolation and extrapolation Generalization
64
Evolutionary Systems – ESEvolutionary Systems – ES
Lately has been given great momentumLately has been given great momentum.
They ar usually known as Genetic Algorithms (GA)
Th l ith f ti i ti d hThey are algorithms for optimization and search.
Initially were studied by John Holland of the University of Michigan back in the 70’s.University of Michigan back in the 70 s.
They resemble the natural evolution65
They resemble the natural evolution.
They may be used in:
Function optimization.
Operations research.
f l l kLearning in artificial neural networks .
Evolution of new artificial neural network architectures.
Evolution of fuzzy rules.
…
They are relatively simple to apply, especially in complicated y y p pp y, p y poptimization problems.
They can find global optimization solutions
66
They can find global optimization solutions.
Some more modern terms that are being used are:
Evolutionary Computation or Evolutionary Algorithms.
These include:
Genetic Algorithms ‐ GA
Genetic Programming GPGenetic Programming ‐ GP
Evolutionary Programming ‐ EP
Evolutionary hardware ‐ EHW
Evolutionary Strategies ‐ ESEvolutionary Strategies ES
Learning Classifier Systems ‐ LCS
67
Artificial Life Systems – ALSArtificial Life Systems ALS
It is a highly simplified simulation of some dynamical features of living organismsliving organisms .
They have a lot with the well established science of cellular t tautomata.
They use simple rules for self‐organization.
e.g Conway’s “Game of Life”
It is common to experience Complexity and Chaotic problems.
68
They can show an emerging behaviour.
Can be used in:
Optimization problemsOptimization problems.
Neural network learningNeural network learning.
. . .
Modern Particle Swarm Systems is an extension of ALS.
69
Fuzzy Systems (FS)Fuzzy Systems (FS)
The fuzzy logic is a superset of the bi‐state Boolean l ilogic.
It has been devised in order to tackle the so calledIt has been devised in order to tackle the so called “half‐truths” and the paradoxes, as well as our need to have partly truth valuesto have partly truth values.
70
Some paradoxes:
“I am a liar”! (Epimenides)
Am I saying the truth?
“The barber who shaves all those who don’t shave themselves” (Bertrand Russel)themselves” (Bertrand Russel)
Wh h h b b ?Who shaves the barber?
71
Fuzzy SystemsFuzzy SystemsExample: When is someone tall?
If P is the set of all humans and h their height, a fuzzy subset μ(h) that could answer the above question is:q
0 if 1,5 m ( 1,5)
if 1,5 2,2 m 0,7
1 if 2,2 m
hh
h
h
(h)μ→ <
−→ ≤ ≤
→ >
=
⎧⎪⎨⎪⎩
Where μ(h) is the established membership function
72
Difference between fuzzy and conventional:
In binary logic, the result can be true or false:
e g 1+1=2e.g. 1+1=2
The world is mostly fuzzy.
e.g. Its cold,
It is greenIt is green, ...
There are vagueness because of “probable events” and b i i diffi l fi d di idi li dbecause it is difficult to find dividing lines among events and objects.
73
Fuzzy SystemsMembership functions
1
μ(h)
0.5
00.5 Height, h [m]2.21.85
S h f i b d f hSuch functions may be constructed for other parameters, and generate a fuzzy logic system with rules:
74
Fuzzy Systems
Some membership functions from the MATLAB library:
1trapmf gbellmf trimf gaussmf gauss2mf smf
0 2
0.4
0.6
0.8
0
0.2
0.6
0.8
1zmf psigmf dsigmf pimf sigmf
0
0.2
0.4
75
Fuzzy Systemsy yAND fuzzy rule:
AND
76
Most applications are in the fields of Automatic Control, in Information Systems and in supporting , y pp gDecision‐making.
77
Fuzzy Cognitive Maps ‐ FCMy g p
It is a kind of combination between ANNs and fuzzy logic.
They are made of a set of junctions that represent concepts and from arrows that connect these
Can be used in complicated problems in order to simulate the dynamical interrelationships as in Decision Making and in
junctions.
dynamical interrelationships as in Decision Making and in Strategic Planning.
78Essentially are recurrent diagrams that use fuzzy values.
Concepts that have been considered
An attempt to study interrelations in the Cyprus Problem
Concepts that have been considered30 influencing parameters (concepts) have been explored. These are:
INFLUENCING PARAMETERS (Concepts)C1 Welfare of the Federal State of Cyprus
These are:
C1 Welfare of the Federal State of CyprusC2 Welfare of the Greek Cypriot StateC3 Welfare of the Turkish Cypriot StateypC4 Greek Cypriot nationalismC5 Christian religiousnessC6 Knowledge of Turkish language by the Greek CypriotsC7 Knowledge of Turkish history by the Greek CypriotsC8 Educational level of the Greek CypriotsC9 Turkish Cypriot nationalism
79
C10 Islamic religiousness
Concepts that have been considered
C11 Knowledge of Greek language by the Turkish CypriotsC12 Knowledge of Greek history by the Turkish CypriotsC13 Educational level of the Turkish CypriotsC14 Political interests of EuropeC15 Political interests of USAC16 Political interests of RussiaC1 li i l i fC17 Political interests of UKC18 Political interests of IsraelC19 Political interests of GreeceC19 Political interests of GreeceC20 Political interests of Turkey
80
Concepts that have been considered
C21 Military interests of IsraelC22 Military interests of GreeceC23 Military interests of TurkeyC24 Military interests of UKC25 Mili i f USAC25 Military interests of USAC26 Interests of Anatolian settlersC27 Level of tourism in the federated stateC27 Level of tourism in the federated stateC28 Oil fieldsC29 Quality of environmentC29 Quality of environmentC30 Other natural resources
81
Effects of changes in Islamic religiousness European and USA interests300%Effects of changes in Islamic religiousness, European and USA interests
250%
300%
Percentage change in parameters due to major increase in Islamic Religiousness
200%
g g p j g
Percentage change in parameters due to major increase in European interest
P h i d j i i USA i
150%
Percentage change in parameters due to major increase in USA interest
50%
100%
0%
WelfWelfWelfGC NChrisKnowKnowEducTC NIslamKnowKnowEducPolitiPolitiPolitiPolitiPolitiPolitiPoliti Milit Milit Milit Milit MilitInterLevelOil fiQualiOthe
-50%
lfare of the Federal State
lfare of the GC State
lfare of the TC State
Nationalismristian religiousness
owledge of Turkish langu
owledge of Turkish histo
ucational level of the GC
Nationalismamic religiousness
owledge of Greek langua
owledge of Greek history
ucational level of the TC
itical interests of Europe
itical interests of USA
itical interests of Russia
itical interests of UK
itical interests of Israel
itical interests of Greece
itical interests of Turkey
litary interests of Israel
litary interests of Greece
litary interests of Turkey
litary interests of UK
litary interests of USA
erests of anatolian settlers
vel of tourism in the Fede
fieldsality of environment
her natural resources
82
te of Cyprus
guage by the GC
tory by the GC
C uage by the TC
ry by the TCC pe a e y ce ey ers derated state
Case of increase in the Islamic religiousnessThis results in significant increase in the Greek Cypriot nationalism (+36%) and most importantly a very high increase in Christian religiousness (+283%).
Al it ill lt i i i th ilit i t t fAlso, it will result in an increase in the military interests of Turkey, Greece and Israel.
Of significance is the observation that the welfare of the Federal State of Cyprus will reduce by 7% and the welfare of yp ythe Turkish Cypriot community by 4%.
Some reduction in the level of tourism and the quality of the environment is also observed.
83
Case of increase in the European political interests in Cyprus
The model predicts a significant increase of the knowledge of Greek language and history by the Turkish Cypriots and vice versa.
Thi i d i th f k l d f thThis is more pronounced in the case of knowledge of the Turkish language and history by the Greek Cypriots.
It is however very interesting that the model suggests a significant reduction in Christian religiousness (‐28%) and g g ( )somewhat less (‐5%) in the Islamic religiousness.
Most interesting though, is the suggestion that the interests of the Anatolian settlers will reduce by 20%.
84
Case of increase in the political interests of USA in Cyprus
The model predicts a 12% increase in the political interests of Israel and a 12% increase in the welfare of the Turkish Cypriots.
What is most interesting though, and somewhat unusual, is the d ti f th Ch i ti li i b 16% f th T ki hreduction of the Christian religiousness by 16%, of the Turkish
Cypriot nationalism by 18%, of the Islamic religiousness by 8%, of the political interests of Greece and Turkey by 11% and 16%of the political interests of Greece and Turkey by 11% and 16% respectively, and of the military interests of UK by 18%.
85