Computational Intelligence
John Sum
Institute of Technology Management
National Chung Hsing University
Taichung, Taiwan ROC
John Sum Computational Intelligence 2
OUTLINE
Historical Background Computational Intelligence Example Problems Methodology
Model Structure Model Parameters Parametric Estimation
Discussion Conclusion
John Sum Computational Intelligence 3
HISTORY
John Sum Computational Intelligence 4
HISTORY
1940 – First computing machine 1957 – Perceptron (First NN model) 1965 – Fuzzy Logic (Rules) 1960s – Genetic Algorithm 1970s – Evolutionary Computing
John Sum Computational Intelligence 5
HISTORY
1980s Neural Computing Swarm Intelligence
1990s (Hybrid) Fuzzy Neural Networks NFG, FGN, GNF, etc
John Sum Computational Intelligence 6
HISTORY
Beyond 1990s: Research areas converge Computational Intelligence Softcomputing Intelligent Systems
Covering Adaptive Systems Fuzzy Systems Neural Networks Evolutionary Computing Data Mining
John Sum Computational Intelligence 7
COMPUTATIONAL INTELLIGENCE Computational Intelligence
Heuristic algorithms (or models) such as in fuzzy systems, neural networks and evolutionary computation.
Techniques that use Simulated annealing, Swarm intelligence, Fractals and Chaos Theory, Artificial immune systems, Wavelets, etc.
John Sum Computational Intelligence 8
COMPUTATIONAL INTELLIGENCE Goal: Problem Solving
Financial forecast Customer segmentation (CRM) Supply chain design (SCM) Business process re-engineering System control Pattern recognition Image compression Homeland security
John Sum Computational Intelligence 9
COMPUTATIONAL INTELLIGENCE Underlying structure of the model is unknown, or the
model is known but it is too complicated Example: DJI versus HIS (Time Series)
Define system structure NL model (NN, ODE, etc.) Rule-based system
Parametric estimation Deterministic search (Gradient descent or Newton’s
method) Stochastic search (SA or MCMC)
John Sum Computational Intelligence 10
COMPUTATIONAL INTELLIGENCE Underlying model structure is known Example: Manufacturing process (SCM)
Define the objective to be maximized Examples: Completion time, Cost, Profit
Optimization Linear programming, ILP, NLP Deterministic search (Gradient descent or Newton’s
method) Stochastic search (SA or MCMC)
John Sum Computational Intelligence 11
EG1: Nonlinear Dynamic System
x )(xg
Noise
y
system
Unknown
John Sum Computational Intelligence 12
EG2: Nonlinear Function
x )(xg
Noise
y
system
Unknown
John Sum Computational Intelligence 13
EG3: Car Price
Predict the price of a car based on Specification of an auto in terms of various
characteristics Assigned insurance risk rating Normalized losses in use as compared to other
cars
Number of attributes: 25 Missing values: Yes!
John Sum Computational Intelligence 14
EG3: Car Price
John Sum Computational Intelligence 15
EG4: Purchasing Preference
John Sum Computational Intelligence 16
EG5: Financial Time Series
7000
8000
9000
10000
11000
12000
13000
14000
1 159 317 475 633 791 949 11071265142315811739
John Sum Computational Intelligence 17
EG5: Financial Time Series
What would happen in the next trading day? (Time series prediction problem) Closing value Open value UP or DOWN
Time series prediction + trading rules What should I do tomorrow? HOLD, SELL or BUY When should I BUY and SELL?
John Sum Computational Intelligence 18
System Structure Data Types Model
Dynamic System
Unknown Continuous RNN, Fuzzy NN
Nonlinear Function
Unknown Continuous BPN, RBF, Fuzzy NN
Car Price UnknownContinuous
DiscreteBPN, RBF, Fuzzy NN
Purchasing Preference
Known (SEM)
DiscreteSEM
Bayesian Net
Financial Time Series
Unknown Continuous BPN, RBF, Fuzzy NN
Remarks on EG1 ~ EG5
John Sum Computational Intelligence 19
COMPUTATIONAL INTELLIGENCE
Statement of Problem Given a set of data collected (or measured) from a system (probably an unknown system), devise a model (by whatever structure, technique, method in CI) that mimics the behavior of that system as ‘good’ as possible.
Making use of the devised model to (1) interpret the behavior of the system, (2) predict the future behavior of the system, (3) control the behavior of the system, (4) make money.
John Sum Computational Intelligence 20
METHODOLOGY
Step 1: Data Collection Experiments or measurements Questionnaire Magazine Public data sets
Step 2: Model Structure Assumption IF it is known, SKIP this step. ELSE, DEFINE a model structure
John Sum Computational Intelligence 21
METHODOLOGY
Step 3: Parametric Estimation Gradient descent Newton’s method Exhaustive search Genetic algorithms (*) Evolutionary algorithms (*) Swarm intelligence Simulated annealing (*) Markov Chain Monte Carlo (*)
John Sum Computational Intelligence 22
METHODOLOGY
Step 4: Model Validation (is it a reasonable good model) Hypothesis test Validation/Testing set Leave one out validation
Step 5: Model Reduction (would there be a simpler model that is also reasonable good) AIC, BIC, MDL Pruning (using testing set)
John Sum Computational Intelligence 23
METHODOLOGY
Beyond Model Reduction Any redundant input Any redundant sample (or outlier) Any better structure (alternative) How do we determine a ‘good’ model
John Sum Computational Intelligence 24
NN MODEL STRUCTURES
Perceptron Multilayer Perceptron (MLP or BPN) Adaptive Resonance Theory Model (ART) Competitive Learning (CL) Hopfield Network, Associative Network Bidirectional Associative Model (BAM) Recurrent Neural Network (RNN) Boltzmann Machine Brain-State-In-A-Box (BSB) Radial Basis Function Network (RBF Net) Bayesian Networks Self Organizing Map (SOM or Kohonen Map) Learning Vector Quantization (LVQ) Support Vector Machine (SVM) Support Vector Regression (SVR) PCA, ICA, MCA Winner-Take-All Network (WTA) Spike neural networks
Remarks Not all of them is able to learn,
eg BSB, WTA Might need to combine two str
uctures to solve a single problem
Multiple definitions on the ‘neuron’
John Sum Computational Intelligence 25
NN MODEL STRUCTURES
Supply Chain Management (Optimization Problem) Hopfield Network
Customer Segmentation (Clustering Problem) CL, SOM, LVQ, ART
Dynamic Systems Modeling RNN, Recurrent RBF
Car Price/NL Function (Function Approximation) MLP, RBF Net, Bayesian Net, SVR, +SOM/LVQ
Financial TS (FA or Time Series Prediction) RNN, SVR, MLP, RBF Net, + SOM/LVQ
John Sum Computational Intelligence 26
FUZZY MODEL STRUCTURE
John Sum Computational Intelligence 27
FUZZY MODEL STRUCTURE
John Sum Computational Intelligence 28
NN MODEL PARAMETERS MLP
Input Weights Output Weights Neuron model
RNN Input Weights Output Weights Recurrent Weights Neuron model
John Sum Computational Intelligence 29
NN MODEL PARAMETERS
John Sum Computational Intelligence 30
NN MODEL PARAMETERS
John Sum Computational Intelligence 31
NN MODEL PARAMETERS
John Sum Computational Intelligence 32
FUZZY MODEL PARAMETERS
John Sum Computational Intelligence 33
PARAMETRIC ESTIMATION
John Sum Computational Intelligence 34
PARAMETRIC ESTIMATION Gradient Descent
John Sum Computational Intelligence 35
PARAMERTIC ESTIMATIONGenetic Algorithm
John Sum Computational Intelligence 36
PARAMERTIC ESTIMATIONGenetic Algorithm
John Sum Computational Intelligence 37
PARAMERTIC ESTIMATIONGenetic Algorithm
John Sum Computational Intelligence 38
DISCUSSIONS
CI is not the only method (or structure) to solve a problem.
Even it can solve, its performance might not be better than other methods.
Should compare with other well-known or existing methods
John Sum Computational Intelligence 39
DISCUSSIONS
SCM Problem LP, LIP, NLP Lagrangian Relaxation Cutting Plane CPLEX
Function Approximation Polynomial Series Trigonometric Series B-Spline
John Sum Computational Intelligence 40
CONCLUSIONS
IF The problem to be solved has been well
formulated The structure has been selected The objective function to evaluation the goodness
of a parametric vector has been defined
THEN Every problem is just an optimization problem
John Sum Computational Intelligence 41
JOHN SUM ([email protected]) Taiwan HK-Chinese, PhD (98) and MPhil (95) from CUHK, BEng (92)
from PolyU HK. Taught in HK Baptist University (98-00), OUHK (00) and PolyU HK (0
0-04), Chung Shan Medical University (05-07) Adj. Associate Prof., Institute of Software, CAS Beijing (99-02) Short visit: CityU HK, Griffith University in Australia, FAU, Boca Rato
n FL US, CAS in Beijing, Ching Mai University in Thailand. Assist. Prof., IEC (07-09), Asso. Prof., ITM (09-) NCHU Taiwan 2000 Marquis Who's Who in the World. Senior Member of IEEE, CI Society, SMC Society (05-) GB Member, Asia Pacific Neural Network Assembly (09-) Associate Editor of the IJCA (05-09) Research Interests include NN, FS, SEM, EC, TM