analysis of nature-inspried optimization algorithms

Analysis of Nature-Inspired Optimization Algorithms

Xin-She Yang

School of Science and TechnologyMiddlesex University

Seminar at Department of Mathematical SciencesUniversity of Essex

20 Feb 2014

Xin-She Yang (Middlesex University) Algorithms 20 Feb 2014 1 / 48

Overview Overview

Overview

Introduction

What is an Algorithm?

Nature-Inspired Optimization Algorithms

Applications in Engineering Design

Self-Organization and Optimization Algorithms

Conclusions


The Essence of an Algorithm The Essence of an Algorithm

The Essence of an Algorithm

Essence of an Optimization Algorithm

To move to a new, better point x(t+1) from an existing location x(t).

x1

x2

x(t)

x(t+1)

?

Population-based algorithms use multiple, interacting paths.

Different algorithms

Different strategies/approaches in generating these moves!


The Essence of an Algorithm Optimization Algorithms

Optimization Algorithms

Deterministic

Newton’s method (1669, published in 1711), Newton-Raphson(1690), hill-climbing/steepest descent (Cauchy 1847), least-squares(Gauss 1795),

linear programming (Dantzig 1947), conjugate gradient (Lanczos etal. 1952), interior-point method (Karmarkar 1984), etc.


The Essence of an Algorithm Steepest Descent/Hill Climbing

Steepest Descent/Hill Climbing

Gradient-Based Methods

Use gradient/derivative information – very efficient for local search.



Newton’s Method

xt+1 = xt −H−1∇f , H =

∂2f∂x1

2 · · · ∂2f∂x1∂xd

.... . .

...∂2f

∂xd∂x1· · · ∂2f

∂xd2

.

Quasi-Newton

If H is replaced by I, we have

xt+1 = xt − αI∇f (xt).

Here α controls the step length.

Generation of new moves by gradient.




In essence, an algorithm can be written (mathematically) as

xt+1 = A(xt , α),

For any given xt , the algorithm will generate a new solution xt+1.

Functional or a Dynamical System?

We can view the above equation as

a functional,

or a dynamical system,

or a Markov chain,

or a self-organizing system.

The behavoir of the system (algorithm) can be controlled by A and itsparameter α.


Nature-Inspired Algorithms Stochastic/Metaheuristic Algorithms

Stochastic/Metaheuristic Algorithms

Genetic algorithms (1960s/1970s), evolutionary strategy (Rechenberg& Swefel 1960s), evolutionary programming (Fogel et al. 1960s).

Simulated annealing (Kirkpatrick et al. 1983), Tabu search (Glover1980s), ant colony optimization (Dorigo 1992), genetic programming(Koza 1992), particle swarm optimization (Kennedy & Eberhart1995), differential evolution (Storn & Price 1996/1997), harmonysearch (Geem et al. 2001), honeybee algorithm (Nakrani & Tovey2004), artificial bee colony (Karaboga, 2005).

Firefly algorithm (Yang 2008), cuckoo search (Yang & Deb 2009), batalgorithm (Yang, 2010), ...


Simulated Annealling Simulated Annealling

Simulated Annealling

Metal annealing to increase strength =⇒ simulated annealing.

Probabilistic Move: p ∝ exp[−E/kBT ].

kB=Boltzmann constant (e.g., kB = 1), T=temperature, E=energy.

E ∝ f (x),T = T0αt (cooling schedule) , (0 < α < 1).

T → 0 =⇒ p → 0, =⇒ hill climbing.This is equivalent to a random walk

xt+1 = xt + p(xt , α).


Simulated Annealling An Example

An Example


Simulated Annealling Genetic Algorithms

Genetic Algorithms

Not easy to write a set of explicit equations!

crossover mutation


Simulated Annealling

Generation of new solutions by crossover, mutation and elistism.


Swarm Intelligence

Swarm Intelligence

Ants, bees, birds, fish ... Simple rules lead to complex behaviour.

Swarming Starlings


Swarm Intelligence PSO

PSO

Particle Swarm Optimization (Kennedy and Eberhart, 1995)

xi

g∗

xj

vt+1i = vt

i + αǫ1(g∗ − xt

i ) + βǫ2(x∗

i − xti ),

xt+1i = xt

i + vt+1i .

α, β = learning parameters, ǫ1, ǫ2=random numbers.


Swarm Intelligence PSO Demo and Disadvantages

PSO Demo and Disadvantages

xi

vi

t+1

=

1 1

−(αǫ1 + βǫ2) 1

xi

vi

t

+

0

αǫ1g∗ + βǫ2x

∗

i

.

PSO Demo Premature convergence


Swarm Intelligence Firefly Algorithm (Yang, 2008)

Firefly Algorithm (Yang, 2008)

Firefly Behaviour and Idealization

Fireflies are unisex and brightness varies with distance.

Less bright ones will be attracted to bright ones.

If no brighter firefly can be seen, a firefly will move randomly.

xt+1i = xt

i + β0e−γr2

ij (xj − xi ) + α ǫti .

Generation of new solutions by random walk and attraction.


Swarm Intelligence What is FA so efficient?

What is FA so efficient?

Advantages of Firefly Algorithm

Automatically subdivide the whole population into subgroups, andeach subgroup swarms around a local mode/optimum.

Control modes/ranges by varying γ.

Control randomization by tuning parameters such as α.

Suitable for multimodal, nonlinear, global optimization problems.


Swarm Intelligence Variants of Firefly Algorithm

Variants of Firefly Algorithm

About 1400 papers about firefly algorithm since 2008. Its literature isdramatically expanding.

Variants for specific applications:

Continuous optimization, mixed integer programming, ...

Discrete firefly algorithm for scheduling, travelling-salesman problem,combinatorial optimization ...

Image processing and compression ...

Clustering, classifications and feature selection ...

Chaotic firefly algorithm ...

Hybrid firefly algorithm with other algorithms ...

Multiobjective firefly algorithm.


Swarm Intelligence Cuckoo Breeding Behaviour

Cuckoo Breeding Behaviour

Evolutionary Advantages

Dumps eggs in the nests of host birds and let these host birds raise theirchicks.

Cuckoo Video (BBC)


Swarm Intelligence Cuckoo Search (Yang and Deb, 2009)

Cuckoo Search (Yang and Deb, 2009)

Cuckoo Behaviour and IdealizationEach cuckoo lays one egg (solution) at a time, and dumps its egg in arandomly chosen nest.

The best nests with high-quality eggs (solutions) will carry out to thenext generation.

The egg laid by a cuckoo can be discovered by the host bird with aprobability pa and a nest will then be built.


Swarm Intelligence Cuckoo Search

Cuckoo Search

Local random walk:

xt+1i = xt

i + s ⊗ H(pa − ǫ)⊗ (xtj − xt

k).

[xi , xj , xk are 3 different solutions, H(u) is a Heaviside function, ǫ is arandom number drawn from a uniform distribution, and s is the step size.

Global random walk via Levy flights:

xt+1i = xt

i + αL(s, λ), L(s, λ) =λΓ(λ) sin(πλ/2)

π

1

s1+λ, (s ≫ s0).

Generation of new moves by Levy flights, random walk and elitism.


Swarm Intelligence CS Demo: Highly Efficient!

CS Demo: Highly Efficient!

About 1000 papers about cuckoo search since 2009. Its literature isdramatically expanding.[See X. S. Yang, Cuckoo Search and Firefly Algorithm: Theory and Applications,

Springer, (2013).]

CS Demo Efficient search with a focus


Swarm Intelligence Bat Algorithm (Yang, 2010)

Bat Algorithm (Yang, 2010)

BBC Video

Microbats use echolocation for hunting

Ultrasonic short pulses as loud as 110dB with a short period of 5 to20 ms. Frequencies of 25 kHz to 100 kHz.

Speed up pulse-emission rate and increase loudness when homing at aprey.


Swarm Intelligence Bat Algorithm

Bat Algorithm

Acoustics of bat echolocation

λ =v

f∼ 2 mm to 14 mm.

Rules used in the bat algorithm:

fi = fmin + (fmax − fmin)β, β ∈ [0, 1],

vt+1i = v t

i + (xti − x∗)fi , xt+1

i = xti + vt

i .

Variations of Loudness and Pulse Rate

At+1i ← αAt

i , α ∈ (0, 1], r t+1i = r0

i [1− exp(−γt)].

There are about 240 papers since 2010.


Swarm Intelligence Other Algorithms

Other Algorithms

Genetic algorithms

Differential evolution

Artificial immune system

Harmony search

Memetic algorithm

...

Reviews

Yang, X. S., Engineering Optimization: An Introduction with

Metaheuristic Applications, John Wiley & Sons, (2010).

Yang, X. S., Nature-Inspired Metaheuristic Algorithms, Luniver Press,(2008).


Applications Applications

Applications

Design optimization: structural engineering, product design ...

Scheduling, routing and planning: often discrete, combinatorialproblems ...

Applications in almost all areas (e.g., finance, economics, engineering,industry, ...)


Applications Pressure Vessel Design Optimization

Pressure Vessel Design Optimization

r

d1

r

L d2


Applications Pressure Vessel Design

Pressure Vessel Design

This is a well-known test problem for optimization (e.g., see Cagnina et al.

2008) and it can be written as

minimize f (x) = 0.6224d1rL + 1.7781d2r2 + 3.1661d2

1 L + 19.84d21 r ,

subject to

g1(x) = −d1 + 0.0193r ≤ 0g2(x) = −d2 + 0.00954r ≤ 0g3(x) = −πr2L− 4π

3 r3 + 1296000 ≤ 0g4(x) = L− 240 ≤ 0.

The simple bounds are

0.0625 ≤ d1, d2 ≤ 99× 0.0625, 10.0 ≤ r , L ≤ 200.0.

The best solution (Yang, 2010; Gandomi and Yang, 2011)

f∗ = 6059.714, x∗ = (0.8125, 0.4375, 42.0984, 176.6366).


Applications Speed Reducer/Gear Box Design

Speed Reducer/Gear Box Design

Mixed-Integer Programming:

Continuous variables and integers.


Applications

f (x1, x2, x3, x4, x5, x6, x7) = 0.7854x1x22 (3.3333x2

3 + 14.9334x3 − 43.0934)

−1.508x1(x26 + x2

7 ) + 7.4777(x36 + x3

7 ) + 0.7854(x4x26 + x5x

27 ),

subject to

g1 = 27x1x2

2 x3− 1 ≤ 0, g2 = 397.5

x1x22 x2

3− 1 ≤ 0,

g3 =1.93x3

4

x2x3d41− 1 ≤ 0, g4 =

1.93x35

x2x3d42− 1 ≤ 0,

g5 = 1110x3

6

√

(745x4hx3

)2 + 16.9 × 106 − 1 ≤ 0,

g6 = 185x3

7

√

(745x5hx3

)2 + 157.5 × 106 − 1 ≤ 0,

g7 = x2x340 − 1 ≤ 0, g8 = 5x2

x1− 1 ≤ 0,

g9 = x112x2− 1 ≤ 0, g10 = 1.5x6+1.9

x4− 1 ≤ 0,

g11 = 1.1x7+1.9x5

− 1 ≤ 0.

Simple bounds are 2.6 ≤ x1 ≤ 3.6, 0.7 ≤ h ≤ 0.8, 17 ≤ x3 ≤ 28,7.3 ≤ x4 ≤ 8.3, 7.8 ≤ x5 ≤ 8.3, 2.9 ≤ x6 ≤ 3.9, and 5.0 ≤ x7 ≤ 5.5. z

must be integers.


Applications Dome Design

Dome Design

120-bar dome: Divided into 7 groups, 120 design elements, about 200 constraints

(Gandomi and Yang 2011; Yang et al. 2012).


Applications Tower Design

Tower Design

26-storey tower: 942 design elements, 244 nodal links, 59 groups/types, > 4000

nonlinear constraints (Yang et al. 2011; Gandomi & Yang 2012).


Applications Car Design

Car Design

Design better, safer and energy-efficient cars

Minimize weight, low crash deflection (< 32 mm), lower impact force (< 4kN), .... Even a side barrier has 11 design variables!




In essence, an algorithm can be written (mathematically) as

xt+1 = A(xt , α),

For any given xt , the algorithm will generate a new solution xt+1.

Functional or a Dynamical System?

We can view the above equation as

a functional,

or a dynamical system,

or a Markov chain,

or a self-organizing system.

The behavoir of the system (algorithm) can be controlled by A and itsparameter α.


The Essence of an Algorithm Self-Organization

Self-Organization

Self-organizing systems are everywhere,physical, chemical, biological, social, artificial ...


The Essence of an Algorithm Conditions for Self-Organization

Conditions for Self-Organization

Complex Systems:

Large size, a sufficient number of degrees of freedom,a huge number of (possible) states S .

Allow the systems to evolve for a long time.

Enough diversity (perturbation, noise, edge of chaos,far-from-equilibrium).

Selection (unchanging laws etc).

That is

S(θ)α(t)−→ S(π).

S + α forms a larger system, so S is self-organizing, driven by α(t).


The Essence of an Algorithm Optimization Process

Optimization Process

Optimization is a process to find an optimal solution (state), from allpossible states of a problem objective g , carried out by an algorithm A anddriven by tuning some algorithm-dependent parameters p(t) = (p1, ..., pk ).

An Algorithm = Iterative Procedure

xt+1 = A(xt ,p(t)).

Optimization

g(xt=0)A(t)−→ g(x∗).

An algorithm has to use information of problem states during iterations(e.g., gradient information, selection of the fittest, best solutions etc).


The Essence of an Algorithm Similarities Between Self-Organization and Optimization

Similarities Between Self-Organization and Optimization

Self-organization:

Noise, perturbation =⇒ diversity.

Selection mechanism =⇒ structure.

Far-from-equilibrium and large perturbation =⇒ potentially faster tore-organize.

Optimization Algorithms:

Randomization, stochastic components, exploration=⇒ to escape local optima.

Selection and exploitation =⇒ convergence & optimal solutions.

High-degrees of randomization=⇒ more likely to reach global optimality (but may be slow).


The Essence of an Algorithm But there are significant differences!

But there are significant differences!

Self-Organization:

Avenues to self-organization may be unclear.

Time may not be important.

Optimization (especially metaheuristics):

How to make an algorithm converge is very important.

Speed of convergence is crucial(to reach truly global optimality with the minimum computingefforts).

However, we lack good theories to understand either self-organization ormetaheuristic optimization.


The Essence of an Algorithm Multi-Agent System (Swarm Intelligence?)

Multi-Agent System (Swarm Intelligence?)

For a multi-agent system or a swarm, an algorithm can be considered as aset of interacting Markov chain or a complex dynamical system

x1

x2...xn

t+1

= A[x1, ..., xn; ǫ1, ..., ǫm; p1(t), ..., pk(t)]

x1

x2...xn

t

.

A population of solutions xt+1i (i = 1, ..., n) are generated from xt

i ,controlled by k parameters and m random numbers.

In principle, the behaviour of an algorithm is controlled by the eigenvaluesof A, but in practice, it is almost impossible to figure out the eigenvalues(apart from very simple/rare cases).


Some Interesting Results Genetic Algorithms

Genetic Algorithms

For a binary genetic algorithm with p0 = 0.5 with n chromosomes of lengthm, the probability of (premature) convergence at any time/iteration t is

P(t,m) =[

1−6p0(1 − p0)

n

(

1−2

n

)t]m.

For a population of n = 40, m = 100, t = 100 generations, we have

P(t,m) =[

− 16× 0.5(1 − 0.5)

40(1−

2

40)100

]100≈ 0.978.

For genetic algorithms with a given accuracy ζ, the number of iterationst(ζ) needed is (possibly converged prematurely)

t(ζ) ≤[ ln(1− ζ)

ln{

1−min[(1− µ)nL, µnL]}

]

,

where µ =mutation rate, L =string length, and n =population size.


Some Interesting Results Cuckoo Search with Guaranteed Global Convergence

Cuckoo Search with Guaranteed Global Convergence

Simplify the cuckoo search (CS) algorithm as

xt+1i ← xt

i if r < pa

xt+1i ← xt

i + α⊗ L(λ) if r > pa.

Markov Chain Theory

The probability of convergence to the global optimal set G is

limt→∞

P(xt ∈ G )→ 1.

See Wang et al. (2012) for details.


Exploration & Exploitation Key Components in All Metaheuristics

Key Components in All Metaheuristics

So many algorithms – what are the common characteristics?

What are the key components?

How to use and balance different components?

What controls the overall behaviour of an algorithm?


Exploration & Exploitation Exploration and Exploitation

Exploration and Exploitation

Characteristics of Metaheuristics

Exploration and Exploitation, or Diversification and Intensification.

Exploitation/Intensification

Intensive local search, exploiting local information.E.g., hill-climbing.

Exploration/Diversification

Exploratory global search, using randomization/stochastic components.E.g., hill-climbing with random restart.


Exploration & Exploitation Summary

Summary

Exploitation

Exp

lora

tion

uniformsearch

steepestdescent

Tabu Nelder-Mead

CS

PSO/FAEP/ESSA Ant/Bee

Genetic algorithms

Newton-Raphson

Best?

Free lunch?


Open Problems Open Problems

Open Problems

Mathematical Analysis: A unified mathematical framework (e.g.,Markov chain theory, dynamical systems) is needed.

Comparison: What are the best performance measures?

Parameter tuning: How to tune the algorithm-dependentparameters so that an algorithm can achieve the best performance?

Scalability: Can the algorithms that work for small-scale problems bedirectly applied to large-scale problems?

Intelligence: Smart algorithms may be a buzz word, but can trulyintelligent algorithms be developed?


Open Problems Bibliography

Bibliography

Xin-She Yang, Nature-Inspired Optimization Algorithms, Elsevier, (2014).

Xin-She Yang, Cuckoo Search and Firefly Algorithm: Theory and Applications, Springer,(2013).

Xin-She Yang and Suash Deb, Multiobjective cuckoo search for design optimization,Computers & Operations Research, 40(6), 1616–1624 (2013).

A. H. Gandomi, X. S. Yang, S. Talatahari, S. Deb, Coupled eagle strategy and differentialevolution for unconstrained and constrained global optimization, Computers &Mathematics with Applications, 63(1), 191–200 (2012).

A. H. Gandomi, G. J. Yun, X. S. Yang, S. Talatahari, Chaos-enhanced accelerated particleswarm optimization, Communications in Nonlinear Science and Numerical Simulation,18(2), 327–340 (2013).

X. S. Yang and S. Deb, Two-stage eagle strategy with differential evolution, Int. Journalof Bio-Inspired Computation, 4(1), 1–5 (2012).

X. S. Yang, Multiobjective firefly algorithm for continuous optimization, Engineering withComputers, 29(2), 175–184 (2013).


Open Problems Thank you

Thank you

Xin-She Yang, Z. H. Cui, R. B. Xiao, A. H. Gandomi, M. Karamanoglu, SwarmIntelligence and Bio-Inspired Computation: Theory and Applications, Elsevier, (2013).

Xin-She Yang, A. H. Gandomi, S. Talatahari, A. H. Alavi, Metaheuristics in Water,Geotechnical and Transport Engineering, Elsevier, (2012).

A. H. Gandomi, Xin-She Yang, A. H. Alavi, Mixed variable structural optimization usingfirefly algorithm, Computers & Structures, vol. 89, no. 23, 2325–2336 (2011).

Xin-She Yang and A. H. Gandomi, Bat algorithm: a novel approach for global engineeringoptimization, Engineering Computations, vol. 29, no. 5, 464–483 (2012).

Xin-She Yang, A new metaheuristic bat-inspired algorithm, in: Nature InspiredCooperative Strategies for Optimization (NISCO 2010) (Eds. J. R. Gonzalez et al.),Studies in Computational Intelligence (SCI 284), Springer, pp. 65–74 (2010).

Xin-She Yang, Artificial Intelligence, Evolutionary Computing and Metaheuristics — In theFootsteps of Alan Turing, Studies in Computational Intelligence (SCI 427), SpringerHeidelberg, (2013).


analysis of nature-inspried optimization algorithms

Technology

bat algorithm yang

algorithm essence

given xt

system algorithm

xt h1 f

yang school of science

new solution xt

random walk xt