alidn i lapplied numerical analysis -...
TRANSCRIPT
A li d N i lA li d N i lApplied Numerical Applied Numerical AnalysisAnalysisAnalysisAnalysis
Pseudo Random Number GeneratorPseudo Random Number GeneratorLecturer: Emad FatemizadehLecturer: Emad FatemizadehLecturer: Emad FatemizadehLecturer: Emad Fatemizadeh
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
What is random number:What is random number:What is random number:What is random number:•• A sequence in which each term is unpredictableA sequence in which each term is unpredictable
29, 95, 11, 60, 2229, 95, 11, 60, 22
Application:Application:•• Monte Carlo SimulationsMonte Carlo Simulations•• Generation of Cryptographic KeysGeneration of Cryptographic Keys
Password generationPassword generation
•• Many Combinatorial Optimization AlgorithmsMany Combinatorial Optimization Algorithms•• Many Combinatorial Optimization AlgorithmsMany Combinatorial Optimization Algorithms•• GamesGames
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
RN’s Types:RN’s Types:RN s Types:RN s Types:•• True random:True random:
Generated in nonGenerated in non--deterministic waysdeterministic waysGenerated in nonGenerated in non--deterministic ways.deterministic ways.•• Not predictable Not predictable •• Not repeatable.Not repeatable.
•• Pseudo random:Pseudo random:Numbers that Numbers that appearappear random, but are random, but are bt i d i d t i i ti bt i d i d t i i ti obtained in a deterministic, obtained in a deterministic, •• RepeatableRepeatable•• PredictablePredictable
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
RN’s Generation:RN’s Generation:RN s Generation:RN s Generation:•• True Random:True Random:
Physical Phenomenon:Physical Phenomenon:Physical Phenomenon:Physical Phenomenon:•• Decay times of radioactive materialDecay times of radioactive material•• Electrical noise from a resistor or semiconductorElectrical noise from a resistor or semiconductor•• Radio channel or audible noiseRadio channel or audible noise•• Keyboard timingsKeyboard timings
•• Pseudo Random:Pseudo Random:•• Pseudo Random:Pseudo Random:Mathematical AlgorithmMathematical Algorithm
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Preliminary:Preliminary:Preliminary:Preliminary:•• PDF (Probability Density Function):PDF (Probability Density Function):
( ) ( )Prob X f≤ ≤ +
( )Xf x
( ) ( )
( ) ( )
Prob
0, 1
X
X X
x X x x f x x
f x f x dx+∞
≤ ≤ + =
≥ =∫•• CDF (Cumulative Density Function):CDF (Cumulative Density Function):
( ) ( )−∞∫
( )XF x
( ) ( ) ( )
( ) ( )
Probx
X XX x F x f s ds−∞
−∞ ≤ ≤ = = ∫
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
( ) ( )1, 0X XF F+∞ = −∞ =
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Preliminary:Preliminary:Preliminary:Preliminary:•• Samples:Samples:
Uniform:Uniform:Uniform:Uniform:
( )1
X
a x bf x b a
⎧ ≤ ≤⎪= −⎨
NormalNormal
( )0 O.W.
Xf b a⎨⎪⎩
( ) ( )2
22
1 exp22
X
xf x
μσπσ
⎛ ⎞−= −⎜ ⎟
⎜ ⎟⎝ ⎠
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
⎝ ⎠
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Pseudo Random Number Generator:Pseudo Random Number Generator:Pseudo Random Number Generator:Pseudo Random Number Generator:•• Uniform distributionUniform distribution
Theoretically all others distribution can be be Theoretically all others distribution can be be f ff fgenerate from Uniform.generate from Uniform.
•• Ideal: Between [0 1]Ideal: Between [0 1]In computer we generate random integer [0,2In computer we generate random integer [0,2NN--1] or 1] or In computer we generate random integer [0,2In computer we generate random integer [0,2 1] or 1] or [0,2[0,2NN] then convert to [0,1]] then convert to [0,1]
•• Desired Properties:Desired Properties:L P i dL P i dLong PeriodLong PeriodUncorrelatedUncorrelatedFastFast
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Linear Congruential Generators:Linear Congruential Generators:Linear Congruential Generators:Linear Congruential Generators:
th d l 0 th d l 0( )1 mod 0n nX aX c m n+ = + ≥
m: the modulus m>0m: the modulus m>0a: the multiplier 0≤a<ma: the multiplier 0≤a<mc: the increment 0≤c<mc: the increment 0≤c<mXX00 the seed point 0≤Xthe seed point 0≤X00<m<mXX0:0: the seed point 0≤Xthe seed point 0≤X00<m<mmod: Integer remindermod: Integer reminder
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Linear Congruential Generators:Linear Congruential Generators:Linear Congruential Generators:Linear Congruential Generators:
Three Classes:Three Classes:( )1 mod 0n nX aX c m n+ = + ≥
Three Classes:Three Classes:•• m=2m=2NN, c>0, c>0•• m=2m=2NN c=0 c=0•• m=2m=2 , c=0, c=0•• m=prime, c=0m=prime, c=0Two first are fast and easy Third has Two first are fast and easy Third has Two first are fast and easy. Third has Two first are fast and easy. Third has high randomness.high randomness.Choice of m a and c are critical!Choice of m a and c are critical!
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Choice of m, a, and c are critical!Choice of m, a, and c are critical!
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Linear Congruential Generators:Linear Congruential Generators:Linear Congruential Generators:Linear Congruential Generators:•• Let a=1,c=5,m=16 and xLet a=1,c=5,m=16 and x00=1. =1. •• The sequence of pseudoThe sequence of pseudo random random •• The sequence of pseudoThe sequence of pseudo--random random
integers generated by this algorithm is: integers generated by this algorithm is: 1,6,15,12,13,2,11,8,9,14,7,4,5,10,3,01,6,15,12,13,2,11,8,9,14,7,4,5,10,3,0,,111,6,15,12,13,2,11,8,9,14,7,4,5,10,3,01,6,15,12,13,2,11,8,9,14,7,4,5,10,3,0,,11,6,15,12,13,2,11,8,9,14,6,15,12,13,2,11,8,9,14,,……..
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Common Examples:Common Examples:Common Examples:Common Examples:
Name m a c period
ANSI CANSI C 223131 11035152451103515245 1234512345 223131
ParkPark--MillerMiller 223131--11 1680716807 00 223131--22
drand48drand48 224848 2521490391725214903917 1111 224848drand48drand48 224848 2521490391725214903917 1111 224848
Hayes 64Hayes 64--bitbit 226464 63641362238467930056364136223846793005 11 226464
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Improvement on LCG Method:Improvement on LCG Method:Improvement on LCG Method:Improvement on LCG Method:•• Multiple Recursive GeneratorMultiple Recursive Generator
1 -1 2 -2 -( ) mod i i i k i kX a X a X a X c m= + + + +
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Matlab Coding:Matlab Coding:Matlab Coding:Matlab Coding:
x(1) = 1;x(1) 1;
a = 16807;
m = 2^31 – 1;
c = 0;
for n=1:10000,
x(n+1)= mod(a*x(n)+c m); x(n+1)= mod(a*x(n)+c,m);
end;
x = x/(m+1);
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
hist(x)
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Matlab Coding:Matlab Coding:Matlab Coding:Matlab Coding:
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Lagged Fibonacci Generators:Lagged Fibonacci Generators:Lagged Fibonacci Generators:Lagged Fibonacci Generators:•• Remember Fibonacci Sequence:Remember Fibonacci Sequence:
XX =X=X +X+X ; X; X = 0; X= 0; X =1=1XXnn=X=Xnn--11+X+Xnn--22; X; X00 = 0; X= 0; X11=1=1
•• General Formulation:General Formulation:( ) modX X X m= ( ) mod
: An Operator: +, -, *r>s>0
n n r n sX X X m− −=
••
N
r>s>0m=2
random seed required!r
•
••
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
random seed required!Popular Value of (r,s): (17,5), (55,24), (127,97), (607,273), (1279,418)r
•
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
How to generate other distributions:How to generate other distributions:How to generate other distributions:How to generate other distributions:•• A mathematical theory:A mathematical theory:
If we pass a Uni[0 1] random variables from If we pass a Uni[0 1] random variables from If we pass a Uni[0,1] random variables from If we pass a Uni[0,1] random variables from inverse of desired cumulative density inverse of desired cumulative density function, then output will have distribution function, then output will have distribution f b bilit d it f tif b bilit d it f tiof probability density function:of probability density function:
[ ]0,1u U∼
( ) ( )1X Xx F u f x−= ∼
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
How to generate other distributions:How to generate other distributions:How to generate other distributions:How to generate other distributions:•• Example:Example:
We want 10000 random variable with We want 10000 random variable with We want 10000 random variable with We want 10000 random variable with distribution of distribution of
( ) 0xe xf
−⎧ ≥⎨
CDF is:CDF is:
( )0 O.W.Xf x = ⎨⎩
1 0x−⎧ ≥( ) ( )
( ) [ ]
11 0ln 1
0 O.W.
l 1 0 1
x
X Xe x
F x F x
U
−−⎧ − ≥
= ⇒ = − −⎨⎩
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
( ) [ ]ln 1 0,1x x U= − − ∼
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Method is good but Method is good but NOTNOT practical:practical:Method is good but Method is good but NOTNOT practical:practical:•• For example consider normal For example consider normal
distribution:distribution:distribution:distribution:
( ) ( )2 2
2 21 1 ??xx s
f x e F x e ds− −
= ⇒ = =∫( ) ( )2 2
2 2??X Xf x e F x e ds
π π −∞
= ⇒ = =∫
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
BoxBox--Muller Method for Normal:Muller Method for Normal:BoxBox Muller Method for Normal:Muller Method for Normal:•• Suppose uSuppose u11 and uand u2 2 are two independent are two independent
uniform random variable then:uniform random variable then:uniform random variable, then:uniform random variable, then:
π= − 1 22log( ) cos(2 )x u u
are jointly normal random variable;are jointly normal random variable;
π= − 1 22log( ) sin(2 )y u u
are jointly normal random variable;are jointly normal random variable;
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Central Limit Theorem for Normal:Central Limit Theorem for Normal:Central Limit Theorem for Normal:Central Limit Theorem for Normal:•• Average of N independent random Average of N independent random
variable (with zero mean) from any variable (with zero mean) from any variable (with zero mean) from any variable (with zero mean) from any distribution converge in limits to distribution converge in limits to normal!normal!
=→∞=∑
∼1lim
N
ii
N
xy Normal
N→∞mNy mN
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
1 21 2
4 12
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Matlab Command:Matlab Command:Matlab Command:Matlab Command:•• rand(m,n): A mrand(m,n): A m××n matrix with Uni[0,1]n matrix with Uni[0,1]•• s = rand('state') : s = rand('state') : Return Current state.Return Current state.s rand( state ) : s rand( state ) : Return Current state.Return Current state.•• rand('state',s): rand('state',s): Resets the state to s.Resets the state to s.•• rand('state',0): rand('state',0): Resets the generator to Resets the generator to ( , )( , ) gg
its initial state. its initial state. •• rand('state',j): rand('state',j): For integer j, resets the For integer j, resets the
t t it jt t it j th t tth t tgenerator to its jgenerator to its j--th state.th state.•• rand('state',sum(100*clock)): rand('state',sum(100*clock)): Resets it Resets it
to a different state each timeto a different state each time
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
to a different state each time.to a different state each time.
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Matlab Command:Matlab Command:Matlab Command:Matlab Command:•• rand(m,n): A mrand(m,n): A m××n matrix with Uni[0,1]n matrix with Uni[0,1]•• How to u[a b]?How to u[a b]?•• How to u[a,b]?How to u[a,b]?
a = 10; b=20;a = 10; b=20;u=rand(1 1000);u=rand(1 1000);u=rand(1,1000);u=rand(1,1000);x = a + (bx = a + (b--a) * u;a) * u;
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Matlab Command:Matlab Command:Matlab Command:Matlab Command:•• randrandnn(m,n): A m(m,n): A m××n matrix with N[0,1]n matrix with N[0,1]•• s = rand('state') : s = rand('state') : Return Current state.Return Current state.s rand( state ) : s rand( state ) : Return Current state.Return Current state.•• rand('state',s): rand('state',s): Resets the state to s.Resets the state to s.•• rand('state',0): rand('state',0): Resets the generator to Resets the generator to ( , )( , ) gg
its initial state. its initial state. •• rand('state',j): rand('state',j): For integer j, resets the For integer j, resets the
t t it jt t it j th t tth t tgenerator to its jgenerator to its j--th state.th state.•• rand('state',sum(100*clock)): rand('state',sum(100*clock)): Resets it Resets it
to a different state each timeto a different state each time
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
to a different state each time.to a different state each time.
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Matlab Command:Matlab Command:Matlab Command:Matlab Command:•• rand(m,n): A mrand(m,n): A m××n matrix with Uni[0,1]n matrix with Uni[0,1]•• How to N[mHow to N[m δδ22]?]?•• How to N[m,How to N[m,δδ22]?]?
m=10; s=16;m=10; s=16;u=randn(1 1000);u=randn(1 1000);u=randn(1,1000);u=randn(1,1000);x = m + sqrt(s) * u;x = m + sqrt(s) * u;
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Matlab Command:Matlab Command:Matlab Command:Matlab Command:•• p = randperm(n); Random permutation p = randperm(n); Random permutation
of 1:n:of 1:n:of 1:n:of 1:n:p=randperm(5);p=randperm(5);
•• P=[2 5 1 3 4];P=[2 5 1 3 4];
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Pseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number GeneratorPseudo Random Number Generator
Matlab Command:Matlab Command:Matlab Command:Matlab Command:•• Other distribution:Other distribution:
exprndexprnd Exponential random numbersExponential random numbersexprndexprnd Exponential random numbersExponential random numberstrndtrnd Student's t random numbersStudent's t random numbersraylrndraylrnd Rayleigh random numbersRayleigh random numbersyy y gy gSee See StatisticsStatistics Toolbox for others.Toolbox for others.
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Monte Carlo MethodMonte Carlo MethodMonte Carlo MethodMonte Carlo Method
Monte Carlo IntegrationMonte Carlo IntegrationMonte Carlo IntegrationMonte Carlo Integration••ππ estimation as a simple example:estimation as a simple example:
l fl fGenerate N couples of RVs Generate N couples of RVs
4insideˆ N
2
inside
TotalNπ =
-4 -2 2 4
-2
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
-4
Monte Carlo MethodMonte Carlo MethodMonte Carlo MethodMonte Carlo Method
MATLAb experiment:MATLAb experiment:MATLAb experiment:MATLAb experiment:N = 1000;AB = 2*(rand(2,N)-0.5);
( )a = AB(1,:);b = AB(2,:);c = sqrt(a.^2+b.^2);PIh = 4*length(find(c<=1))/length(c);PIh 4 length(find(c< 1))/length(c);err = 100*abs(PIh-pi)/pi;
NN 101022 101033 101044 101044NN 1010 1010 1010 1010PihPih 3.32003.3200 3.10803.1080 3.12003.1200 3.13893.1389
%% 5 67895 6789 1 06931 0693 0 68730 6873 0 08510 0851
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
err%err% 5.67895.6789 1.06931.0693 0.68730.6873 0.08510.0851
Monte Carlo MethodMonte Carlo MethodMonte Carlo MethodMonte Carlo Method
Monte Carlo IntegrationMonte Carlo IntegrationMonte Carlo IntegrationMonte Carlo Integration••Another ExampleAnother Example::
l fl fGenerate N couples of RVs Generate N couples of RVs
insideˆ N 4inside
TotalNπ =
2
-4 -2 2 4
-2
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
-4
Monte Carlo MethodMonte Carlo MethodMonte Carlo MethodMonte Carlo Method
Simple Approach:Simple Approach:Simple Approach:Simple Approach:•• Random Point Selection and count!Random Point Selection and count!
f N∫( ) :x y
h
( ) White
W t de ehi Ra
Ng x dxN N
≈+∫( )
( )
, :1)2)
x ya x bf x y h≤ ≤
≤ ≤( ), :1)x ya x b≤ ≤
( )2) f x y h≤ ≤
( )1)2)0
a x by f x
≤ ≤
≤ ≤
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
a b
Monte Carlo IntegrationMonte Carlo IntegrationMonte Carlo IntegrationMonte Carlo Integration
Simple Formulation:Simple Formulation:Simple Formulation:Simple Formulation:•• Expected Value:Expected Value:{ } ( ) { } 1b b
UniE x xf x dx E x xdx= → =∫ ∫{ } ( ) { }
( ){ } ( )1
xa a
b
E x xf x dx E x xdxb a
E g x g x dx
= ⎯⎯→ =−
=
∫ ∫
∫•• Integral:Integral:
( ){ } ( )a
E g x g x dxb a
=− ∫
1 b
( ){ } ( )1
1 1a
b bN N
E g x g x dxb a
b a
=− ∫
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
( ) ( ) ( ) ( )1 1
1 1N N
i ii ia a
b ag x g x dx g x dx g xN b a N= =
−≈ ⇒ ≈
−∑ ∑∫ ∫
Monte Carlo IntegrationMonte Carlo IntegrationMonte Carlo IntegrationMonte Carlo Integration
Example and MATLAB Code:Example and MATLAB Code:Example and MATLAB Code:Example and MATLAB Code:
[ ]1
sin sin , Uni 0,N
i ii
I dN
π πθ θ θ θ π= ≈ ∑∫ ∼10
Exact value of I: 2iN =
N = 100;a = 0;b = pi;x = rand(1,N)*(b-a)+a;Ih = sum(sin(x))*(b-a)/N;
NN 101022 101033 101044 101044NN 101022 101033 101044 101044
II 1.99421.9942 2.02042.0204 2.00742.0074 2.00112.0011
err%err% 0.29130.2913 --1.01861.0186 --0.36830.3683 --0.05650.0565
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
err%err% 0.29130.2913 1.01861.0186 0.36830.3683 0.05650.0565
Function MinimizationFunction MinimizationFunction MinimizationFunction Minimization
Problem Statement:Problem Statement:Problem Statement:Problem Statement:
( )minx
f x
Methods:Methods:
x
•• Pure MathematicalPure Mathematical•• Direct SearchDirect Search
( ) ( )0,f x f′ = ∇ =x 0
•• Steepest DescentSteepest Descent
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Function MinimizationFunction MinimizationFunction MinimizationFunction Minimization
Example:Example:Example:Example:•• Best Elliptic that match a set of points!Best Elliptic that match a set of points!
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Function MinimizationFunction MinimizationFunction MinimizationFunction Minimization
Example:Example:Example:Example:•• Try to solve this problem:Try to solve this problem:
( )h fN( )i i 1
22 22 2
we have a set of x ,yN
i
Ni ix ye r
=
⎡ ⎤⎛ ⎞ ⎛ ⎞+⎢ ⎥⎜ ⎟ ⎜ ⎟∑1
2 2 2
0 ???
i i
ie r
a b
e e e
=
= + −⎢ ⎥⎜ ⎟ ⎜ ⎟⎝ ⎠ ⎝ ⎠⎢ ⎥⎣ ⎦
∂ ∂ ∂→
∑
0 ???a b r= = = →
∂ ∂ ∂
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Function MinimizationFunction MinimizationFunction MinimizationFunction Minimization
Multivariable function F(Multivariable function F(xx):):Multivariable function F(Multivariable function F(xx):):•• Gradient: A vector:Gradient: A vector:
TF F F⎡ ⎤∂ ∂ ∂
Hessian: A Matrix:Hessian: A Matrix:
( )1 2
, , ,n
F F FGx x x
⎡ ⎤∂ ∂ ∂= ⎢ ⎥∂ ∂ ∂⎣ ⎦
x …
•• Hessian: A Matrix:Hessian: A Matrix:
( ) ( )ij n nH H
×⎡ ⎤= ⎣ ⎦x x
( ) ( )2n n
iji j
FH
x x
×⎣ ⎦∂
=∂ ∂
xx
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
i j
Function MinimizationFunction MinimizationFunction MinimizationFunction Minimization
Multivariable function F(Multivariable function F(xx):):Multivariable function F(Multivariable function F(xx):):•• Taylor Series:Taylor Series:
1
Example: Example:
( ) ( ) ( ) ( )12
T TF F G H+ = + + +x h x x h h x h
•• Example: Example:
( ) ( ) ( ) 1 21 2 1 2, cos sin x xF x x x x e= + +
( ) ( ) ( )
( ) ( ) ( )
1 2 1 2
1 2 1 2
1 2 1 2 2 1
21 2 1 2
, sin ,cos
cos 1
x x x x
x x x x
G x x x x e x x e
x x e x x eG
⎡ ⎤= − + +⎣ ⎦⎡ ⎤− + +⎢ ⎥
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
( ) ( ) ( )( ) ( )1 2 1 2
1 2 1 21 2 2
1 2 2 1
,1 sinx x x xG x x
x x e x x e= ⎢ ⎥+ − +⎣ ⎦
Function MinimizationFunction MinimizationFunction MinimizationFunction Minimization
Steepest Descent Rule:Steepest Descent Rule:Steepest Descent Rule:Steepest Descent Rule:•• Change function in direction of Change function in direction of ––G(G(xx))
Example in 1DExample in 1DExample in 1DExample in 1D
0G∇ >0G∇ < 0G∇ >0G∇ <
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Function MinimizationFunction MinimizationFunction MinimizationFunction Minimization
Steepest Descent Rule:Steepest Descent Rule:Steepest Descent Rule:Steepest Descent Rule:•• Very Simple solution:Very Simple solution:
Some methods solve this 1D Some methods solve this 1D
( )( )1 0,k k kGε+ = + −∇x x x x
•• Some methods solve this 1D Some methods solve this 1D minimization:minimization:
( )( )( )
mint
F tG
tG
+
← +
x x
x x x
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
( )
Function MinimizationFunction MinimizationFunction MinimizationFunction Minimization
More Advanced Method:More Advanced Method:More Advanced Method:More Advanced Method:
( ) ( ) ( ) ( )12
T TF F G H+ ≈ + +x h x x h h x h
( ) ( ) 12
n n n
i i i ij ji i i
F F G h h H h= = =
+ = + +∑ ∑∑x h x
( )1
0, 1, 2, ,n
k kj jjk
FG H h k n
h =
∂ += + = =
∂ ∑x h
1
1
, 1, 2, ,n
kj j kj
H h G k n=
= − =∑
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
1kH G−∴ = −h
Function MinimizationFunction MinimizationFunction MinimizationFunction Minimization
Matlab CommandMatlab CommandMatlab CommandMatlab Command•• Fmin(f,x0): 1DFmin(f,x0): 1D•• Fminsearch(f x0): nDFminsearch(f x0): nD•• Fminsearch(f,x0): nDFminsearch(f,x0): nD•• Optimization toolboxOptimization toolbox
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh
Function MinimizationFunction MinimizationFunction MinimizationFunction Minimization
Matlab Command:Matlab Command:Matlab Command:Matlab Command:•• p=rand(2,1);p=fminsearch(@ellip,p);p=rand(2,1);p=fminsearch(@ellip,p);
f ti lli ( )function e = ellip(x);a = x(1);b = x(2);t = 0:359;t 0 359;xi = 5*sin(t*pi/180);yi = 40*cos(t*pi/180);e = 0;for i=1:360for i=1:360,
e = e+((xi(i)/a).^2+(yi(i)/b).^2-1).^2;end;
Applied Numerical MethodsApplied Numerical MethodsE. FatemizadehE. Fatemizadeh