monte carlo methods and statistical computing: my …home.iitk.ac.in/~kundu/tirupati-2014-1.pdfmonte...

Monte Carlo Methods and Statistical Computing:My Personal Experience

Debasis Kundu

Department of Mathematics & Statistics

Indian Institute of Technology Kanpur

November 29, 2014

Debasis Kundu Monte Carlo Methods and Statistical Computing: My Personal Exp

Preface

A Brief History

Application

Major Ingradients

What we can do?

Statistical Computation

Stories Untold

Outline

1 Preface

2 A Brief History

3 Application

4 Major Ingradients

5 What we can do?

6 Statistical Computation

7 Stories Untold


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Limitations:

1. I must admit that the topics I am going tocover are definitely not exhaustive.

2. Topics are purely of my own interest whichhave developed over the last 30 years.

3. I am not going to describe any statisticalpackage.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Advantages:

1. Packages have their own problems.

2. Different packages can give different answerseven on a relatively simple problem.

3. We should know the limitations of thepackages.

4. I will try to provide a general approach


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Journals:

1. Journal of Computational and GraphicalStatistics.

2. Computational Statistics and Data Analysis.

3. Journal of Statistical Computation andSimulation.

4. Statistical Computing

5. Communications in Statistics - Simulationand Computation


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Books:

1. Simulation by Sheldon Ross

2. Nonuniform Random Deviate Generator, L.Devroye

3. Simulation modelling and analysis, Law andKelton Journal of Statistical Computation andSimulation.

4. Statistical Computing: J.F. Keneddy and R.Gentle

5. Statistical Computing, D. Kundu and A. Basu


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Outline

1 Preface

2 A Brief History

3 Application

4 Major Ingradients

5 What we can do?


7 Stories Untold


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Monte Carlo Method: Definition

1. A broad class of numerical alogrithm dependson repeated random sampling.

2. If it is not possible to obtain the exactanalytical solution often Monta Carlo methodcan be used to provide a very good approximatesolution


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Monte Carlo Method: A brief history

1. It was invented by Stanislaw Ulam, a famousPolish Mathematician, in the late 1940.

2. John von Neumann first wrote the computercode to perform Monte Carlo simulations

3. Metropolis gave this name


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Outline

1 Preface

2 A Brief History

3 Application

4 Major Ingradients

5 What we can do?


7 Stories Untold


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Where it can be used?

1. Calculating the area below a curve.

2. Calculating multidimensional integration.

3. Optimization.

4. Analyzing any complicated stochastic system(model).


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Examples

Suppose we want to compute

∫ b

a

e−x2dx .

Or suppose we want to compute

∫ b1

a1

. . .

∫ bk

ak

f (x1, . . . , xk)dx1 . . . dxk .


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Examples:Contd.

Suppose we want to find the maximum or minimumof the following function

f (x1, . . . , xk),

where a1 ≤ x1 ≤ b1, . . . , ak ≤ xk ≤ bk .

Or suppose we want to analyze the followingnon-linear model

y(x1, . . . , xk) = f (x1, . . . , xk , θ) + e.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Outline

1 Preface

2 A Brief History

3 Application

4 Major Ingradients

5 What we can do?


7 Stories Untold


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Back Ground

1. Knowledge of Basic Probability.

2. Discrete and Continuous random variables.

3. Stochastic models.

4. Generation of random numbers.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Knowlwedge of Basic Probability

1. Idea of a random experiment.

2. Basic idea of convergence of randomvariables.

3. Weak and strong law of large numbers.

4. Central limit theorem.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Discrete Random Variables

1. Uniform.

2. Binomial.

3. Geometric.

4. Poisson.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Continuous Random Variables

1. Uniform.

2. Exponential.

3. Normal.

4. Gamma.

5. Log-concave probability density function


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Generation of Random Numbers

First we need to know how to generate Uniformrandom numbers. This is the most basic problem.In this respect we use group theory results andmachine powers.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Generation of Non-Uniform Random Numbers

The most popular method is the inversetransformation. The following result can be used. IfX is a random variable with the distributionfunction F (x), then F (X ) follows uniformdistribution. Therefore

X = F−1(U)


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Generation of Discrete Random Numbers

All the discrete distributions can be generated usinginverse transformation method. SupposeP(X = ai) = pi , for i = 1, 2, . . .. Without loss ofgenerality we can assume a1 < a2 < . . .. Draw auniform random number say u, if∑k−1

i=1 pi < u <∑k

i=1 pi , then X takes the value ak .


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Generation of Continuous Random Numbers

Many continuous random variables can begenerated using inverse transformation method, forexample exponential, Weibull, generalizedexponential distributions etc. On the other handseveral well known distribution cannot be obtainedusing inverse transformation method. For examplenormal, gamma etc.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Generation of Continuous Random Numbers

If a continuous distribution cannot be generatedusing inverse transformation method, one of themost useful method is the acceptance rejectionmethod. The idea is as follows. If we want togenerate from f (x), try to find g(x), from whichgeneration is simple so that it satisfies the following

f (x) ≤ cg(x).


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Acceptance Rejection Method: Algorithm

1. Generate Y from g(x).

2. Generate a uniform random vaiable U .

3. If U ≤ f (Y )/cg(Y ), set X = Y , otherwisereturn to 1.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Acceptance Rejection Method: Theorem

Theorem:

1. The random variable generated by thismethod has density function f (x)

2. The number of iterations of the algorithmthat are needed is a geometric random variablewith mean c ,


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Acceptance Rejection Method: Example

Example 1:

Suppose we want to generate from

f (x) = 20x(1− x)3; 0 < x < 1.

Takeg(x) = 1, 0 < x < 1.

c = 135/64.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold


Example 2:


f (x) =2√πx1/2e−x ; 0 < x < ∞.

Take

g(x) =2

3e−2x/3 0 < x < ∞.

and

c =33/2

(2πe)1/2.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold


Example 3:


f (x) =2√2π

e−x2/2; 0 < x < ∞.

Takeg(x) = e−x ; 0 < x < ∞.

andc =

√2e/π.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Outline

1 Preface

2 A Brief History

3 Application

4 Major Ingradients

5 What we can do?


7 Stories Untold


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Very Simple Example

Consider the following simple linear regression model

Y = Xb+ e

We know the LSE’s can be obtained as

b̂ = (XTX)−1

XTY.

We have a complete very nice theory when all thecomponents of the errors are i.i.d. normal randomvariables.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Very Simple Example: Contd.

Consider some slightly different conditions of thesame model.

1. What will happen if the errors are notnormal?

2. What will happen if the errors are heavy tail?

3. What will happen if there are outliers?

4. What will happen if the errors are correlated?


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Very Simple Example: Contd.

In all these cases Monte Carlo Method can be usedto asses the performances of the estmimators. It isvery simple also.

1. Generate e

2. Generate Y.

3. Calculate b̂.

4. Repeat step 1 to step 4, several times.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Example

Consider the following simple linear regression model

Y = Xb+ e

Suppose we want to estimate b by minimizing theleast absolute errors i.e.

b̂ = argmin|Y − Xb|.

Theories are quite complicated. All the results areasymptotic in nature.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Example

Consider the following non-linear regression model

Y = f (X, θ) + e

Here f is a known function the vector X is alsoknown, the paramete vector θ is unknown. Theproblem is to estimate the parameter vector θ,based on a sample of size n.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Example

Natural estimators will be

θ̂ = argmin|n∑

i=1

Yi − f (Xi , θ)|2.

or

θ̂ = argmin|n∑

i=1

Yi − f (Xi , θ)|.

Theories are quite complicated. All the results areasymptotic in nature.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Example

Monte Carlo method can be used to asses theperofrmance of the estimators.

Based on the Monte Carlo method the biasesand the mean squared errors can be calculated.

Based on Bootstrap method confidence intervalsalso can be obtained.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Example: Importance Sampling

In Bayesian analysis often we need to compute theposterior mean as follows:

θ = E (h(X)) =

∫h(x)f (x)dx.

Here f (x) is the PDF of X, and x can be a veryhigh dimensional. In Bayesian analysis f (x) is theposterior density function.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold


Monte Carlo simulation technique can be used toapproximate the value of θ as follows:

θ̂ =1

N

N∑

i=1

h(Xi),

here X1, . . . ,XN is a random sample of size N fromf (X).


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold


Often it is observed that it is not very easy togenerate samples from f (x).

θ =

∫h(x)f (x)dx =

∫h(x)f (x)

g(x)g(x)d(x).

θ̂ =1

N

N∑

i=1

h(xi)f (xi)

g(xi),

here X1, . . . ,XN is a random sample of size N fromg(X).


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Outline

1 Preface

2 A Brief History

3 Application

4 Major Ingradients

5 What we can do?


7 Stories Untold


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Important Issues

1. Finding Maximum likelihood estimators in ageneral problem.

2. Finding least squares estimators of linearregresssion model when the design matrix isclose to a singular matrix

3. Non-linear regression model if the number ofparameters are very high


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

MLE

It basically invloves maximizing a function of theform:

f (θ1, . . . , θp)

Standard method is to use Newton-Raphsonmethod:


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Newton-Raphson Method

Assuming sufficiently smooth f (θ), we want to solve

∂f (θ)

∂θ= 0

Standard method is to use Newton-Raphsonmethod. Using Taylor series expansion, it can beeasily obtained:

θ(k+1) = θ(k) −[∂2f (θ(k))

∂θ∂θT

]−1∂f (θ(k))

∂θ


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Profile Likelihood Method

1. For fixed θ1, . . . , θk , try to maximize withrespect to θk+1, . . . , θp

2. Maximize with respect to θ1, . . . , θk .


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

EM Algorithm

Suppose the data are coming from a mixture model,and we compute the MLEs of the unknownparameters

f (x) =k∑

j=1

πj fj(x ; θj),

πj ≥ 0,∑k

j=1 πj = 1.


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Mixture Model: MLE

Based on a random sample x1, . . . , xn, we want tocompute the MLEs of the unknown parameters

L(π, θ) =n∏

i=1

k∑

j=1

πj fj(xi ; θj).


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Missing Value Problem

We treat this as a missing value problem

1. Assume the data are of the form (x , δ)

2. Compute E (δ|Data)3. Continue the process


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Copula Method

Any multivarite distribution can be written uniquelyas follows:

F (x1, . . . , xp; θ) = C (F1(x1; θ1), . . . , Fp(xp : θp); γ)

First estimate the marginal parameters, and thenestimate the copula parameters


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Non-linear regression

Consider the following model

y(t) =

p∑

k=1

[Ak cos(ωkt) + Bk sin(ωkt)] + e(t)

Estimate the unknown parameters


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Outline

1 Preface

2 A Brief History

3 Application

4 Major Ingradients

5 What we can do?


7 Stories Untold


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Very important areas

Bayesian comutation: mainly MCMC

Classification problem

Small n large p problem

Non-parametric regression

Functional data analysis


Preface

A Brief History

Application

Major Ingradients

What we can do?


Stories Untold

Thank You


monte carlo methods and statistical computing: my …home.iitk.ac.in/~kundu/tirupati-2014-1.pdfmonte...

Documents