estimators and estimates: an estimator is a mathematical formula. an estimate is a number obtained...
TRANSCRIPT
Estimators and estimates:
An estimator is a mathematical formula.
An estimate is a number obtained by applyingthis formula to a set of sample data.
1
ESTIMATORS
It is important to distinguish between estimators and estimates. Definitions are given above.
EMU, ECON 503, M. Balcılar
Population characteristic Estimator
Mean: X
4
ESTIMATORS
n
iixn
X1
1
A common example of an estimator is the sample mean, which is the usual estimator of the population mean.
EMU, ECON 503, M. Balcılar
Population characteristic Estimator
Mean: X
4
ESTIMATORS
n
iixn
X1
1
Here it is defined for a random variable X and a sample of n observations.
EMU, ECON 503, M. Balcılar
Population characteristic Estimator
Mean: X
Population variance:
4
ESTIMATORS
Another common estimator is s2, defined above. It is used to estimate the population variance, X
2.
2X
n
iixn
X1
1
n
ii Xx
ns
1
22 )(1
1
EMU, ECON 503, M. Balcılar
Estimators are random variables
8
ESTIMATORS
)...(11
11
n
n
ii xxn
xn
X
An estimator is a special kind of random variable. We will demonstrate this in the case of the sample mean.
EMU, ECON 503, M. Balcılar
Estimators are random variables
8
ESTIMATORS
)...(11
11
n
n
ii xxn
xn
X
iXi ux
We saw in the previous sequence that each observation on X can be decomposed into a fixed component and a random component.
EMU, ECON 503, M. Balcılar
Estimators are random variables
8
ESTIMATORS
)...(11
11
n
n
ii xxn
xn
X
iXi ux
uunn
uunn
X
XX
nXX
)(1
)...(1
)...(1
1
So the sample mean is the average of n fixed components and n random components.
EMU, ECON 503, M. Balcılar
Estimators are random variables
8
ESTIMATORS
It thus has a fixed component X and a random component u, the average of the random components in the observations in the sample.
)...(11
11
n
n
ii xxn
xn
X
iXi ux
uunn
uunn
X
XX
nXX
)(1
)...(1
)...(1
1
EMU, ECON 503, M. Balcılar
10
ESTIMATORS
probability density
function of X
X XXX
probability density
function of X
The graph compares the probability density functions of X and X. As we have seen, they have the same fixed component. However the distribution of the sample mean is more concentrated.
EMU, ECON 503, M. Balcılar
10
ESTIMATORS
Its random component tends to be smaller than that of X because it is the average of the random components in all the observations, and these tend to cancel each other out.
probability density
function of X
X XXX
probability density
function of X
EMU, ECON 503, M. Balcılar
Unbiasedness of X:
1
UNBIASEDNESS AND EFFICIENCY
XXn
nn
nn
xExEn
xxEn
xxn
EXE
1)(...)(
1
)...(1
)...(1
)(
1
11
Suppose that you wish to estimate the population mean X of a random variable X given a sample of observations. We will demonstrate that the sample mean is an unbiased estimator, but not the only one.
EMU, ECON 503, M. Balcılar
Unbiasedness of X:
2
UNBIASEDNESS AND EFFICIENCY
XXn
nn
nn
xExEn
xxEn
xxn
EXE
1)(...)(
1
)...(1
)...(1
)(
1
11
We use the second expected value rule to take the (1/n) factor out of the expectation expression.
EMU, ECON 503, M. Balcılar
Unbiasedness of X:
3
UNBIASEDNESS AND EFFICIENCY
XXn
nn
nn
xExEn
xxEn
xxn
EXE
1)(...)(
1
)...(1
)...(1
)(
1
11
Next we use the first expected value rule to break up the expression into the sum of the expectations of the observations.
EMU, ECON 503, M. Balcılar
Unbiasedness of X:
4
UNBIASEDNESS AND EFFICIENCY
XXn
nn
nn
xExEn
xxEn
xxn
EXE
1)(...)(
1
)...(1
)...(1
)(
1
11
Each expectation is equal to X, and hence the expected value of the sample mean is X.
EMU, ECON 503, M. Balcılar
probabilitydensityfunction
X
estimator B
How do we choose among them? The answer is to use the most efficient estimator, the one with the smallest population variance, because it will tend to be the most accurate.
UNBIASEDNESS AND EFFICIENCY
estimator A
12EMU, ECON 503, M. Balcılar
probabilitydensityfunction
estimator B
In the diagram, A and B are both unbiased estimators but B is superior because it is more efficient.
UNBIASEDNESS AND EFFICIENCY
estimator A
13
X
EMU, ECON 503, M. Balcılar
1
CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE
Suppose that you have alternative estimators of a population characteristic , one unbiased, the other biased but with a smaller population variance. How do you choose between them?
probabilitydensityfunction
estimator B
estimator A
EMU, ECON 503, M. Balcılar
CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE
One way is to define a loss function which reflects the cost to you of making errors, positive or negative, of different sizes.
2
error (positive)error (negative)
loss
EMU, ECON 503, M. Balcılar
3
CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE
A widely-used loss function is the mean square error of the estimator, defined as the expected value of the square of the deviation of the estimator about the true value of the population characteristic.
probabilitydensityfunction
222 )()()(MSE ZZZEZ
estimator B
EMU, ECON 503, M. Balcılar
4
CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE
The mean square error involves a trade-off between the population variance of the estimator and its bias. Suppose you have a biased estimator like estimator B above, with expected value Z.
probabilitydensityfunction
Z
bias
222 )()()(MSE ZZZEZ
estimator B
EMU, ECON 503, M. Balcılar
5
CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE
The mean square error can be shown to be equal to the sum of the population variance of the estimator and the square of the bias.
probabilitydensityfunction
Z
bias
222 )()()(MSE ZZZEZ
estimator B
EMU, ECON 503, M. Balcılar
6
CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE
To demonstrate this, we start by subtracting and adding Z .
22
22
22
22
22
2
2
)(
))((2)(
)()(2)(
))((2)()(
))((2)()(
)(
)()(MSE
ZZ
ZZZZZ
ZZZZ
ZZZZ
ZZZZ
ZZ
ZE
ZEEZE
ZZE
ZE
ZEZ
EMU, ECON 503, M. Balcılar
7
CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE
We expand the quadratic using the rule (a + b)2 = a2 + b2 + 2ab, where a = Z - Z and b = Z - .
22
22
22
22
22
2
2
)(
))((2)(
)()(2)(
))((2)()(
))((2)()(
)(
)()(MSE
ZZ
ZZZZZ
ZZZZ
ZZZZ
ZZZZ
ZZ
ZE
ZEEZE
ZZE
ZE
ZEZ
EMU, ECON 503, M. Balcılar
8
CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE
We use the first expected value rule to break up the expectation into its three components.
22
22
22
22
22
2
2
)(
))((2)(
)()(2)(
))((2)()(
))((2)()(
)(
)()(MSE
ZZ
ZZZZZ
ZZZZ
ZZZZ
ZZZZ
ZZ
ZE
ZEEZE
ZZE
ZE
ZEZ
EMU, ECON 503, M. Balcılar
9
CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE
The first term in the expression is by definition the population variance of Z.
22
22
22
22
22
2
2
)(
))((2)(
)()(2)(
))((2)()(
))((2)()(
)(
)()(MSE
ZZ
ZZZZZ
ZZZZ
ZZZZ
ZZZZ
ZZ
ZE
ZEEZE
ZZE
ZE
ZEZ
EMU, ECON 503, M. Balcılar
10
CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE
(Z - ) is a constant, so the second term is a constant.
22
22
22
22
22
2
2
)(
))((2)(
)()(2)(
))((2)()(
))((2)()(
)(
)()(MSE
ZZ
ZZZZZ
ZZZZ
ZZZZ
ZZZZ
ZZ
ZE
ZEEZE
ZZE
ZE
ZEZ
EMU, ECON 503, M. Balcılar
11
CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE
In the third term, (Z - ) may be brought out of the expectation, again because it is a constant, using the second expected value rule.
22
22
22
22
22
2
2
)(
))((2)(
)()(2)(
))((2)()(
))((2)()(
)(
)()(MSE
ZZ
ZZZZZ
ZZZZ
ZZZZ
ZZZZ
ZZ
ZE
ZEEZE
ZZE
ZE
ZEZ
EMU, ECON 503, M. Balcılar
12
CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE
Now E(Z) is Z, and E(-Z) is -Z.
22
22
22
22
22
2
2
)(
))((2)(
)()(2)(
))((2)()(
))((2)()(
)(
)()(MSE
ZZ
ZZZZZ
ZZZZ
ZZZZ
ZZZZ
ZZ
ZE
ZEEZE
ZZE
ZE
ZEZ
EMU, ECON 503, M. Balcılar
13
CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE
Hence the third term is zero and the mean square error of Z is shown be the sum of the population variance of Z and the bias squared.
22
22
22
22
22
2
2
)(
))((2)(
)()(2)(
))((2)()(
))((2)()(
)(
)()(MSE
ZZ
ZZZZZ
ZZZZ
ZZZZ
ZZZZ
ZZ
ZE
ZEEZE
ZZE
ZE
ZEZ
EMU, ECON 503, M. Balcılar
14
CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE
In the case of the estimators shown, estimator B is probably a little better than estimator A according to the MSE criterion.
probabilitydensityfunction
estimator B
estimator A
EMU, ECON 503, M. Balcılar
n
1 50
1
The sample mean is the usual estimator of a population mean, for reasons discussed in the previous sequence. In this sequence we will see how its properties are affected by the sample size.
probability density function of X
50 100 150 200
n = 1
EFFECT OF INCREASING THE SAMPLE SIZE ON THE DISTRIBUTION OF x
0.08
0.04
0.02
0.06
X
EMU, ECON 503, M. Balcılar
n
1 50
2
Suppose that a random variable X has population mean 100 and standard deviation 50, as in the diagram. Suppose that we do not know the population mean and we are using the sample mean to estimate it.
50 100 150 200
n = 1
EFFECT OF INCREASING THE SAMPLE SIZE ON THE DISTRIBUTION OF x
0.08
0.04
0.02
0.06
probability density function of X
X
EMU, ECON 503, M. Balcılar
n
1 50
3
The sample mean will have the same population mean as X, but its standard deviation will be 50/ , where n is the number of observations in the sample.
50 100 150 200
n = 1
EFFECT OF INCREASING THE SAMPLE SIZE ON THE DISTRIBUTION OF x
n
0.08
0.04
0.02
0.06
probability density function of X
X
EMU, ECON 503, M. Balcılar
n
1 50
4
The larger is the sample, the smaller will be the standard deviation of the sample mean.
50 100 150 200
n = 1
EFFECT OF INCREASING THE SAMPLE SIZE ON THE DISTRIBUTION OF x
0.08
0.04
0.02
0.06
probability density function of X
X
EMU, ECON 503, M. Balcılar
n
1 50
5
If n is equal to 1, the sample consists of a single observation. X is the same as X and its standard deviation is 50.
50 100 150 200
n = 1
EFFECT OF INCREASING THE SAMPLE SIZE ON THE DISTRIBUTION OF x
0.08
0.04
0.02
0.06
probability density function of X
X
EMU, ECON 503, M. Balcılar
n
1 504 25
6
We will see how the shape of the distribution changes as the sample size is increased.
50 100 150 200
n = 4
EFFECT OF INCREASING THE SAMPLE SIZE ON THE DISTRIBUTION OF x
0.08
0.04
0.02
0.06
probability density function of X
X
EMU, ECON 503, M. Balcılar
n
1 504 25
25 10
7
The distribution becomes more concentrated about the population mean.
50 100 150 200
n = 25
EFFECT OF INCREASING THE SAMPLE SIZE ON THE DISTRIBUTION OF x
0.08
0.04
0.02
0.06
probability density function of X
X
EMU, ECON 503, M. Balcılar
n
1 504 25
25 10100 5
8
To see what happens for n greater than 100, we will have to change the vertical scale.
50 100 150 200
0.08
0.04
n = 100
EFFECT OF INCREASING THE SAMPLE SIZE ON THE DISTRIBUTION OF x
0.02
0.06
probability density function of X
X
EMU, ECON 503, M. Balcılar
n
1 504 25
25 10100 5
9
We have increased the vertical scale by a factor of 10.
50 100 150 200
n = 100
EFFECT OF INCREASING THE SAMPLE SIZE ON THE DISTRIBUTION OF x
0.8
0.4
0.2
0.6
probability density function of X
X
EMU, ECON 503, M. Balcılar
n
1 504 25
25 10100 5
1000 1.6
10
The distribution continues to contract about the population mean.
50 100 150 200
n = 1000
EFFECT OF INCREASING THE SAMPLE SIZE ON THE DISTRIBUTION OF x
0.8
0.4
0.2
0.6
probability density function of X
X
EMU, ECON 503, M. Balcılar
n
1 504 25
25 10100 5
1000 1.65000 0.7
11
In the limit, the variance of the distribution tends to zero. The distribution collapses to a spike at the true value. The sample mean is therefore a consistent estimator of the population mean.
50 100 150 200
n = 5000
EFFECT OF INCREASING THE SAMPLE SIZE ON THE DISTRIBUTION OF x
0.8
0.4
0.2
0.6
probability density function of X
X
EMU, ECON 503, M. Balcılar