linear regression hypothesis testing and estimation
TRANSCRIPT
Linear Regression
Hypothesis testing and Estimation
Assume that we have collected data on two variables X and Y. Let
(x1, y1) (x2, y2) (x3, y3) … (xn, yn)
denote the pairs of measurements on the on two variables X and Y for n cases in a sample (or population)
The Statistical Model
Each yi is assumed to be randomly generated from a normal distribution with
mean i = + xi and standard deviation . (, and are unknown)
yi
+ xi
xi
Y = + X
slope =
The Data The Linear Regression Model
• The data falls roughly about a straight line.
0
20
40
60
80
100
120
140
160
40 60 80 100 120 140
Y = + X
unseen
The Least Squares Line
Fitting the best straight line
to “linear” data
LetY = a + b X
denote an arbitrary equation of a straight line.a and b are known values.This equation can be used to predict for each value of X, the value of Y.
For example, if X = xi (as for the ith case) then the predicted value of Y is:
ii bxay ˆ
The residual
can be computed for each case in the sample,
The residual sum of squares (RSS) is
a measure of the “goodness of fit of the line
Y = a + bX to the data
iiiii bxayyyr ˆ
,ˆ,,ˆ,ˆ 222111 nnn yyryyryyr
n
iii
n
iii
n
ii bxayyyrRSS
1
2
1
2
1
2 ˆ
The optimal choice of a and b will result in the residual sum of squares
attaining a minimum.
If this is the case than the line:
Y = a + bX
is called the Least Squares Line
n
iii
n
iii
n
ii bxayyyrRSS
1
2
1
2
1
2 ˆ
The equation for the least squares line
Let
n
iixx xxS
1
2
n
iiyy yyS
1
2
n
iiixy yyxxS
1
Linear Regression
Hypothesis testing and Estimation
The Least Squares Line
Fitting the best straight line
to “linear” data
n
x
xxxS
n
iin
ii
n
iixx
2
1
1
2
1
2
n
yx
yx
n
ii
n
iin
iii
11
1
n
y
yyyS
n
iin
ii
n
iiyy
2
1
1
2
1
2
n
iiixy yyxxS
1
Computing Formulae:
Then the slope of the least squares line can be shown to be:
n
ii
n
iii
xx
xy
xx
yyxx
S
Sb
1
2
1
and the intercept of the least squares line can be shown to be:
xS
Syxbya
xx
xy
The residual sum of Squares
22
1 1
ˆn n
i i i ii i
RSS y y y a bx
2
xy
yyxx
SS
S
Computing formula
Estimating , the standard deviation in the regression model :
22
ˆ1
2
1
2
n
bxay
n
yys
n
iii
n
iii
xx
xyyy S
SS
n
2
2
1
This estimate of is said to be based on n – 2 degrees of freedom
Computing formula
Sampling distributions of the estimators
The sampling distribution slope of the least squares line :
n
ii
n
iii
xx
xy
xx
yyxx
S
Sb
1
2
1
It can be shown that b has a normal distribution with mean and standard deviation
n
ii
xx
bb
xxS
1
2
and
Thus
has a standard normal distribution, and
b
b
xx
b bz
S
b
b
xx
b bt
ssS
has a t distribution with df = n - 2
(1 – )100% Confidence Limits for slope :
t/2 critical value for the t-distribution with n – 2 degrees of freedom
xxS
st ˆ
2/
Testing the slope
The test statistic is:
0 0 0: vs : AH H
0
xx
bt
sS
- has a t distribution with df = n – 2 if H0 is true.
The Critical Region
Reject
0 0 0: vs : AH H
0/ 2 / 2if or
xx
bt t t t
sS
df = n – 2
This is a two tailed tests. One tailed tests are also possible
The sampling distribution intercept of the least squares line :
It can be shown that a has a normal distribution with mean and standard deviation
n
ii
aa
xx
x
n
1
2
21 and
xS
Syxbya
xx
xy
Thus
has a standard normal distribution and
2
2
1
1
a
a
n
ii
a az
xn x x
2
2
1
1
a
a
n
ii
a at
s xs
n x x
has a t distribution with df = n - 2
(1 – )100% Confidence Limits for intercept :
t/2 critical value for the t-distribution with n – 2 degrees of freedom
1
ˆ2
2/xxS
x
nst
Testing the intercept
The test statistic is:
0 0 0: vs : AH H
- has a t distribution with df = n – 2 if H0 is true.
0
2
2
1
1
n
ii
at
xs
n x x
The Critical Region
Reject
0 0 0: vs : AH H
0/ 2 / 2if or
a
at t t t
s
df = n – 2
Example
The following data showed the per capita consumption of cigarettes per month (X) in various countries in 1930, and the death rates from lung cancer for men in 1950. TABLE : Per capita consumption of cigarettes per month (Xi) in n = 11 countries in 1930, and the death rates, Yi (per 100,000), from lung cancer for men in 1950.
Country (i) Xi Yi
Australia 48 18Canada 50 15Denmark 38 17Finland 110 35Great Britain 110 46Holland 49 24Iceland 23 6Norway 25 9Sweden 30 11Switzerland 51 25USA 130 20
Australia
CanadaDenmark
Finland
Great Britain
Holland
Iceland
NorwaySweden
Switzerland
USA
0
5
10
15
20
25
30
35
40
45
50
0 20 40 60 80 100 120 140
deat
h ra
tes f
rom
lung
can
cer
(195
0)
Per capita consumption of cigarettes
404,541
2
n
iix
914,161
n
iii yx
018,61
2
n
iiy
Fitting the Least Squares Line
6641
n
iix
2261
n
iiy
55.1432211
66454404
2
xxS
73.1374
11
2266018
2
yyS
82.3271
11
22666416914 xyS
Fitting the Least Squares Line
First compute the following three quantities:
Computing Estimate of Slope (), Intercept () and standard deviation (),
288.055.14322
82.3271
xx
xy
S
Sb
756.611
664288.0
11
226
xbya
35.8
2
12
xx
xyyy S
SS
ns
95% Confidence Limits for slope :
t.025 = 2.262 critical value for the t-distribution with 9 degrees of freedom
xxS
st ˆ
2/
0.0706 to 0.3862
8.350.288 2.262
1432255
95% Confidence Limits for intercept :
1
ˆ2
2/xxS
x
nst
-4.34 to 17.85
t.025 = 2.262 critical value for the t-distribution with 9 degrees of freedom
2664 111
6.756 2.262 8.35 11 1432255
Iceland
NorwaySweden
DenmarkCanada
Australia
HollandSwitzerland
Great Britain
Finland
USA
0
5
10
15
20
25
30
35
40
45
50
0 20 40 60 80 100 120 140
Per capita consumption of cigarettes
deat
h ra
tes
from
lung
can
cer
(195
0)
Y = 6.756 + (0.228)X
95% confidence Limits for slope 0.0706 to 0.3862
95% confidence Limits for intercept -4.34 to 17.85
Testing the positive slope
The test statistic is:
0 : 0 vs : 0 AH H
0
xx
bt
sS
The Critical Region
Reject
0 : 0 in favour of : 0 AH H
0.05
0if =1.833
xx
bt t
sS
df = 11 – 2 = 9
A one tailed test
and conclude
0 : 0 H
0Since
xx
bt
sS
0.28841.3 1.833
8.351432255
we reject
: 0 AH
Confidence Limits for Points on the Regression Line
• The intercept is a specific point on the regression line.
• It is the y – coordinate of the point on the regression line when x = 0.
• It is the predicted value of y when x = 0.• We may also be interested in other points on the
regression line. e.g. when x = x0
• In this case the y – coordinate of the point on the regression line when x = x0 is + x0
x0
+ x0
y = + x
(1- )100% Confidence Limits for + x0 :
1 20
2/0xxS
xx
nstbxa
t/2 is the /2 critical value for the t-distribution with n - 2 degrees of freedom
Prediction Limits for new values of the Dependent variable y
• An important application of the regression line is prediction.
• Knowing the value of x (x0) what is the value of y?
• The predicted value of y when x = x0 is:
• This in turn can be estimated by:.
ˆ 0xy
00 ˆˆˆ bxaxy
The predictor
• Gives only a single value for y. • A more appropriate piece of information would
be a range of values.• A range of values that has a fixed probability of
capturing the value for y.• A (1- )100% prediction interval for y.
00 ˆˆˆ bxaxy
(1- )100% Prediction Limits for y when x = x0:
11
20
2/0xxS
xx
nstbxa
t/2 is the /2 critical value for the t-distribution with n - 2 degrees of freedom
Example
In this example we are studying building fires in a city and interested in the relationship between:
1. X = the distance of the closest fire hall and the building that puts out the alarm
and
2. Y = cost of the damage (1000$)
The data was collected on n = 15 fires.
The DataFire Distance Damage
1 3.4 26.22 1.8 17.83 4.6 31.34 2.3 23.15 3.1 27.56 5.5 36.07 0.7 14.18 3.0 22.39 2.6 19.610 4.3 31.311 2.1 24.012 1.1 17.313 6.1 43.214 4.8 36.415 3.8 26.1
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
50.0
0.0 2.0 4.0 6.0 8.0
Distance (miles)
Dam
age
(100
0$)
Scatter Plot
Computations
Fire Distance Damage
1 3.4 26.22 1.8 17.83 4.6 31.34 2.3 23.15 3.1 27.56 5.5 36.07 0.7 14.18 3.0 22.39 2.6 19.6
10 4.3 31.311 2.1 24.012 1.1 17.313 6.1 43.214 4.8 36.415 3.8 26.1
2.491
n
iix
2.3961
n
iiy
16.1961
2
n
iix
5.113761
2
n
iiy
65.14701
n
iii yx
Computations Continued
28.3152.491
n
xx
n
ii
4133.26152.3961
n
yy
n
ii
Computations Continued
784.34152.4916.196
2
2
1
1
2
n
xxS
n
iin
iixx
517.911152.3965.11376
2
2
1
1
2
n
yyS
n
iin
iiyy
n
yxyxS
n
ii
n
iin
iiixy
11
1
114.171152.3962.4965.1470
Computations Continued
92.4784.34
114.171ˆ xx
xy
S
Sb
28.1028.3919.44133.26ˆ xbya
2
2
n
SS
Ss xx
xyyy
316.213
784.34114.171517.911
2
95% Confidence Limits for slope :
t.025 = 2.160 critical value for the t-distribution with 13 degrees of freedom
xxS
st ˆ
2/
4.07 to 5.77
95% Confidence Limits for intercept :
1
ˆ2
2/xxS
x
nst
7.21 to 13.35
t.025 = 2.160 critical value for the t-distribution with 13 degrees of freedom
0.0
10.0
20.0
30.0
40.0
50.0
60.0
0.0 2.0 4.0 6.0 8.0
Distance (miles)
Dam
age
(100
0$)
Least Squares Line
y=4.92x+10.28
(1- )100% Confidence Limits for + x0 :
1 20
2/0xxS
xx
nstbxa
t/2 is the /2 critical value for the t-distribution with n - 2 degrees of freedom
95% Confidence Limits for + x0 :
x 0 lower upper
1 12.87 17.522 18.43 21.803 23.72 26.354 28.53 31.385 32.93 36.826 37.15 42.44
0.0
10.0
20.0
30.0
40.0
50.0
60.0
0.0 2.0 4.0 6.0 8.0
Distance (miles)
Dam
age
(100
0$)
95% Confidence Limits for + x0
Confidence limits
(1- )100% Prediction Limits for y when x = x0:
11
20
2/0xxS
xx
nstbxa
t/2 is the /2 critical value for the t-distribution with n - 2 degrees of freedom
95% Prediction Limits for y when x = x0
x 0 lower upper
1 9.68 20.712 14.84 25.403 19.86 30.214 24.75 35.165 29.51 40.246 34.13 45.45
0.0
10.0
20.0
30.0
40.0
50.0
60.0
0.0 2.0 4.0 6.0 8.0
Distance (miles)
Dam
age
(100
0$)
95% Prediction Limits for y when x =x0
Prediction limits
Linear RegressionSummary
Hypothesis testing and Estimation
(1 – )100% Confidence Limits for slope :
t/2 critical value for the t-distribution with n – 2 degrees of freedom
xxS
st ˆ
2/
Testing the slope
The test statistic is:
0 0 0: vs : AH H
0
xx
bt
sS
- has a t distribution with df = n – 2 if H0 is true.
(1 – )100% Confidence Limits for intercept :
t/2 critical value for the t-distribution with n – 2 degrees of freedom
1
ˆ2
2/xxS
x
nst
Testing the intercept
The test statistic is:
0 0 0: vs : AH H
- has a t distribution with df = n – 2 if H0 is true.
0
2
2
1
1
n
ii
at
xs
n x x
(1- )100% Confidence Limits for + x0 :
1 20
2/0xxS
xx
nstbxa
t/2 is the /2 critical value for the t-distribution with n - 2 degrees of freedom
(1- )100% Prediction Limits for y when x = x0:
11
20
2/0xxS
xx
nstbxa
t/2 is the /2 critical value for the t-distribution with n - 2 degrees of freedom
Correlation
The statistic:
Definition
n
ii
n
ii
n
iii
yyxx
xy
yyxx
yyxx
SS
Sr
1
2
1
2
1
is called Pearsons correlation coefficient
1. -1 ≤ r ≤ 1, |r| ≤ 1, r2 ≤ 1
2. |r| = 1 (r = +1 or -1) if the points
(x1, y1), (x2, y2), …, (xn, yn) lie along a straight line. (positive slope for +1, negative slope for -1)
Properties
The test for independence (zero correlation)
The test statistic:
22
1
rt n
r
Reject H0 if |t| > ta/2 (df = n – 2)
H0: X and Y are independent
HA: X and Y are correlated
The Critical region
This is a two-tailed critical region, the critical region could also be one-tailed
Example
In this example we are studying building fires in a city and interested in the relationship between:
1. X = the distance of the closest fire hall and the building that puts out the alarm
and
2. Y = cost of the damage (1000$)
The data was collected on n = 15 fires.
The DataFire Distance Damage
1 3.4 26.22 1.8 17.83 4.6 31.34 2.3 23.15 3.1 27.56 5.5 36.07 0.7 14.18 3.0 22.39 2.6 19.610 4.3 31.311 2.1 24.012 1.1 17.313 6.1 43.214 4.8 36.415 3.8 26.1
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
50.0
0.0 2.0 4.0 6.0 8.0
Distance (miles)
Dam
age
(100
0$)
Scatter Plot
Computations
Fire Distance Damage
1 3.4 26.22 1.8 17.83 4.6 31.34 2.3 23.15 3.1 27.56 5.5 36.07 0.7 14.18 3.0 22.39 2.6 19.6
10 4.3 31.311 2.1 24.012 1.1 17.313 6.1 43.214 4.8 36.415 3.8 26.1
2.491
n
iix
2.3961
n
iiy
16.1961
2
n
iix
5.113761
2
n
iiy
65.14701
n
iii yx
Computations Continued
28.3152.491
n
xx
n
ii
4133.26152.3961
n
yy
n
ii
Computations Continued
784.34152.4916.196
2
2
1
1
2
n
xxS
n
iin
iixx
517.911152.3965.11376
2
2
1
1
2
n
yyS
n
iin
iiyy
n
yxyxS
n
ii
n
iin
iiixy
11
1
114.171152.3962.4965.1470
The correlation coefficient
171.1140.961
34.784 911.517xy
xx yy
Sr
S S
The test for independence (zero correlation)
The test statistic:
2 2
0.9612 13 12.525
1 1 0.961
rt n
r
We reject H0: independence, if |t| > t0.025 = 2.160
H0: independence, is rejected
Relationship between Regression and Correlation
Recall xy
xx yy
Sr
S S
Also
ˆ xy yy xy yy y
xx xx xx xxx yy
S S S S sr r
S S S sS S
since and 1 1
yyxxx y
SSs s
n n
Thus the slope of the least squares line is simply the ratio of the standard deviations × the correlation coefficient
The test for independence (zero correlation)
Uses the test statistic:
22
1
rt n
r
H0: X and Y are independent
HA: X and Y are correlated
Note: andˆ yy
xx
Sr
S ˆxx
yy
Sr
S
1. The test for independence (zero correlation)H0: X and Y are independent
HA: X and Y are correlated
are equivalent
The two tests
2. The test for zero slopeH0: = 0.
HA: ≠ 0
1. the test statistic for independence:
22
1
rt n
r
2 2 2 2
1 1
xy xy
xx yy xx
xy xyyy
xx yy xx yy
S S
S S St n n
S SS
S S S S
Thus
2
ˆ
12
the same statistic for testing for slope.
xy
xx
xyyy xx
xxxx
S
SsS
S n SSS
zero
Regression (in general)
In many experiments we would have collected data on a single variable Y (the dependent variable ) and on p (say) other variables X1, X2, X3, ... , Xp (the independent variables). One is interested in determining a model that describes the relationship between Y (the response (dependent) variable) and X1, X2, …, Xp (the predictor (independent) variables.
This model can be used for– Prediction– Controlling Y by manipulating X1, X2, …, Xp
The Model:is an equation of the form
Y = f(X1, X2,... ,Xp | 1, 2, ... , q) +
where 1, 2, ... , q are unknown parameters of the function f and is a random disturbance (usually assumed to have a normal distribution with mean 0 and standard deviation .
0
20
40
60
80
100
120
140
160
40 60 80 100 120 140
Examples:
1. Y = Blood Pressure, X = age
The model
Y = + X + thus 1 = and 2 = .
This model is called:
the simple Linear Regression Model
Y = + X
8
8.5
9
9.5
10
10.5
11
11.5
12
12.5
1930 1940 1950 1960 1970 1980 1990 2000 2010
2. Y = average of five best times for running the 100m, X = the year
The model
Y = e-X + thus 1 = 2 = and 2 =
.
This model is called:
the exponential Regression Model
Y = e-X +
2. Y = gas mileage ( mpg) of a car brand
X1 = engine size
X2 = horsepower
X3 = weight
The model
Y = 0 + 1 X1 + 2 X2 + 3 X3 + .
This model is called:
the Multiple Linear Regression Model
The Multiple Linear Regression Model
In Multiple Linear Regression we assume the following model
Y = 0 + 1 X1 + 2 X2 + ... + p Xp +
This model is called the Multiple Linear Regression Model.
Again are unknown parameters of the model and where0, 1, 2, ... , p are unknown parameters and
is a random disturbance assumed to have a normal distribution with mean 0 and standard deviation .
The importance of the Linear model
1. It is the simplest form of a model in which each dependent variable has some effect on the independent variable Y. – When fitting models to data one tries to find the
simplest form of a model that still adequately describes the relationship between the dependent variable and the independent variables.
– The linear model is sometimes the first model to be fitted and only abandoned if it turns out to be inadequate.
2. In many instance a linear model is the most appropriate model to describe the dependence relationship between the dependent variable and the independent variables.
– This will be true if the dependent variable increases at a constant rate as any or the independent variables is increased while holding the other independent variables constant.
3. Many non-Linear models can be Linearized (put into the form of a Linear model by appropriately transformation the dependent variables and/or any or all of the independent variables.) – This important fact ensures the wide utility of
the Linear model. (i.e. the fact the many non-linear models are linearizable.)
An Example
The following data comes from an experiment that was interested in investigating the source from which corn plants in various soils obtain their phosphorous.
–The concentration of inorganic phosphorous (X1) and
the concentration of organic phosphorous (X2) was
measured in the soil of n = 18 test plots.–In addition the phosphorous content (Y) of corn grown in the soil was also measured. The data is displayed below:
InorganicPhosphorous
X1
OrganicPhosphorous
X2
Plant Available
PhosphorousY
InorganicPhosphorous
X1
OrganicPhosphorous
X2
Plant Available
Phosphorous
Y
0.4 53 64 12.6 58 51
0.4 23 60 10.9 37 76
3.1 19 71 23.1 46 96
0.6 34 61 23.1 50 77
4.7 24 54 21.6 44 93
1.7 65 77 23.1 56 95
9.4 44 81 1.9 36 54
10.1 31 93 26.8 58 168
11.6 29 93 29.9 51 99
Coefficients
Intercept 56.2510241 (0)
X1 1.78977412 (1)
X2 0.08664925 (2)
Equation:
Y = 56.2510241 + 1.78977412 X1 + 0.08664925 X2
The Multiple Linear Regression Model
In Multiple Linear Regression we assume the following model
Y = 0 + 1 X1 + 2 X2 + ... + p Xp +
This model is called the Multiple Linear Regression Model.
Again are unknown parameters of the model and where0, 1, 2, ... , p are unknown parameters and
is a random disturbance assumed to have a normal distribution with mean 0 and standard deviation .
Summary of the Statistics used in
Multiple Regression
The Least Squares Estimates:
0 1 2, , , , ,p
2
1
ˆn
i ii
RSS y y
2
0 1 1 2 21
n
i i i p pii
y x x x
- the values that minimize
The Analysis of Variance Table Entries
a) Adjusted Total Sum of Squares (SSTotal)
b) Residual Sum of Squares (SSError)
c) Regression Sum of Squares (SSReg)
Note:
i.e. SSTotal = SSReg +SSError
SSTotal n
i1
yi y_2. d.f. n 1
RSS SSError n
i1
yi yi2. d.f. n p 1
SSReg SS1,2, . . . , p n
i1
yi y_2. d.f. p
n
i1
yi y_2
n
i1
yi y_2
n
i1
yi yi 2 .
The Analysis of Variance Table
Source Sum of Squares d.f. Mean Square F
Regression SSReg p SSReg/p = MSReg MSReg/s2
Error SSError n-p-1 SSError/(n-p-1) =MSError = s2
Total SSTotal n-1
Uses: 1. To estimate 2 (the error variance).
- Use s2 = MSError to estimate 2.
2. To test the Hypothesis
H0: 1 = 2= ... = p = 0.
Use the test statistic 2
Reg RegErrorF MS MS MS s
Reg 1ErrorSS p SS n p
- Reject H0 if F > F(p,n-p-1).
3. To compute other statistics that are useful in describing the relationship between Y (the dependent variable) and X1, X2, ... ,Xp (the independent variables).a) R2 = the coefficient of determination
= SSReg/SSTotal
=
= the proportion of variance in Y explained by
X1, X2, ... ,Xp
1 - R2 = the proportion of variance in Y
that is left unexplained by X1, X2, ... , Xp
= SSError/SSTotal.
ˆ y i y 2
i1
n
y i y 2
i1
n
b) Ra2 = "R2 adjusted" for degrees of freedom.
= 1 -[the proportion of variance in Y that is left unexplained by X1, X2,... , Xp adjusted for
d.f.]1 Error TotalMS MS
11
1Error
Total
SS n p
SS n
11
1Error
Total
n SS
n p SS
211 1
1
nR
n p
c) R=R2 = the Multiple correlation coefficient of Y with X1, X2, ... ,Xp
=
= the maximum correlation between Y and a linear combination of X1, X2, ... ,Xp
Comment: The statistics F, R2, Ra2 and R are
equivalent statistics.
SSRe g
SSTotal
Using Statistical Packages
To perform Multiple Regression
Using SPSS
Note: The use of another statistical package such as Minitab is similar to using SPSS
After starting the SSPS program the following dialogue box appears:
If you select Opening an existing file and press OK the following dialogue box appears
The following dialogue box appears:
If the variable names are in the file ask it to read the names. If you do not specify the Range the program will identify the Range:
Once you “click OK”, two windows will appear
One that will contain the output:
The other containing the data:
To perform any statistical Analysis select the Analyze menu:
Then select Regression and Linear.
The following Regression dialogue box appears
Select the Dependent variable Y.
Select the Independent variables X1, X2, etc.
If you select the Method - Enter.
All variables will be put into the equation.
There are also several other methods that can be used :
1. Forward selection
2. Backward Elimination
3. Stepwise Regression
Forward selection
1. This method starts with no variables in the equation
2. Carries out statistical tests on variables not in the equation to see which have a significant effect on the dependent variable.
3. Adds the most significant.
4. Continues until all variables not in the equation have no significant effect on the dependent variable.
Backward Elimination
1. This method starts with all variables in the equation
2. Carries out statistical tests on variables in the equation to see which have no significant effect on the dependent variable.
3. Deletes the least significant.
4. Continues until all variables in the equation have a significant effect on the dependent variable.
Stepwise Regression (uses both forward and backward techniques)
1. This method starts with no variables in the equation
2. Carries out statistical tests on variables not in the equation to see which have a significant effect on the dependent variable.
3. It then adds the most significant.
4. After a variable is added it checks to see if any variables added earlier can now be deleted.
5. Continues until all variables not in the equation have no significant effect on the dependent variable.
All of these methods are procedures for attempting to find the best equation
The best equation is the equation that is the simplest (not containing variables that are not important) yet adequate (containing variables that are important)
Once the dependent variable, the independent variables and the Method have been selected if you press OK, the Analysis will be performed.
The output will contain the following table
Model Summary
.822a .676 .673 4.46Model1
R R SquareAdjustedR Square
Std. Errorof the
Estimate
Predictors: (Constant), WEIGHT, HORSE, ENGINEa.
R2 and R2 adjusted measures the proportion of variance in Y that is explained by X1, X2, X3, etc (67.6% and 67.3%)
R is the Multiple correlation coefficient (the maximum correlation between Y and a linear combination of X1, X2, X3, etc)
The next table is the Analysis of Variance Table
The F test is testing if the regression coefficients of the predictor variables are all zero. Namely none of the independent variables X1, X2, X3, etc have any effect on Y
ANOVAb
16098.158 3 5366.053 269.664 .000a
7720.836 388 19.899
23818.993 391
Regression
Residual
Total
Model1
Sum ofSquares df
MeanSquare F Sig.
Predictors: (Constant), WEIGHT, HORSE, ENGINEa.
Dependent Variable: MPGb.
The final table in the output
Gives the estimates of the regression coefficients, there standard error and the t test for testing if they are zeroNote: Engine size has no significant effect on Mileage
Coefficientsa
44.015 1.272 34.597 .000
-5.53E-03 .007 -.074 -.786 .432
-5.56E-02 .013 -.273 -4.153 .000
-4.62E-03 .001 -.504 -6.186 .000
(Constant)
ENGINE
HORSE
WEIGHT
Model1
B Std. Error
UnstandardizedCoefficients
Beta
Standardized
Coefficients
t Sig.
Dependent Variable: MPGa.
The estimated equation from the table below:
Is:
Coefficientsa
44.015 1.272 34.597 .000
-5.53E-03 .007 -.074 -.786 .432
-5.56E-02 .013 -.273 -4.153 .000
-4.62E-03 .001 -.504 -6.186 .000
(Constant)
ENGINE
HORSE
WEIGHT
Model1
B Std. Error
UnstandardizedCoefficients
Beta
Standardized
Coefficients
t Sig.
Dependent Variable: MPGa.
5.53 5.56 4.6244.0
1000 100 1000Mileage Engine Horse Weight Error
Note the equation is:
Mileage decreases with:
5.53 5.56 4.6244.0
1000 100 1000Mileage Engine Horse Weight Error
1. With increases in Engine Size (not significant, p = 0.432)With increases in Horsepower (significant, p = 0.000)With increases in Weight (significant, p = 0.000)
Logistic regression
Recall the simple linear regression model:
y = 0 + 1x +
where we are trying to predict a continuous dependent variable y from a continuous independent variable x.
This model can be extended to Multiple linear regression model:
y = 0 + 1x1 + 2x2 + … + + pxp + Here we are trying to predict a continuous dependent variable y from a several continuous dependent variables x1 , x2 , … , xp .
Now suppose the dependent variable y is binary.
It takes on two values “Success” (1) or “Failure” (0)
This is the situation in which Logistic Regression is used
We are interested in predicting a y from a continuous dependent variable x.
Example
We are interested how the success (y) of a new antibiotic cream is curing “acne problems” and how it depends on the amount (x) that is applied daily.
The values of y are 1 (Success) or 0 (Failure).
The values of x range over a continuum
The logisitic Regression ModelLet p denote P[y = 1] = P[Success].
This quantity will increase with the value of x.
The ratio: 1
p
pis called the odds ratio
This quantity will also increase with the value of x, ranging from zero to infinity.
The quantity: ln1
p
p
is called the log odds ratio
Example: odds ratio, log odds ratio
Suppose a die is rolled:Success = “roll a six”, p = 1/6
1 16 6
516 6
1
1 1 5
p
p
The odds ratio
1ln ln ln 0.2 1.69044
1 5
p
p
The log odds ratio
The logisitic Regression Model
i. e. :
0 1
1xp
ep
In terms of the odds ratio
0 1ln1
px
p
Assumes the log odds ratio is linearly related to x.
The logisitic Regression Model
0 1
1xp
ep
or
Solving for p in terms x.
0 1 1xp e p
0 1 0 1x xp pe e
0 1
0 11
x
x
ep
e
0
0.2
0.4
0.6
0.8
1
0 2 4 6 8 10
Interpretation of the parameter 0
(determines the intercept)
p
0
01
e
e
x
0
0.2
0.4
0.6
0.8
1
0 2 4 6 8 10
Interpretation of the parameter 1
(determines when p is 0.50 (along with 0))
p 0 1
0 1
1 1
1 1 1 2
x
x
ep
e
x
00 1
1
0 or x x
when
Also0 1
0 11
x
x
dp d e
dx dx e
0
1
x
when
0 1 0 1 0 1 0 1
0 1
1 1
2
1
1
x x x x
x
e e e e
e
0 1
0 1
1 12 41
x
x
e
e
1
4
is the rate of increase in p with respect to x when p = 0.50
0
0.2
0.4
0.6
0.8
1
0 2 4 6 8 10
Interpretation of the parameter 1
(determines slope when p is 0.50 )
p
x
1slope 4
The data
The data will for each case consist of
1. a value for x, the continuous independent variable
2. a value for y (1 or 0) (Success or Failure)
Total of n = 250 cases
case x y
1 0.8 02 2.3 13 2.5 04 2.8 15 3.5 16 4.4 17 0.5 08 4.5 19 4.4 110 0.9 011 3.3 112 1.1 013 2.5 114 0.3 115 4.5 116 1.8 017 2.4 118 1.6 019 1.9 120 4.6 1
case x y
230 4.7 1231 0.3 0232 1.4 0233 4.5 1234 1.4 1235 4.5 1236 3.9 0237 0.0 0238 4.3 1239 1.0 0240 3.9 1241 1.1 0242 3.4 1243 0.6 0244 1.6 0245 3.9 0246 0.2 0247 2.5 0248 4.1 1249 4.2 1250 4.9 1
Estimation of the parameters
The parameters are estimated by Maximum Likelihood estimation and require a statistical package such as SPSS
Using SPSS to perform Logistic regression
Open the data file:
Choose from the menu:
Analyze -> Regression -> Binary Logistic
The following dialogue box appears
Select the dependent variable (y) and the independent variable (x) (covariate).
Press OK.
Here is the output
The Estimates and their S.E.
The parameter Estimates
SE
X 1.0309 0.1334Constant -2.0475 0.332
1 1.03090 -2.0475
Interpretation of the parameter 0
(determines the intercept)
0
0
-2.0475
-2.0475intercept 0.1143
1 1
e e
e e
Interpretation of the parameter 1
(determines when p is 0.50 (along with 0))
0
1
2.04751.986
1.0309x
Another interpretation of the parameter 1
1
4
is the rate of increase in p with respect to x when p = 0.50
1 1.03090.258
4 4
The dependent variable y is binary.
It takes on two values “Success” (1) or “Failure” (0)
The Logistic Regression Model
We are interested in predicting a y from a continuous dependent variable x.
The logisitic Regression ModelLet p denote P[y = 1] = P[Success].
This quantity will increase with the value of x.
The ratio: 1
p
pis called the odds ratio
This quantity will also increase with the value of x, ranging from zero to infinity.
The quantity: ln1
p
p
is called the log odds ratio
The logisitic Regression Model
i. e. :
0 1
1xp
ep
In terms of the odds ratio
0 1ln1
px
p
Assumes the log odds ratio is linearly related to x.
The logisitic Regression Model
In terms of p
0 1
0 11
x
x
ep
e
0
0.2
0.4
0.6
0.8
1
0 2 4 6 8 10
The graph of p vs x
p 0 1
0 11
x
x
ep
e
x
The Multiple Logistic Regression model
Here we attempt to predict the outcome of a binary response variable Y from several independent variables X1, X2 , … etc
0 1 1ln1 p p
pX X
p
0 1 1
0 1 1 or
1
p p
p p
X X
X X
ep
e
Multiple Logistic Regression an example
In this example we are interested in determining the risk of infants (who were born prematurely) of developing BPD (bronchopulmonary dysplasia)
More specifically we are interested in developing a predictive model which will determine the probability of developing BPD from
X1 = gestational Age and X2 = Birthweight
For n = 223 infants in prenatal ward the following measurements were determined
1. X1 = gestational Age (weeks),
2. X2 = Birth weight (grams) and3. Y = presence of BPD
The datacase Gestational Age Birthweight presence of BMD
1 28.6 1119 12 31.5 1222 03 30.3 1311 14 28.9 1082 05 30.3 1269 06 30.5 1289 07 28.5 1147 08 27.9 1136 19 30 972 0
10 31 1252 011 27.4 818 012 29.4 1275 013 30.8 1231 014 30.4 1112 015 31.1 1353 116 26.7 1067 117 27.4 846 118 28 1013 019 29.3 1055 020 30.4 1226 021 30.2 1237 022 30.2 1287 023 30.1 1215 024 27 929 125 30.3 1159 026 27.4 1046 1
The resultsVariables in the Equation
-.003 .001 4.885 1 .027 .998
-.505 .133 14.458 1 .000 .604
16.858 3.642 21.422 1 .000 2.1E+07
Birthweight
GestationalAge
Constant
Step1
a
B S.E. Wald df Sig. Exp(B)
Variable(s) entered on step 1: Birthweight, GestationalAge.a.
ln 16.858 .003 .5051
pBW GA
p
16.858 .003 .505
1BW GAp
ep
16.858 .003 .505
16.858 .003 .5051
BW GA
BW GA
ep
e
Graph: Showing Risk of BPD vs GA and BrthWt
0
0.2
0.4
0.6
0.8
1
700 900 1100 1300 1500 1700
GA = 27
GA = 28
GA = 29
GA = 30
GA = 31
GA = 32
Non-Parametric Statistics