section 3: simple linear regression · section3: simplelinearregression tengyuanliang,chicagobooth ...
TRANSCRIPT
![Page 1: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/1.jpg)
Section 3: Simple Linear Regression
Tengyuan Liang, Chicago Booth
https://tyliang.github.io/BUS41000/
Suggested Reading:OpenIntro Statistics, Chapters 7Statistics for Business, Part IV
1
![Page 2: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/2.jpg)
Week 5
2
![Page 3: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/3.jpg)
Mini-Lecture 1
Simple Linear Regression
view recording_week5_1
3
![Page 4: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/4.jpg)
Regression: General Introduction
I Regression analysis is the most widely used statistical tool forunderstanding relationships among variables
I It provides a conceptually simple method for investigatingfunctional relationships between one or more factors and anoutcome of interest
I The relationship is expressed in the form of an equation or amodel connecting the response or dependent variable and oneor more explanatory or predictor variable
4
![Page 5: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/5.jpg)
Why?
Straight prediction questions:
I For how much will my house sell?I How many runs per game will the Red Sox score this year?I Will this person like that movie? (Netflix rating system)
Explanation and understanding:
I What is the impact of MBA on income?I How does the returns of a mutual fund relates to the market?I Does Walmart discriminate against women regarding salaries?
5
![Page 6: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/6.jpg)
1st Example: Predicting House Prices
Problem:
I Predict market price based on observed characteristics
Solution:
I Look at property sales data where we know the price and someobserved characteristics.
I Build a decision rule that predicts price as a function of theobserved characteristics.
6
![Page 7: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/7.jpg)
Predicting House Prices
What characteristics do we use?
We have to define the variables of interest and develop a specificquantitative measure of these variables
I Many factors or variables affect the price of a houseI sizeI number of bathsI garage, air conditioning, etcI neighborhood
7
![Page 8: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/8.jpg)
Predicting House Prices
To keep things super simple, let’s focus only on size.
The value that we seek to predict is called thedependent (or output) variable, and we denote this:
I Y = price of house (e.g. thousands of dollars)
The variable that we use to guide prediction is theexplanatory (or input) variable, and this is labelled:
I X = size of house (e.g. thousands of square feet)
8
![Page 9: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/9.jpg)
Predicting House Prices
What does this data look like?
9
![Page 10: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/10.jpg)
Predicting House Prices
It is much more useful to look at a scatterplot
1.0 1.5 2.0 2.5 3.0 3.5
6080
100120140160
size
price
In other words, view the data as points in the X × Y plane.
10
![Page 11: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/11.jpg)
Regression Model
Y = response or outcome variable
X = explanatory or input variables
A linear relationship is written
Y = b0 + b1X
11
![Page 12: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/12.jpg)
Linear Prediction
Appears to be a linear relationship between price and size:As size goes up, price goes up.
The line shown was fit by the “eyeball” method.
12
![Page 13: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/13.jpg)
Linear Prediction
Recall that the equation of a line is:
Y = b0 + b1X
Where b0 is the intercept and b1 is the slope.
The intercept value is in units of Y ($1,000).The slope is in units of Y per units of X ($1,000/1,000 sq ft).
13
![Page 14: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/14.jpg)
Linear Prediction
Y
X
b0
2 1
b1
Y = b0 + b1X
Our “eyeball” line has b0 = 35, b1 = 40.
14
![Page 15: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/15.jpg)
Linear Prediction
We can now predict the price of a house when we know only thesize; just read the value off the line that we’ve drawn.
For example, given a house with of size X = 2.2.
Predicted price Y = 35 + 40(2.2) = 123.
Note: Conversion from 1,000 sq ft to $1,000 is done for us by theslope coefficient (b1)
15
![Page 16: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/16.jpg)
Mini-Lecture 2
Prediction Interval
view recording_week5_2
16
![Page 17: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/17.jpg)
Prediction and the Modeling Goal
There are two things that we want to know:
I What value of Y can we expect for a given X?I How sure are we about this forecast? Or how different could Y
be from what we expect?
Our goal is to measure the accuracy of our forecasts or how muchuncertainty there is in the forecast. One method is to specify arange of Y values that are likely, given an X value.
Prediction Interval: probable range for Y-values given X
17
![Page 18: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/18.jpg)
Prediction and the Modeling Goal
Key Insight: To construct a prediction interval, we will have toassess the likely range of error values corresponding to a Y valuethat has not yet been observed!
We will build a probability model (e.g., normal distribution).
Then we can say something like “with 95% probability the error willbe no less than -$28,000 or larger than $28,000”.
We must also acknowledge that the “fitted’ ’ line may be fooled byparticular realizations of the residuals.
18
![Page 19: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/19.jpg)
The Simple Linear Regression Model
The power of statistical inference comes from the ability to makeprecise statements about the accuracy of the forecasts.
In order to do this we must invest in a probability model.
Simple Linear Regression Model: Y = β0 + β1X + ε
ε ∼ N(0, σ2)
I β0 + β1X represents the “true line’ ’; The part of Y thatdepends on X .
I The error term ε is independent “idiosyncratic noise’ ’; Thepart of Y not associated with X .
19
![Page 20: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/20.jpg)
The Simple Linear Regression ModelY = β0 + β1X + ε
1.6 1.8 2.0 2.2 2.4 2.6
160
180
200
220
240
260
x
y
The conditional distribution for Y given X is Normal:
Y |X = x ∼ N(β0 + β1x , σ2).
20
![Page 21: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/21.jpg)
The Simple Linear Regression Model – Example
You are told (without looking at the data) that
β0 = 40; β1 = 45; σ = 10
and you are asked to predict price of a 1500 square foot house.
What do you know about Y from the model?
Y = 40 + 45(1.5) + ε
= 107.5 + ε
Thus our prediction for price is Y |X = 1.5 ∼ N(107.5, 102)
and a 95% Prediction Interval for Y is 87.5 < Y < 127.5
21
![Page 22: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/22.jpg)
Discussion 1 & 2
Simple Linear Regression: Modeling and Prediction
22
![Page 23: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/23.jpg)
2nd Example: Offensive Performance in Baseball
1. Problems:I Evaluate/compare traditional measures of offensive performanceI Help evaluate the worth of a player
2. Solutions:I Compare prediction rules that forecast runs as a function of
either AVG (batting average), SLG (slugging percentage) orOBP (on base percentage)
23
![Page 24: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/24.jpg)
2nd Example: Offensive Performance in Baseball
24
![Page 25: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/25.jpg)
Baseball Data – Using AVG
Each observation corresponds to a team in MLB. Each quantity isthe average over a season.
I Y = runs per game; X = AVG (average)
LS fit: Runs/Game = -3.93 + 33.57 AVG25
![Page 26: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/26.jpg)
Baseball Data – Using SLG
I Y = runs per gameI X = SLG (slugging percentage)
LS fit: Runs/Game = -2.52 + 17.54 SLG26
![Page 27: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/27.jpg)
Baseball Data – Using OBP
I Y = runs per gameI X = OBP (on base percentage)
LS fit: Runs/Game = -7.78 + 37.46 OBP27
![Page 28: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/28.jpg)
Baseball Data
I What is the best prediction rule?I Let’s compare the predictive ability of each model using the
average squared error
1N
N∑i=1
e2i =
∑Ni=1
(Runsi − Runsi
)2
N
28
![Page 29: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/29.jpg)
Place your Money on OBP!!!
Average Squared ErrorAVG 0.083SLG 0.055OBP 0.026
29
![Page 30: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/30.jpg)
Linear Prediction
Yi = b0 + b1Xi
I b0 is the intercept and b1 is the slopeI We find b0 and b1 using Least Squares
30
![Page 31: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/31.jpg)
Mini-Lecture 3
Estimation with Least Squares
view recording_week5_3
31
![Page 32: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/32.jpg)
Linear Prediction
Can we do better than the eyeball method?
We desire a strategy for estimating the slope and interceptparameters in the model Y = b0 + b1X
A reasonable way to fit a line is to minimize the amount by whichthe fitted value differs from the actual value.
This amount is called the residual.
32
![Page 33: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/33.jpg)
Linear Prediction
What is the “fitted value”?
Yi
Xi
Ŷi
The dots are the observed values and the line represents our fittedvalues given by Yi = b0 + b1X1.
33
![Page 34: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/34.jpg)
Linear Prediction
What is the “residual” for the ith ‘observation’?
Yi
Xi
Ŷi ei = Yi – Ŷi = Residual i
We can write Yi = Yi + (Yi − Yi ) = Yi + ei .
34
![Page 35: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/35.jpg)
Least Squares
Ideally we want to minimize the size of all residuals:
I If they were all zero we would have a perfect line.I Trade-off between moving closer to some points and at the
same time moving away from other points.
The line fitting process:
I Give weights to all of the residuals.I Minimize the “total’ ’ of residuals to get best fit.
Least Squares chooses b0 and b1 to minimize ∑Ni=1 e2
i
N∑i=1
e2i = e2
1+e22+· · ·+e2
N = (Y1−Y1)2+(Y2−Y2)2+· · ·+(YN−YN)2
35
![Page 36: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/36.jpg)
Least Squares
LS chooses a different line from ours:
I b0 = 38.88 and b1 = 35.39I What do b0 and b1 mean again?
LS line
Our line
36
![Page 37: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/37.jpg)
Least Squares – Excel Output
SUMMARY OUTPUT
Regression StatisticsMultiple R 0.909209967R Square 0.826662764Adjusted R Square 0.81332913Standard Error 14.13839732Observations 15
ANOVAdf SS MS F Significance F
Regression 1 12393.10771 12393.10771 61.99831126 2.65987E-06Residual 13 2598.625623 199.8942787Total 14 14991.73333
Coefficients Standard Error t Stat P-value Lower 95%Intercept 38.88468274 9.09390389 4.275906499 0.000902712 19.23849785Size 35.38596255 4.494082942 7.873900638 2.65987E-06 25.67708664
Upper 95%58.5308676345.09483846
37
![Page 38: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/38.jpg)
The Least Squares Criterion
The formulas for b0 and b1 that minimize the least squares criterionare:
b1 = rxy ×sysx
b0 = Y − b1X
where,
I X and Y are the sample mean of X and YI corr(x , y) = rxy is the sample correlationI sx and sy are the sample standard deviation of X and Y
38
![Page 39: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/39.jpg)
CovarianceMeasure the direction and strength of the linear relationship between Y and X
Cov(Y , X) =
∑ni=1 (Yi − Y )(Xi − X)
n − 1
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
−20 −10 0 10 20
−40
−20
020
X
Y
(Yi − Y )(Xi − X) > 0
(Yi − Y )(Xi − X) < 0(Yi − Y )(Xi − X) > 0
(Yi − Y )(Xi − X) < 0
X
Y I sy = 15.98, sx = 9.7I Cov(X ,Y ) = 125.9
How do we interpret that?
39
![Page 40: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/40.jpg)
Correlation
Correlation is the standardized covariance:
corr(X ,Y ) = cov(X ,Y )√s2x s2
y= cov(X ,Y )
sx sy
The correlation is scale invariant and the units of measurementdon’t matter: It is always true that −1 ≤ corr(X ,Y ) ≤ 1.
This gives the direction (- or +) and strength (0→ 1)of the linear relationship between X and Y .
40
![Page 41: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/41.jpg)
Correlation
corr(Y ,X ) = cov(X ,Y )√s2x s2
y= cov(X ,Y )
sx sy= 125.9
15.98× 9.7 = 0.812
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
−20 −10 0 10 20
−40
−20
020
X
Y
(Yi − Y )(Xi − X) > 0
(Yi − Y )(Xi − X) < 0(Yi − Y )(Xi − X) > 0
(Yi − Y )(Xi − X) < 0
X
Y
41
![Page 42: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/42.jpg)
Correlation
-3 -2 -1 0 1 2 3
-3-2
-10
12
3
corr = 1
-3 -2 -1 0 1 2 3
-3-2
-10
12
3
corr = .5
-3 -2 -1 0 1 2 3
-3-2
-10
12
3
corr = .8
-3 -2 -1 0 1 2 3
-3-2
-10
12
3
corr = -.8
42
![Page 43: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/43.jpg)
Correlation
Only measures linear relationships:corr(X ,Y ) = 0 does not mean the variables are not related!
-3 -2 -1 0 1 2
-8-6
-4-2
0
corr = 0.01
0 5 10 15 20
05
1015
20
corr = 0.72
Also be careful with influential observations.
Excel: correl, stdev, . . .
R: cor, sd, . . .43
![Page 44: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/44.jpg)
Discussion 3
Estimation with Least Squares
44
![Page 45: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/45.jpg)
More on Least Squares
From now on, terms “fitted values” (Yi) and “residuals” (ei) referto those obtained from the least squares line.
The fitted values and residuals have some special properties. Letslook at the housing data analysis to figure out what these propertiesare. . .
45
![Page 46: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/46.jpg)
The Fitted Values and X
1.0 1.5 2.0 2.5 3.0 3.5
80100
120
140
160
X
Fitte
d V
alue
s
corr(y.hat, x) = 1
46
![Page 47: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/47.jpg)
The Residuals and X
1.0 1.5 2.0 2.5 3.0 3.5
-20
-10
010
2030
X
Residuals
corr(e, x) = 0
mean(e) = 0
47
![Page 48: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/48.jpg)
Why?
What is the intuition for the relationship between Y and e and X?Lets consider some “crazy” alternative line:
1.0 1.5 2.0 2.5 3.0 3.5
6080
100
120
140
160
X
Y
LS line: 38.9 + 35.4 X
Crazy line: 10 + 50 X
48
![Page 49: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/49.jpg)
Fitted Values and Residuals
This is a bad fit! We are underestimating the value of small housesand overestimating the value of big houses.
1.0 1.5 2.0 2.5 3.0 3.5
-20
-10
010
2030
X
Cra
zy R
esid
uals
corr(e, x) = -0.7mean(e) = 1.8
Clearly, we have left some predictive ability on the table!
49
![Page 50: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/50.jpg)
Summary: LS is the best we can do!!
As long as the correlation between e and X is non-zero, we couldalways adjust our prediction rule to do better.
We need to exploit all of the predictive power in the X values andput this into Y , leaving no “Xness’ ’ in the residuals.
In Summary: Y = Y + e where:
I Y is “made from X ’ ’; corr(X , Y ) = 1.I e is unrelated to X ; corr(X , e) = 0.
50
![Page 51: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/51.jpg)
Week 6
51
![Page 52: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/52.jpg)
Mini-Lecture 4
Variance Decomposition
view recording_week6_1
52
![Page 53: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/53.jpg)
Decomposing the Variance
How well does the least squares line explain variation in Y ?Remember that Y = Y + e
Since Y and e are uncorrelated, i.e. corr(Y , e) = 0,
var(Y ) = var(Y + e) = var(Y ) + var(e)∑ni=1(Yi − Y )2
n − 1 =∑n
i=1(Yi − ¯Y )2
n − 1 +∑n
i=1(ei − e)2
n − 1
Given that e = 0, and ¯Y = Y (why?) we get to:
n∑i=1
(Yi − Y )2 =n∑
i=1(Yi − Y )2 +
n∑i=1
e2i
53
![Page 54: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/54.jpg)
Decomposing the Variance
SSR: Variation in Y explained by the regression line.SSE: Variation in Y that is left unexplained.
SSR = SST ⇒ perfect fit.Be careful of similar acronyms; e.g. SSR for “residual” SS.
54
![Page 55: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/55.jpg)
A Goodness of Fit Measure: R2
The coefficient of determination, denoted by R2,measures goodness of fit:
R2 = SSRSST = 1− SSE
SST
I 0 < R2 < 1.I The closer R2 is to 1, the better the fit.
55
![Page 56: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/56.jpg)
Back to the House Data
!""#$%%&$'()*"$+,Applied Regression AnalysisCarlos M. Carvalho
Back to the House Data
''-''.
''/
R2 = SSRSST = 0.82 = 12395
14991
56
![Page 57: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/57.jpg)
Back to Baseball
Three very similar, related ways to look at a simple linearregression. . . with only one X variable, life is easy!
R2 corr SSEOBP 0.88 0.94 0.79SLG 0.76 0.87 1.64AVG 0.63 0.79 2.49
57
![Page 58: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/58.jpg)
Summary of Simple Linear Regression
The model isYi = β0 + β1Xi + εi
εi ∼ N(0, σ2).
The SLR has 3 basic parameters:
I β0, β1 (linear pattern)
I σ (variation around the line).
Assumptions:
I independence means that knowing εi doesn’t affect your viewsabout εj
I identically distributed means that we are using the samenormal for every εi
58
![Page 59: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/59.jpg)
Estimation for the SLR Model
SLR assumes every observation in the dataset was generated by themodel:
Yi = β0 + β1Xi + εi
This is a model for the conditional distribution of Y given X.
We use Least Squares to estimate β0 and β1:
β1 = b1 = rxy ×sysx
β0 = b0 = Y − b1X
59
![Page 60: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/60.jpg)
Estimation of Error Variance
We estimate σ2 with:
s2 = 1n − 2
n∑i=1
e2i = SSE
n − 2
(2 is the number of regression coefficients; i.e. 2 for β0 and β1).
We have n − 2 degrees of freedom because 2 have been “used up’ ’in the estimation of b0 and b1.
We usually use s =√SSE/(n − 2), in the same units as Y . It’s
also called the regression standard error.
60
![Page 61: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/61.jpg)
Estimation of Error Variance
Where is s in the Excel output?
!""#$%%%&$'()*"$+Applied Regression Analysis Carlos M. Carvalho
Estimation of 2
!,"-"$).$s )/$0,"$123"($4506507
8"9"9:"-$;,"/"<"-$=45$.""$>.0?/*?-*$"--4-@$-"?*$)0$?.$estimated.0?/*?-*$*"<)?0)4/&$ ).$0,"$.0?/*?-*$*"<)?0)4/
.
Remember that whenever you see “standard error’ ’ read it asestimated standard deviation: σ is the standard deviation.
61
![Page 62: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/62.jpg)
Discussion 4
Summary of Simple Linear Regression
62
![Page 63: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/63.jpg)
One Picture Summary of SLRI The plot below has the house data, the fitted regression line
(b0 + b1X ) and ±2 ∗ s. . .I From this picture, what can you tell me about β0, β1 and σ2?
How about b0, b1 and s2?
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
1.0 1.5 2.0 2.5 3.0 3.5
6080
100
120
140
160
size
pric
e
63
![Page 64: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/64.jpg)
Mini-Lecture 5
Inference for Least Squares Estimate:
Sampling Distribution, Confidence Interval, and Testing (P-values)
view recording_week6_2
64
![Page 65: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/65.jpg)
Sampling Distribution of Least Squares Estimates
How much do our estimates depend on the particular randomsample that we happen to observe? Imagine:
I Randomly draw different samples of the same size.I For each sample, compute the estimates b0, b1, and s.
If the estimates don’t vary much from sample to sample, then itdoesn’t matter which sample you happen to observe.
If the estimates do vary a lot, then it matters which sample youhappen to observe.
65
![Page 66: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/66.jpg)
Sampling Distribution of Least Squares Estimates
66
![Page 67: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/67.jpg)
Sampling Distribution of Least Squares Estimates
67
![Page 68: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/68.jpg)
Sampling Distribution of b1
The sampling distribution of b1 describes how estimator b1 = β1varies over different samples with the X values fixed.
It turns out that b1 is normally distributed (approximately):b1 ∼ N(β1, s2
b1).
I b1 is unbiased: E [b1] = β1.I sb1 is the standard error of b1. In general, the standard error is
the standard deviation of an estimate. It determines how closeb1 is to β1.
I This is a number directly available from the regression output.
68
![Page 69: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/69.jpg)
Sampling Distribution of b1
Can we intuit what should be in the formula for sb1?
I How should s figure in the formula?I What about n?I Anything else?
s2b1 = s2∑
(Xi − X )2 = s2
(n − 1)s2x
Three Factors:sample size (n), error variance (s2), and X -spread (sx ).
69
![Page 70: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/70.jpg)
Sampling Distribution of b0
The intercept is also normal and unbiased: b0 ∼ N(β0, s2b0
).
s2b0 = var(b0) = s2
(1n + X 2
(n − 1)s2x
)
What is the intuition here?
70
![Page 71: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/71.jpg)
Confidence Intervals
Since b1 ∼ N(β1, s2b1
), Thus:
I 68% Confidence Interval: b1 ± 1× sb1I 95% Confidence Interval: b1 ± 2× sb1I 99% Confidence Interval: b1 ± 3× sb1
Same thing for b0
I 95% Confidence Interval: b0 ± 2× sb0
The confidence interval provides you with a set of plausible valuesfor the parameters
71
![Page 72: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/72.jpg)
Example: Runs per Game and AVGRegression with all points
SUMMARY OUTPUT
Regression Statistics
Multiple R 0.798496529R Square 0.637596707Adjusted R Square 0.624653732Standard Error 0.298493066Observations 30
ANOVAdf SS MS F Significance F
Regression 1 4.38915033 4.38915 49.26199 1.239E-‐07Residual 28 2.494747094 0.089098Total 29 6.883897424
Coefficients Standard Error t Stat P-‐value Lower 95% Upper 95%
Intercept -‐3.936410446 1.294049995 -‐3.04193 0.005063 -‐6.587152 -‐1.2856692AVG 33.57186945 4.783211061 7.018689 1.24E-‐07 23.773906 43.369833
[b1 − 2× sb1 ; b1 + 2× sb1 ] ≈ [23.77; 43.36]
72
![Page 73: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/73.jpg)
Testing
Suppose we want to assess whether or not β1 equals a proposedvalue β0
1 . This is called hypothesis testing.
Formally we test the null hypothesis:
H0 : β1 = β01
vs. the alternative
H1 : β1 6= β01
73
![Page 74: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/74.jpg)
Testing
That are 2 ways we can think about testing:
1. Building a test statistic. . . the t-stat,
t = b1 − β01
sb1
This quantity measures how many standard deviations theestimate (b1) from the proposed value (β0
1).
If the absolute value of t is greater than 2, we need to worry(why?). . . we reject the hypothesis.
74
![Page 75: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/75.jpg)
Testing
2. Looking at the confidence interval. If the proposed value isoutside the confidence interval you reject the hypothesis.
Notice that this is equivalent to the t-stat. An absolute value for tgreater than 2 implies that the proposed value is outside theconfidence interval. . . therefore reject.
This is my preferred approach for the testing problem. You can’t gowrong by using the confidence interval!
75
![Page 76: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/76.jpg)
Discussion 5
Inference for Least Squares Estimate:
Sampling Distribution, Confidence Interval, and Testing (P-values)
76
![Page 77: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/77.jpg)
P-values
I The p-value provides a measure of how weird your estimate isif the null hypothesis is true
I Small p-values are evidence against the null hypothesisI In the AVG vs. R/G example. . . H0 : β1 = 0. How weird is our
estimate of b1 = 33.57?I Remember: b1 ∼ N(β1, s2
b1). . . If the null was true (β1 = 0),
b1 ∼ N(0, s2b1
)
77
![Page 78: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/78.jpg)
P-values
I Where is 33.57 in the picture below?
−40 −20 0 20 40
0.00
0.02
0.04
0.06
0.08
b1 (if β1=0)The p-value is the probability of seeing b1 equal or greater than33.57 in absolute terms. Here, p-value=0.000000124!!
Small p-value = bad null78
![Page 79: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/79.jpg)
P-values
I H0 : β1 = 0. . . p-value = 1.24E-07. . . reject!SUMMARY OUTPUT
Regression Statistics
Multiple R 0.798496529R Square 0.637596707Adjusted R Square 0.624653732Standard Error 0.298493066Observations 30
ANOVAdf SS MS F Significance F
Regression 1 4.38915033 4.38915 49.26199 1.239E-‐07Residual 28 2.494747094 0.089098Total 29 6.883897424
Coefficients Standard Error t Stat P-‐value Lower 95% Upper 95%
Intercept -‐3.936410446 1.294049995 -‐3.04193 0.005063 -‐6.587152 -‐1.2856692AVG 33.57186945 4.783211061 7.018689 1.24E-‐07 23.773906 43.369833
79
![Page 80: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/80.jpg)
P-values
I How about H0 : β0 = 0? How weird is b0 = −3.936?
−10 −5 0 5 10
0.00
0.05
0.10
0.15
0.20
0.25
0.30
b0 (if β0=0)The p-value (the probability of seeing b0 equal or greater than-3.936 in absolute terms) is 0.005.
Small p-value = bad null80
![Page 81: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/81.jpg)
P-values
I H0 : β0 = 0. . . p-value = 0.005. . . we still reject, but not withthe same strength.
SUMMARY OUTPUT
Regression Statistics
Multiple R 0.798496529R Square 0.637596707Adjusted R Square 0.624653732Standard Error 0.298493066Observations 30
ANOVAdf SS MS F Significance F
Regression 1 4.38915033 4.38915 49.26199 1.239E-‐07Residual 28 2.494747094 0.089098Total 29 6.883897424
Coefficients Standard Error t Stat P-‐value Lower 95% Upper 95%
Intercept -‐3.936410446 1.294049995 -‐3.04193 0.005063 -‐6.587152 -‐1.2856692AVG 33.57186945 4.783211061 7.018689 1.24E-‐07 23.773906 43.369833
81
![Page 82: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/82.jpg)
Testing – Summary
I Large t or small p-value mean the same thing. . .
I p-value < 0.05 is equivalent to a t-stat > 2 in absolute value
I Small p-value means something weird happen if the nullhypothesis was true. . .
I Bottom line, small p-value → REJECT! Large t → REJECT!
I But remember, always look at the confidence interveal!
82
![Page 83: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/83.jpg)
Example: Mutual FundsLet’s investigate the performance of the Windsor Fund, anaggressive large cap fund by Vanguard. . .
!""#$%%&$'()*"$+,Applied Regression AnalysisCarlos M. Carvalho
Another Example of Conditional Distributions
-"./0$(11#$2.$2$032.."45426$17$.8"$*2.29$
The plot shows monthly returns for Windsor vs. the S&P50083
![Page 84: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/84.jpg)
Example: Mutual Funds
Consider a CAPM regression for the Windsor mutual fund.
rw = β0 + β1rsp500 + ε
Let’s first test β1 = 0
H0 : β1 = 0. Is the Windsor fund related to the market?
H1 : β1 6= 0
84
![Page 85: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/85.jpg)
Example: Mutual Funds
!""#$%%%&$'()*"$+,Applied Regression Analysis Carlos M. Carvalho
-"./(($01"$!)2*345$5"65"33)42$72$8$,9:;<
b! sb! bsb!
!
Hypothesis Testing – Windsor Fund Example
I t = 32.10. . . reject β1 = 0!!I the 95% confidence interval is [0.87; 0.99]. . . again, reject!!
85
![Page 86: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/86.jpg)
Example: Mutual Funds
Now let’s test β1 = 1. What does that mean?
H0 : β1 = 1 Windsor is as risky as the market.
H1 : β1 6= 1 and Windsor softens or exaggerates market moves.
We are asking whether or not Windsor moves in a different way thanthe market (e.g., is it more conservative?).
86
![Page 87: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/87.jpg)
Example: Mutual Funds
!""#$%%%&$'()*"$+,Applied Regression Analysis Carlos M. Carvalho
-"./(($01"$!)2*345$5"65"33)42$72$8$,9:;<
b! sb! bsb!
!
Hypothesis Testing – Windsor Fund Example
I t = b1−1sb1
= −0.06430.0291 = −2.205. . . reject.
I the 95% confidence interval is [0.87; 0.99]. . . again, reject,but...
87
![Page 88: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/88.jpg)
Mini-Lecture 6
Forecasting
view recording_week6_3
88
![Page 89: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/89.jpg)
Forecasting
The conditional forecasting problem: Given covariate Xf and sampledata {Xi ,Yi}ni=1, predict the “future’ ’ observation yf .
The solution is to use our LS fitted value: Yf = b0 + b1Xf .
This is the easy bit. The hard (and very important!) part offorecasting is assessing uncertainty about our predictions.
89
![Page 90: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/90.jpg)
Forecasting
Just remember that you are uncertain about b0 and b1! As apractical matter if the confidence intervals are big you should becareful!! Some statistical software will give you a larger (andcorrect) predictive interval.
A large predictive error variance (high uncertainty) comes from
I Large s (i.e., large ε’s).
I Small n (not enough data).
I Small sx (not enough observed spread in covariates).
I Large difference between Xf and X .
90
![Page 91: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/91.jpg)
Forecasting
●
●
●
●●
●
●
●
●●
●
●
●
●
●
0 1 2 3 4 5 6
050
100
150
200
250
300
Size
Pric
e
I Red lines: prediction intervalsI Green lines: “plug-in’ ’prediction intervals
91
![Page 92: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/92.jpg)
House Data – one more time!
I R2 = 82%I Great R2, we are happy using this model to predict house
prices, right?
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
1.0 1.5 2.0 2.5 3.0 3.5
6080
100
120
140
160
size
pric
e
92
![Page 93: Section 3: Simple Linear Regression · Section3: SimpleLinearRegression TengyuanLiang,ChicagoBooth SuggestedReading: OpenIntroStatistics,Chapters7 StatisticsforBusiness](https://reader034.vdocuments.site/reader034/viewer/2022050503/5f950ec24d15581f0479e4d5/html5/thumbnails/93.jpg)
House Data – one more time!I But, s = 14 leading to a predictive interval width of about
US$60,000!! How do you feel about the model now?
I As a practical matter, s is a much more relevant quantity thanR2. Once again, intervals are your friend!
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
1.0 1.5 2.0 2.5 3.0 3.5
6080
100
120
140
160
size
pric
e
93