Chapter 20

Linear Regression

What if…We believe that an important relation

between two measures exists?For example, we ask 5 people about

their salary and education levelFor each observation we have two

measures, and those two measures came from the same person

What would we “predict”? Does more education mean more salary? Does more salary mean more education? Does more education mean less salary? Does more salary mean less education? Are salary and education related?

RegressionDescriptive vs. Inferential Bivariate data - measurements on two

variables for each observation– Heights (X) and weights (Y)– IQ (X) and SAT(Y) scores – Years of educ. (X) and Annual salary (Y)– Number of Policemen (X) and Number of

crimes (Y) in US cities

Regression How are the two

sets of scores related?

Using a scatterplot we can “look” at the relationship

Constructed by plotting each of the bivariate observations (X, Y)

9 11 13 15 17 19 210

10

20

30

40

50

60

70

Yrs of Education

Annu

al S

alar

y (i

n 10

00's

)

Regression Which one’s X and

which one’s Y? That’s up to you, but… Generally, the X

variable is thought of as the “predictor” variable

We try to predict a Y score given an X score

9 11 13 15 17 19 210

10

20

30

40

50

60

70

Yrs of Education

Annu

al S

alar

y (i

n 10

00's

)

Regression If the scores seem

to “line up,” we call this a “linear relationship”

9 11 13 15 17 19 210

10

20

30

40

50

60

70

Yrs of Education

Annu

al S

alar

y (i

n 10

00's

)

Interpreting Scatterplots If the following

relations hold:low x - high ymid x - mid yhigh x - low y,

“A negative linear relationship”

9 11 13 15 17 19 210

10

20

30

40

50

60

70

Yrs of Education

Annu

al S

alar

y (i

n 10

00's

)

Interpreting Scatterplots If the following

relations hold:low x - low ymid x - mid yhigh x - high

y,

“A positive linear relationship”

2 4 6 80123456789

10

Police per 1000 citizensN

umbe

r of

Cri

mes

(10

00s)

Interpreting Scatterplots

However, there also can be “no relation” also

2 4 6 899

100101102103104105106107

Shoe Size

IQ

Interpreting Scatterplots

Curvelinear

50 55 60 65 70 75100

105

110

115

120

125

130

135

HeightW

eigh

t

Measuring Linear RelationshipsThe first measure of a linear

relationship (not in the book) is COVARIANCE (sXY)

Or

SPXY is known as the “Sum of Products” or the sum of the products of the deviations of X and Y from their means

Easy Calculation

Covariance Interpretation:

– positive = positive linear relationship– negative = negative linear relationship– zero = no relationship

Magnitude (strength of the relationship)?– Uninterpretable– for example, a large covariance does not

necessarily mean strong relationship

But, we can use covariance Which line best fits our

data? Do we just draw one

that looks good? No, we can use

something called “least squares regression” to find the equation of the best-fit line (“Best-fit linear regression”)

9 11 13 15 17 19 2120253035404550556065

Yrs of EducationAn

nual

Sal

ary

(in

1000

's)

Linear EquationsYi = mXi + bm = slopeb = y-intercept

Finding the Slope

Or…

Finding the y-intercept (b)After finding the slope (m), find b using:

Least Squares Criterion

The best line has the property of least squares

The sum of the squared deviations of the points from the line are a minimum

9 11 13 15 17 19 2120253035404550556065

Yrs of EducationAn

nual

Sal

ary

(in

1000

's)

What’s the “least” again?What are we trying to minimize?

– The best fit line will be described by the function Yi = mXi + b

– Thus, for any Xi, we can estimate a corresponding Yi value

– Problem: for some Xi’s we already have Yi’s

– So, let’s call the estimated value (“Y-sub-I-hat”), to differentiate it from the “real” Yi

Least Squares Criterion For example, when

Xi = 15we would estimate that = 44,000

But, we have a “real” Yi value corresponding to Xi =15 (35,000)

9 11 13 15 17 19 2120253035404550556065

Yrs of Education

Annu

al S

alar

y (i

n 10

00's

)When Xi = 15

Our estimatedY value is44,000

A “real”Y valueof 35,000

iY

Minimize this… For every Xi, we have the a value Yi, and an

estimate of Yi ( ) Consider the quantity:

– Which is the deviation of the real score from the estimated score, for any give Xi value

The sum of these deviations will be zero

• But, by squaring those deviations and summing,

• We want the line that makes the above quantity the minimum (the least squares criterion)

• This is also called the sums of squares error or SSE (how much do our estimates “err” from our real values?)

How accurate are our Estimates?Two ways to measure how “good” our

estimates are:– Standard Error of the Estimate– Coefficient of Determination (not covered

in our book, yet)

Standard Error of the Estimate

but, this term is very hard to interpret. (Hurrah, there are better ways to measure the goodness of the fit!)

Coefficient of Determinationcd = r2

Now You:ID INCOME NUMDRK

2001 1 1

2002 6 2

2003 5 8

2004 4 1

2005 6 3

Practice:ID INCOME NUMDRK

XY

2001 1 1

2002 6 2

2003 5 8

2004 4 1

2005 6 3

Σ

n

M

SS(X)

Practice:ID INCOME NUMDRK

XY

2001 1 1 1

2002 6 2 12

2003 5 8 40

2004 4 1 4

2005 6 3 18

Σ 22 15 75

n 5 5

M 4.4 3

SS(X) 17.2 34

Practice:ID INCOME NUMDRK

XY

2001 1 1 1

2002 6 2 12

2003 5 8 40

2004 4 1 4

2005 6 3 18

Σ 22 15 75

n 5 5

M 4.4 3

SS(X) 17.2 34

0 1 2 3 4 5 6 70

1

2

3

4

5

6

7

8

9

f(x) = 0.523255813953 x + 0.697674418605