1Building the Regression Model II:

DiagnosticsCHAPTER 10APPLIED LINEAR STATISTICAL MODELS (NETTER)

MEHDI SHAYEGANI M.SHGN@YAHOO.COM

2Building the Regression Model II: Diagnostics We also described the plotting of residuals against predictor

variables not yet in the regression model to determine whether it would be helpful to add one or more of these variables to the model. Added-variable plots provide graphic information about the

marginal importance about predictor variable X.In addition,

these plots can at times be useful for identifying the nature of the marginal relation for a predictor variable in the regression model.

3Added variable plots: Added variable plots or partial regression plot: 1.shows the marginal importance of this variable in reducing the residual variability 2 may provide information about the nature of the marginal

regression relation for the predictor variable Xk under consideration for possible inclusion in the regression model.

X1contains no additional information useful for predicting Y beyond that contained in X2

linear term in XI may be a helpful addition to the regression model

Curvature effect in XI may be a helpful addition to the regression model

4Multiple regression model with two predictor variables X1 and X2

In the previous plots: X2 is already in the regression model and X1 is under

consideration to be added. Plot A) X1 contains no additional information useful for predicting

Y beyond that contained in X2

Plot B,C) addition of X1 to the regression model may be helpful and suggesting the possible nature of the curvature effect by the pattern shown.

5Exampleannual income of managers

average annual income of managers during the past two years (X1)

a score measuring each manager's risk aversion (X2) amount of life insurance carried (Y) Y = -205.72 + 6.2880 X1 +4.738 X2

6Added-variable plot () = + >>>>> (YI) = - () () = *+ >>>>> (I) = - ()

What's the nature relationship in here?

Not linear relation for X1

least s

quare

s line

through

the or

igin.

Slop b

1=6/2

880

7Example A fit of the first-order regression model yields:Y = -205.72 + 6.2880+ 4.738 in attention to plots : Residual plot shows >> a linear relation for X1 is not appropriate in

the model already containing X2

But what is the nature of this relationship in here?For answer we have to use added-variable plot(b).

Added-variable plot shows >> suggested the curvilinear relation between Y and X1 when X2 is already in the regression model is strongly positive

scatter of the points around the least squares line through the origin with slope bl = 6.2880 is much smaller than is the scatter around the horizontal line e( YIX2) =0 indicating that adding XI to the regression model with a linear relation will substantial reduce the error sum of squares

8Residuals- identifying cases These outlying cases may involve large residuals and often have

dramatic effects on the fitted least squares regression function. A case may be outlying or extreme with respect to its Y value, its

X value(s), or both. Case 1 and 2 in may not be too influential because a number of other cases have similar X or Y values that will keep the fitted regression function from being displaced too far by the outlying case Cases 3 and 4, on the other hand, are likely to be very influential in affecting the fit of the regression function.

outlying with respect to its Y value outlying with

respect to their X values

cases 3 and 4 are also outlying with respect to their Y values, given X.

9Identifying Outlying Y Observations

some cases that are outlying or extreme These outlying cases may involve large residuals and often have

dramatic effects on the fitted least squares regression function When more than two predictor variables are included in the

regression model, however, the identification of outlying cases by simple graphic means becomes difficult

Some univariate outliers may not be extreme in a multivariate regression model, and, conversely

We introduce now two refinements to make the analysis of residuals more effective for identifying outlying

observations

10Outlying Y-Use residuals & hat matrix

Analysis the residuals: Hat matrix the fitted value and for residuals we have e = (I - H)Y variance-covariance matrix of the residuals

These variances and covariances are estimated by using MSE as the estimator of the error variance

Estimated: Variance of residuals ei >>>> Covariance between residuals ei ej >>

ith element on the main diagonal of the hat matrix

11Example-residual with hat matrix N = 4 and two predictor variable Fitted model:

s2{e} = 574.9( I - .3877) = 352.0 We see from last Table, column7(s2{ ei }), that the residuals do not have constant variance and residuals for cases are outlying with Respect to the x variable have smaller variance.

The estimated variance-covariance matrix of the residuals, s"{e} = MSE(I - H)

Fitted values

residuals

Diagonal element of hat

12Deleted Residuals- identifying outlying Y The second refinement to make residuals more effective for detecting

outlying Y observations is to measure the ith residual ei = Yi - Yi when the fitted regression is based on all of the cases except the ith one.

The procedure then is to delete the ith case, fit the regression function to the remaining n - 1 cases, and obtain the point estimate of the expected value when the X levels are those of the ith case, to be denoted by Yi(i)

Deleted residual for the ith case >> Thus, deleted residuals will at times identify outlying Y observations

when ordinary residuals would not identify these. We identify as outlying Y observations those cases whose studentized

deleted residuals are large in absolute value. In addition, we can conduct a formal test by means of the Bonferroni test procedure of whether the case with the largest absolute studentized deleted residual is an outlier.

13Example-deleted residuals we wish to examine whether there are outlying Y

observations for example:XII = 19.5 and XI2=43.1

studentized deleted residual

Test for case 13

largest absolute

studentized deleted

residuals 3 , 8 , 13

14Example deleated residuals

We would like to test whether case 13, which has the largest absolute studentized deleted residual,

case 13 is an outlier resulting from a change in the model? use the Bonferroni simultaneous test procedure with a family

significance level of a = .10

few other outlying cases are influential in determining the fitted regression function because the Bonferroni procedure provides a very conservative test for the presence of an outlier.

15Identifying Outlying X Observations-hat matrix , Leverage Values

The hat matrix also is helpful in directly identifying outlying X observations

The diagonal elements hii of the hat matrix have some useful properties:

hii is a measure of the distance between the X values for the i th case and the means of the X values for all cases. Thus, a large value hii indicates that the ith case is distant from the center of all X observations

The diagonal element hii in this context is called the leverage of the ith case.

If the ith case is outlying in terms of its X observations and therefore has a large leverage value hii.

greater than 2mean hii=2p/n

exceeding .5

existence of a gap between the

leverage values

16Example -x outlying body fat example with two predictor variables-triceps skinfold

thickness (X1) and thigh circumference (X2) Note that the two largest leverage values are h33 = .372 and h15.15

= .333.Both exceed the criterion of twice the mean leverage value,2p/n = 2(3)/20 = .30

both are separated by a substantial gap from the next largest leverage values, h55 = .248 and h11 = .201

Case 15 is outlying for X I Case 3 is outlying in terms of the pattern of multicollinearity

17Identifying Influential Cases-DFFITS, Cook’s Distance- and DFBETAS Measures After identifying cases that are outlying with respect to their Y values

and/or their X values, the next step is to ascertain whether or not these outlying cases are influential.

We shall consider a case to be influential if its exclusion causes major changes in the fitted regression function.

We take up three measures of influence that are widely used in practice, each based on the omission of a single case to measure its influence.

I. Influence on Single Fitted Value-DFFlTSII. Influence on All Fitted Values-Cook's DistanceIII. Influence on the Regression Coefficients-DFBETAS

18Influence on Single Fitted Value- DFFlTS

measure of the influence that case i

OR

the value (DFFITS)for the ith case represents the fitted value Yi increases or decreases with the inclusion of the ith case in fitting the regression model.

we suggest considering a case influential if the absolute value of DFFITS exceeds 1 for small to medium data sets and In for large data sets.

19Example-DFFITS value Body fat example: consider the DFFITS value for case 3, which was identified as outlying with respect to its X values

This value is somewhat larger than our guideline of 1. However, the value is close enough to 1 that the case may not be influential enough to require remedial action.

20Influence on All Fitted Values- Cook's Distance

Cook's distance measure considers the influence of the ith case on all n fitted values.

Cook's distance measure, denoted by Di, is an aggregate influence measure

In matrix term >>

relate Di to the F(p, n - p) distribution and ascertain the corresponding percentile value

If the percentile value is less than about 10 or 20 percent, the i th case has little apparent influence on the fitted values. If, on the other hand, the percentile value is near 50 percent or more, the fitted values obtained with and without the I th case should be considered to differ substantially, implying that the i th case has a major influence on the fit of the regression function.

fitted values when the i th case is deleted

21Example Body fat example two predictor variable: we consider again case 3, which is outlying with regard to its X

values

p = 3 for the model with two predictor variables case 3 clearly has the largest Di value, with the next largest distance measure Dl3 = .212 being substantially smaller. To assess the magnitude of the influence of case 3 (D3 = .490), we refer to the corresponding F distribution, namely, F(p, 17 - p) = F(3, 17).

22Example- Cook's distance Figures: clearly show that one case stands out as most influential (case 3) and that all the other cases are much less influential the size of the plotted points being proportional to Cook's distance measure Di

identifies the most influential case as case 3 but does not provide any information about the magnitude of the residual for this case

assess the magnitude of the influence of case 3 (D3 = .490) F(p, n - p) = F(3, 17) so We find that .490 is the 30.6th percentiles of this

distribution. Hence, it appears that case 3 does influence the regression fit, but the extent of the influence may not be large enough to call for consideration of remedial measures.

residual for the most influential case is large negative

23Influence on the Regression Coefficients-DFBETAS

measure of the influence of the i th case on each regression coefficient bk

is the difference between the estimated regression coefficient bk

based on all n cases and the regression coefficient obtained when the ith case is omitted.

variance of bk is: 2{bk}=2ckk

kth diagonal element of (X'X)-1

regression coefficient obtained when the ith case is omitted

error mean square obtained when the ith case is deleted in fitting the regression model

24DFBETAS The DFBETAS value by; Sign: indicates whether inclusion of a case leads to an increase

or a decrease in the estimated regression coefficient absolute magnitude : shows the size of the difference relative

to the estimated standard deviation of the regression coefficientA large absolute value of (DFBETAS)k(i) is indicative of a large impact

on the ith case on the kth regression coefficientAnd

we recommend considering a case influential if the absolute value of DFBETAS

exceeds 1 for small to medium data sets and for large data sets

We explain this with next example

25Example –DFBETAS value

Body fat example two predictor variable only case that exceeds our guideline of 1 for medium-

size data sets for both b1 and b2

Thus, case 3 is again tagged as potentially influentiaL Again, however, the DFBETAS values do not exceed 1 by very much so that case 3 may not be so influential as to require remedial action.

26Multicollinearity Diagnostics-informal diagnostics

Indications of the presence of serious multicollinearity are given by the following informal diagnostics:

1. Large changes in the estimated regression coefficients when a predictor variable is added or deleted, or when an observation is altered or deleted

2. Non significant results in individual tests on the regression coefficients for important predictor variables.

3. Estimated regression coefficients with an algebraic sign that is the opposite of that expected from theoretical considerations or prior experience.

4. Large coefficients of simple correlation between pairs of predictor variables in the correlation matrix rxx.

5. Wide confidence intervals for the regression on coefficients representing important predictor variables.

27Example- Multicollinearity informal diagnosis

three predictor variables:skinfold thickness (X1), thigh circumference (X2), and midarm circumference (X3)1. predictor variables triceps skinfold thickness and thigh circumference

are highly correlated with each other.2. We also noted large changes in the estimated regression coefficients

and their estimated standard deviations when a variable was added3. Non significant results in individual tests on anticipated important

variables4. estimated negative coefficient when a positive coefficient was

expected.These are suggest serious multicollinearity among the predictor

variables

28Multicollinearity Diagnostics-Variance inflation Factor

A formal method of detecting the presence of multicollinearity that is widely accepted is use of variance inflation factors.

These factors measure how much the variances of the estimated regression coefficients are inflated as compared to when the predictor variables are not linearly related.

variance-covariance matrix of the estimated regression coefficients is:

29Multicollinearity Diagnostics-Variance inflation Factor standardized regression model:

2 is the error term variance for the transformed model variance inflation factor for bk

(VIF)k denote the kth diagonal element of the matrix

transforming the variables by means of the correlation transformation

(VIF)k =1 then  =0 is not linearly related to the other X variables   is the coefficient of multiple

determination when Xk is regressed on the p - 2 other X variables in the model

30Multicollinearity Diagnostics-Variance inflation Factor

The largest VIF value among all X variables is often used as an indicator of the severity of multicollinearity

The mean of the VIF values also provides information about the severity of the multicollinearity in terms of how far the estimated standardized regression coefficients bk are from the true values Bk.

sum of the squared errors: &

effect of multicollinearity on the sum of the squared errors:(mean of the VIF values)

no X variable is linearly related to the others in the regression model

If greater then 1 >>>> serious multicollinearity

31Example -Variance inflation Factor body fat example with three predictor variables

Mean VIF values considerably larger than 1 are indicative of serious multicollinearity problems.

all three VIF values greatly exceed 10, which again indicates that serious multicollinearity problems exist.

Thus, the expected sum of the squared errors in the least squares standardized regression coefficients is nearly 460 times as large as it would be if the X variables were uncorrelated

32Summary in model building Building the regression model:

Model selection Stepwise Methods

Model validation

Collection of new data & Comparison with earlier empirical results & Data Splitting

diagnosticsOutliers influential case multicollinearity interaction effect

Remedial measures

33Example – surgical unit Model selected : lny = + + + Examine the interaction effect with added variable plots for 6 two

factor interaction: To examine interaction effects further, a regression model

containing first-order terms in XI, X2, X3, and X8 was fitted and added-variable plots for the six two-factor interaction terms.

these plots did not suggest that any strong two-variable interactions are present and need to be included in the model.

The residual plots shows no evidence of serious departures from the model.

use a residual plot and an added-variable plot to study graphically the strength of the marginal relationship between X5 and the response when X1, X2, X3, and X8 are already in the model.

34Example- pg413 Multicollinearity was studied by calculating the variance inflation factors .

Multicolinarity among 4 predictor not problem(all >1) plots of four key regression diagnostics: 1.deleted studentized residuals 2.the leverage values 3.Cook's

Distances 4. DFFlTS values Case 17 was identified as outlying with regard to its Yvalue according to its

studentized deleted residuals. outlying by more than three standard deviations. Bonferroni test >> not an outlier

identifying outlying X observations, cases 23, 28, 32, 38, 42, and 52 were identified as outlying according to their leverage values with guide 2p/n=0.185

case 17 is the most influential, with Cook's distance D17=.3306 Referring to the F distribution with 5 and 49 degrees of freedom, we note that the Cook's value corresponds to the 11th percentile(bitween 10 to 30).>>>> It thus appears that the influence of case 17 is not large enough to warrant remedial measures,

35

Tank you