introduction to predictive modeling with examples

Introduction to Predictive Modeling with Examples

Nationwide Insurance Company, November 2D. A. Dickey

Cool < ------------------------ > Nerdy

“Analytics” = “Statistics”“Predictive Modeling” = “Regression”

Part 1: Simple Linear Regression

If the Life Line is long and deep, then this represents a long life full of vitality and health. A short line, if strong and deep, also shows great vitality in your life and the ability to overcome health problems. However, if the line is short and shallow, then your life may have the tendency to be controlled by others

http://www.ofesite.com/spirit/palm/lines/linelife.htm

Wilson & Mather JAMA 229 (1974)

X=life line length Y=age at death

Result: Predicted Age at Death = 79.24 – 1.367(lifeline) (Is this “real”??? Is this repeatable???)

proc sgplot; scatter Y=age X=line; reg Y=age X=line; run ;

We Use LEAST SQUARES

Squared residuals sum to 9609

Error sum of squares SSq versus slope and intercept (truncated at SSq=9700)

“Best” line is the one that minimizes sum of squared residuals. Best for this sample – is it the true relationship for everyone?SAS PROC REG will compute it. What other lines might be the true line for everyone?? Probably not the purple one. Red one has slope 0 (no effect). Is red line unreasonable? Can we reject H0:slope is 0?

Simulation: Age at Death = 67 + 0(life line) + eError e has normal distribution mean 0 variance 200.Simulate 20 cases with n= 50 bodies each.

NOTE: Regression equations : Age(rep:1) = 80.56253 - 1.345896*line.Age(rep:2) = 61.76292 + 0.745289*line.Age(rep:3) = 72.14366 - 0.546996*line.Age(rep:4) = 95.85143 - 3.087247*line.Age(rep:5) = 67.21784 - 0.144763*line.Age(rep:6) = 71.0178 - 0.332015*line.Age(rep:7) = 54.9211 + 1.541255*line.Age(rep:8) = 69.98573 - 0.472335*line.Age(rep:9) = 85.73131 - 1.240894*line.Age(rep:10) = 59.65101 + 0.548992*line.Age(rep:11) = 59.38712 + 0.995162*line.Age(rep:12) = 72.45697 - 0.649575*line.Age(rep:13) = 78.99126 - 0.866334*line.Age(rep:14) = 45.88373 + 2.283475*line.Age(rep:15) = 59.28049 + 0.790884*line.Age(rep:16) = 73.6395 - 0.814287*line.Age(rep:17) = 70.57868 - 0.799404*line.Age(rep:18) = 72.91134 - 0.821219*line.Age(rep:19) = 55.46755 + 1.238873*line.Age(rep:20) = 63.82712 + 0.776548*line.

Predicted Age at Death = 79.24 – 1.367(lifeline)Would NOT be unusual if there is no true relationship .

Conclusion: Estimated slopes varyStandard deviation of estimated slopes = “Standard error” (estimated)Compute t = (estimate – hypothesized)/standard errorp-value is probability of larger |t| when hypothesis is correct (e.g. 0 slope)p-value is sum of two tail areas.Traditionally p<0.05 hypothesized value is wrong. p>0.05 is inconclusive.

Distribution of tUnder H0

proc reg data=life; model age=line; run;

Parameter Estimates

Parameter StandardVariable DF Estimate Error t Value Pr > |t|Intercept 1 79.23341 14.83229 5.34 <.0001Line 1 -1.36697 1.59782 0.86 0.3965

Area 0.19825Area 0.19825 0.39650

-0.86 0.86

Conclusion: insufficient evidence against the hypothesis of no linear relationship.

H0:H1:

H0: InnocenceH1: Guilt

Beyond reasonable doubt

P<0.05

H0: True slope is 0 (no association)H1: True slope is not 0 P=0.3965

Simulation: Age at Death = 67 + 0(life line) + eError e has normal distribution mean 0 variance 200. WHY?Simulate 20 cases with n= 50 bodies each.

Want estimate of variability around the true line. True variance is Use sums of squared residuals (SS).

Sum of squared residuals from the mean is “SS(total)” 9755Sum of squared residuals around the line is “SS(error)” 9609

(1) SS(total)-SS(error) is SS(model) = 146(2) Variance estimate is SS(error)/(degrees of freedom) = 200(3) SS(model)/SS(total) is R2, i.e. proportion of variablity “explained” by the model.

2

Analysis of Variance

Sum of MeanSource DF Squares Square F Value Pr > FModel 1 146.51753 146.51753 0.73 0.3965Error 48 9608.70247 200.18130Corrected Total 49 9755.22000

Root MSE 14.14854 R-Square 0.0150

Part 2: Multiple Regression

Issues: (1) Testing joint importance versus individual significance

(2) Prediction versus modeling individual effects

(3) Collinearity (correlation among inputs)

Example: Hypothetical company’s sales Y depend on TV advertising X1 and Radio Advertising X2.

Y = b0 + b1X1 + b2X2 +e

Jointly critical (can’t omit both!!)

Two engine plane can still fly if engine #1 failsTwo engine plane can still fly if engine #2 failsNeither is critical individually

Data Sales; length sval $8; length cval $8; input store TV radio sales; (more code)cards; 1 869 868 9089 2 836 820 8290 (more data) 40 969 961 10130

proc g3d data=sales; scatter radio*TV=sales/shape=sval color=cval zmin=8000;run;

TV

Sales

Radio

P2 axis

Conclusion: Can predict well with just TV, just radio, or both!

SAS code: proc reg data=next; model sales = TV radio;


Sum of MeanSource DF Squares Square F Value Pr > FModel 2 32660996 16330498 358.84 <.0001 (Can’t omit both)Error 37 1683844 45509Corrected Total 39 34344840

Root MSE 213.32908 R-Square 0.9510 Explaining 95% of variation in sales

Parameter Estimates

Parameter StandardVariable DF Estimate Error t Value Pr > |t|Intercept 1 531.11390 359.90429 1.48 0.1485TV 1 5.00435 5.01845 1.00 0.3251 (can omit TV)radio 1 4.66752 4.94312 0.94 0.3512 (can omit radio)

Estimated Sales = 531 + 5.0 TV + 4.7 radio with error variance 45509 (standard deviation 213).

TV approximately equal to radio so, approximately

Estimated Sales = 531 + 9.7 TV orEstimated Sales = 531 + 9.7 radio

Setting TV = radio (approximate relationship)

Estimated Sales = 531 + 9.7 TVis this the BEST TV line?

Estimated Sales = 531 + 9.7 radiois this the BEST radio line?

Proc Reg Data=Stores; Model Sales = TV; Model Sales = radio; run;


Sum of MeanSource DF Squares Square F Value Pr > F

Model 1 32620420 32620420 718.84 <.0001Error 38 1724420 45379Corrected Total 39 34344840


Parameter StandardVariable DF Estimate Error t Value Pr > |t|

Intercept 1 478.50829 355.05866 1.35 0.1857TV 1 9.73056 0.36293 26.81 <.0001

********************************************************************************************* Analysis of Variance




Parameter StandardVariable DF Estimate Error t Value Pr > |t|

Intercept 1 612.08604 350.59871 1.75 0.0889radio 1 9.58381 0.35797 26.77 <.0001

Sums of squares capture variation explained by each variableType I: How much when it is added to the model? Type II: How much when all other variables are present

(as if it had been added last)

Parameter Estimates

Parameter StandardVariable DF Estimate Error t Value Pr > |t| Type I SS Type II SS

Intercept 1 531.11390 359.90429 1.48 0.1485 3964160640 99106TV 1 5.00435 5.01845 1.00 0.3251 32620420 45254radio 1 4.66752 4.94312 0.94 0.3512 40576 40576

***********************************************************************************

Parameter Estimates

Parameter StandardVariable DF Estimate Error t Value Pr > |t| Type I SS Type II SS

Intercept 1 531.11390 359.90429 1.48 0.1485 3964160640 99106radio 1 4.66752 4.94312 0.94 0.3512 32615742 40576TV 1 5.00435 5.01845 1.00 0.3251 45254 45254

Summary:

Good predictions given by Sales = 531 + 5.0 x TV + 4.7 x Radio or Sales = 479 + 9.7 x TV orSales = 612 + 9.6 x Radio or

(lots of others)

Why the confusion?The evil Multicollinearity!!

(correlated X’s)

Those Mysterious “Degrees of Freedom” (DF)

First Martian information about average height 0 information about variation.

2nd Martian gives first piece of information (DF) about error variance around mean.

n Martiansn-1 DF for error (variation)

Martian Height

Martian Weight

2 points no information on variation of errors

n points n-2 error DF

How Many Table Legs? (regress Y on X1, X2)

X1

X2error

Fit a plane n-3 (37) error DF (2 “model” DF, n-1=39 “total” DF)

Regress Y on X1 X2 … X7 n-8 error DF (7 “model” DF, n-1 “total” DF)

Sum of MeanSource DF Squares SquareModel 2 32660996 16330498Error 37 1683844 45509Corrected Total 39 34344840

Three legs will all touch the floor.

Fourth leg gives first chance to measure error (first error DF).

Grades vs. IQ and Study Time

Data tests; input IQ Study_Time Grade; IQ_S = IQ*Study_Time; cards; 105 10 75110 12 79120 6 68116 13 85122 16 91130 8 79114 20 98102 15 76 ; Proc reg data=tests; model Grade = IQ; Proc reg data=tests; model Grade = IQ Study_Time;

Parameter StandardVariable DF Estimate Error t Value Pr > |t|Intercept 1 62.57113 48.24164 1.30 0.2423IQ 1 0.16369 0.41877 0.39 0.7094

Parameter StandardVariable DF Estimate Error t Value Pr > |t|Intercept 1 0.73655 16.26280 0.05 0.9656IQ 1 0.47308 0.12998 3.64 0.0149Study_Time 1 2.10344 0.26418 7.96 0.0005

Contrast: TV advertising looses significance when radio is added. IQ gains significance when study time is added.

Model for Grades: Predicted Grade = 0.74 + 0.47 x IQ + 2.10 x Study Time

Question: Does an extra hour of study really deliver 2.10 points for everyone regardless of IQ? Current model only allows this.

“Interaction” model: Predicted Grade = 72.21 - 0.13 x IQ - 4.11 x Study Time + 0.053 x IQ x Study Time = (72.21 - 0.13 x IQ )+( - 4.11 + 0.053 x IQ )x Study Time

IQ = 102 predicts Grade = (72.21-13.26)+(5.41-4.11) x Study Time = 58.95+ 1.30 x Study Time IQ = 122 predicts Grade = (72.21-15.86)+(6.47-4.11) x Study Time = 56.35 + 2.36 x Study Time

proc reg; model Grade = IQ Study_Time IQ_S;

Sum of Mean Source DF Squares Square F Value Pr > F

Model 3 610.81033 203.60344 26.22 0.0043 Error 4 31.06467 7.76617 Corrected Total 7 641.87500

Root MSE 2.78678 R-Square 0.9516 Parameter Standard Variable DF Estimate Error t Value Pr > |t|

Intercept 1 72.20608 54.07278 1.34 0.2527 IQ 1 -0.13117 0.45530 -0.29 0.7876 Study_Time 1 -4.11107 4.52430 -0.91 0.4149 IQ_S 1 0.05307 0.03858 1.38 0.2410

(1) Adding interaction makes everything insignificant (individually) !(2) Do we need to omit insignificant terms until only significant ones remain?(3) Has an acquitted defendant proved his innocence?(4) Common sense trumps statistics!

Slope = 1.30

Slope = 2.36

Part 3: Diagnosing Problems in Regression

Main problems are Multicollinearity (correlation among inputs)Outliers

TV $

Radio $

Principal Component Axis 1: P1

Principal Component Axis 2 P2

Proc Corr; Var TV radio sales;

Pearson Correlation Coefficients, N = 40 Prob > |r| under H0: Rho=0

TV radio sales

TV 1.00000 0.99737 0.97457 <.0001 <.0001

radio 0.99737 1.00000 0.97450 <.0001 <.0001

sales 0.97457 0.97450 1.00000 <.0001 <.0001

TV 1.00000 0.99737 <.0001

radio 0.99737 1.00000 <.0001

Principal Components

(1) Center and scale variables to mean 0 variance 1.(2) Call these X1 (TV) and X2 (radio)(3) n variables total variation is n (n=2 here)(4) Find most variable linear combination P1=__X1+__X2

Variances are 1.9973 out of 2 (along P1 axis) standard deviation and 0.0027 out of 2 (along P2 axis) standard deviationRatio of standard deviations (27.6) is “condition number” large unstable regression.Rule of thumb: Ratio 1 is perfect, >30 problematic. Spread on long axis is 27.6 times that on short axis.

Variance Inflation Factor(1) Regress predictor i on all the others getting r-square: Ri

2

(2) VIF is 1/(1- Ri2 ) for variable i (measures collinearity).

(3) VIF > 10 is a problem.

1.99730.0027

Variance Inflation Factor(1) Regress predictor i on all the others getting r-square: Ri

2

(2) VIF is 1/(1- Ri2 ) for variable i (measures collinearity).

(3) VIF > 10 is a problem.

Example:Proc Reg Data=Sales; Model Sales = TV Radio/VIF collinoint;

Parameter Estimates Parameter Standard VarianceVariable DF Estimate Error t Value Pr > |t| Inflation

Intercept 1 531.11390 359.90429 1.48 0.1485 0TV 1 5.00435 5.01845 1.00 0.3251 190.65722radio 1 4.66752 4.94312 0.94 0.3512 190.65722

Collinearity Diagnostics (intercept adjusted)

Condition --Proportion of Variation- Number Eigenvalue Index TV radio

1 1.99737 1.00000 0.00131 0.00131 2 0.00263 27.57948 0.99869 0.99869

We have a MAJOR problem!

(note: other diagnostics besides VIF and condition number are available)

Another problem: Outliers

Example: Add one point to TV-Radio dataTV 1021, radio 954, Sales 9020 Proc Reg: Model Sales = TV radio/ p r;

Analysis of Variance Sum of MeanSource DF Squares Square F Value Pr > FModel 2 33190059 16595030 314.07 <.0001Error 38 2007865 52839Corrected Total 40 35197924


Parameter Estimates

Parameter StandardVariable DF Estimate Error t Value Pr > |t|Intercept 1 689.01260 382.52628 1.80 0.0796TV 1 -6.28994 2.90505 -2.17 0.0367 ???????radio 1 15.78081 2.86870 5.50 <.0001

Dependent Predicted Std Error Student Cook's Obs Variable Value Residual Residual Residual -2-1 0 1 2 D

39 9277 9430 -153.4358 225.3 -0.681 | *| | 0.006 40 10130 9759 370.5848 226.1 1.639 | |*** | 0.030 41 9020 9322 -301.8727 121.9 -2.476 | ****| | 5.224

TV ‚1200 ˆ ‚ + ‚ ‚ + ‚ + ‚ ++ ‚ +++ ‚ + ‚ + + +1000 ˆ ++ ‚ ++++ ‚ ++++ ‚ + + ‚ ++ ‚ ++ ‚ + ‚ + ‚ ++ 800 ˆ+ Šˆƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒˆƒ 800 1000 1200 radio

P1

P2

Ordinary residual for store 41 not too bad (-300.87)

PRESS residuals(1) Remove store i , Sales Y(i)(2) Fit model to other 40 stores(3) Get model prediction P(i) for store I(4) PRESS residual is Y(i)-P(i)

Store number 41

Regular O and PRESS (dot) residuals

proc reg data=raw; model sales = TV radio; output out=out1 r=r press= press; run;

View Along the P2 Axis

P2 (2nd Principal Component)

Part 4: Classification Variables (dummy variables, indicator variables)

Predicted Accidents = 1181 + 2579 X11 X11 is 1 in November, 0 elsewhere. Interpretation: In November, predict 1181+2579(1) = 3660. In any other month predict 1181 + 2579(0) = 1181. 1181 is average of other months. 2579 is added November effect (vs. average of others)

Model for NC Crashes involving Deer: Proc reg data=deer; model deer = X11; Analysis of Variance

Sum of MeanSource DF Squares Square F Value Pr > FModel 1 30473250 30473250 90.45 <.0001Error 58 19539666 336891Corrected Total 59 50012916


Parameter StandardVariable Label DF Estimate Error t Value Pr > |t|Intercept Intercept 1 1181.09091 78.26421 15.09 <.0001X11 1 2578.50909 271.11519 9.51 <.0001

Looks like December and October need dummies too!Proc reg data=deer; model deer = X10 X11 X12; Analysis of Variance




Parameter StandardVariable Label DF Estimate Error t Value Pr > |t|Intercept Intercept 1 929.40000 39.13997 23.75 <.0001X10 1 1391.20000 123.77145 11.24 <.0001X11 1 2830.20000 123.77145 22.87 <.0001X12 1 1377.40000 123.77145 11.13 <.0001

Average of Jan through Sept. is 929 crashes per month. Add 1391 in October, 2830 in November, 1377 in December.

What the heck – let’s do all but one (need “average of rest” so must leave out at least one)Proc reg data=deer; model deer = X1 X2 … X10 X11; Analysis of Variance



Parameter Estimates

Parameter StandardVariable Label DF Estimate Error t Value Pr > |t|

Intercept Intercept 1 2306.80000 81.42548 28.33 <.0001X1 1 -885.80000 115.15301 -7.69 <.0001X2 1 -1181.40000 115.15301 -10.26 <.0001X3 1 -1220.20000 115.15301 -10.60 <.0001X4 1 -1486.80000 115.15301 -12.91 <.0001X5 1 -1526.80000 115.15301 -13.26 <.0001X6 1 -1433.00000 115.15301 -12.44 <.0001X7 1 -1559.20000 115.15301 -13.54 <.0001X8 1 -1646.20000 115.15301 -14.30 <.0001X9 1 -1457.20000 115.15301 -12.65 <.0001X10 1 13.80000 115.15301 0.12 0.9051X11 1 1452.80000 115.15301 12.62 <.0001

Average of rest is just December mean 2307. Subtract 886 in January, add 1452 in November. October (X10) is not significantly different than December.

negative

positive

Add date (days since Jan 1 1960 in SAS) to capture trendProc reg data=deer; model deer = date X1 X2 … X10 X11; Analysis of Variance



Parameter Estimates

Parameter StandardVariable Label DF Estimate Error t Value Pr > |t|Intercept Intercept 1 -1439.94000 547.36656 -2.63 0.0115X1 1 -811.13686 82.83115 -9.79 <.0001X2 1 -1113.66253 82.70543 -13.47 <.0001X3 1 -1158.76265 82.60154 -14.03 <.0001X4 1 -1432.28832 82.49890 -17.36 <.0001X5 1 -1478.99057 82.41114 -17.95 <.0001X6 1 -1392.11624 82.33246 -16.91 <.0001X7 1 -1525.01849 82.26796 -18.54 <.0001X8 1 -1618.94416 82.21337 -19.69 <.0001X9 1 -1436.86982 82.17106 -17.49 <.0001X10 1 27.42792 82.14183 0.33 0.7399X11 1 1459.50226 82.12374 17.77 <.0001date 1 0.22341 0.03245 6.88 <.0001

Trend is 0.22 more accidents per day (1 per 5 days) and is significantly different from 0.

Part 5 Logistic Regression

The problem: response is binary

yes or no, accident or no accident,

claim or no claim, at fault, not at fault

Prediction is prediction of probability (of fault for example)

• Logistic idea: Map p in (0,1) to L in whole real line, p=probability of fabric igniting.

• Use L = ln(p/(1-p))• Model L as linear in flame exposure time.• Predicted L = a + b(time)• Given temperature X, compute a+bX then p =

eL/(1+eL)• p(i) = ea+bXi/(1+ea+bXi) • Write p(i) if response, 1-p(i) if not• Multiply all n of these together, get function

Q(a,b), find a,b to maximize.

Example: Ignition

• Flame exposure time = X• Ignited Y=1, did not ignite Y=0– Y=1, X = 11, 12 14, 15, 17, 25, 30– Y=0, X= 3, 5, 9 10 , 13, 16

• Q=(1-p)(1-p)(1-p)(1-p)pp(1-p)pp(1-p)ppp• P’s all different p=f(exposure time)• Find a,b to maximize likelihood Q(a,b)

Likelihood function (Q)

-2.6

0.23

Example: Shuttle Missions

• O-rings failed in Challenger disaster• Low temperature• Prior flights “erosion” and “blowby” in O-rings• Feature: Temperature at liftoff• Target: problem (1) - erosion or blowby vs. no

problem (0)

introduction to predictive modeling with examples

Documents