lectures 15,16 – additive models, trees, and related methods rice ece697 farinaz koushanfar fall...
TRANSCRIPT
Lectures 15,16 – Additive Models, Trees, and Related Methods
Rice ECE697
Farinaz Koushanfar
Fall 2006
Summary
• Generalized Additive Models
• Tree-Based Methods
• PRIM – Bump Hunting
• Mutlivariate Adaptive Regression Splines (MARS)
• Missing Data
Additive Models
• In real life, effects are nonlinear
•
Note: Some slides are borrowed from Tibshirani
Examples
The Price for Additivity
Data from a study of Diabetic children, Predicting log C-peptide(a blood measurement)
Generalized Additive Models (GAM)Two-class Logistic Regression
Other Examples
Fitting Additive Models
• Given observations xi,yi, a criterion like the penalized sum of squares can be specified for this problem, where ’s are tuning parameters
p
1jjj)X(fY The mean of error term is zero!
N
1i
p
1jj
2
jjj
2p
1jijji
p1
dt)t("f})x(fy{
)f,...,f,(PRSS
Fitting Additive Models
The Backfitting Algorithm for Additive Models
• Initialize:
• Cycle: j=1,2,…,p,1,2,…,p,1,…
• Until the functions fj change less than a prespecified threshold
j,i,0f̂;yN
1j
N
1ii
]})x(f̂y[{Sf̂ N
1jk
ikkijj
N
1iijjjj)x(f̂
N
1f̂f̂
Fitting Additive Models (Cont’d)
Example: Penalized Least square
Example: Fitting GAM for Logistic Regression (Newton-Raphson Algorithm)
Example: Predicting Email Spam
• Data from 4601 mail messages, spam=1, email=0, filter trained for each user separately
• Goal: predict whether an email is spam (junk mail) or good
• Input features: relative frequencies in a message of 57 of the commonly occurring words and punctuation marks in all training set
• Not all errors are equal; we want to avoid filtering out good email, while letting spam get through is not desirable but less serious in its consequences
Predictors
Details
Some Important Features
Results
• Test data confusion matrix for the additive logistic regression model fit to the spam training data
• The overall test error rate is 5.3%
Summary of Additive Logistic Fit• Significant predictors from the additive model fit to the spam
training data. The coefficients represent the linear part of f^j,
along with their standard errors and Z-score. • The nonlinear p-value represents a test of nonlinearity of f^
j
Example: Plots for Spam Analysis
Figure 9.1. Spam analysis: estimated functions for significant predictors. The rug plot along the bottom of each frame indicates the observed values of the corresponding predictor. For many predictors, the nonlinearity picks up the discontinuity at zero.
In Summary
• Additive models are a useful extension to linear models, making them more flexible
• The backfitting procedure is simple and modular
• Limitations for large data mining applications
• Backfitting fits all predictors, which is not desirable when a large number are available
Tree-Based Methods
Node Impurity Measures
Results for Spam Example
Pruned tree for the Spam Example
Classification Rules Fit to the Spam Data
PRIM-Bump Hunting
Number of Observations in a Box
Basis Functions
MARS Forward Modeling Procedure
Multiplication of Basis Functions
MARS on Spam Example