unsupervised forward selection a data reduction algorithm for use with very large data sets david...

Unsupervised Forward SelectionA data reduction algorithm for use

with very large data sets

David Whitley†, Martyn Ford† and David Livingstone†‡

†Centre for Molecular Design, University of Portsmouth†‡ChemQuest

Outline

• Variable selection issues• Pre-processing strategy• Dealing with multicollinearity• Unsupervised forward selection• Model selection strategy• Applications

Variable Selection Issues

• Relevance– statistically significant correlation with response– non-small variance

• Redundancy– linear dependence

– some variables have no unique information

• Multicollinearity– near linear dependence

– some variables have little unique information

0 iiv

0 iiv

Pre-processing Strategy

• Identify variables with a significant correlation with the response

• Remove variables with small variance• Remove variables with no unique information• Identify a set of variables on which to construct a

model

Effect of Multicollinearity

ii xxxxxy 55443322110

izxx 15

Build regression models of the form

where

Increasing reduces the collinearity between x5 and x1

and x1 - x4 , y, zi and ei are random N(0,1)

Effect of Multicollinearity

Q2

Dealing with Multicollinearity

• Examine pair-wise correlations between variables, and remove one from each pair with high correlation

• Corchop (Livingstone & Rahr, 1989) aims to remove the smallest number of variables while breaking the largest number of pair-wise collinearities

Unsupervised Forward Selection1 Select the first two variables with the smallest pair-

wise correlation coefficient2 Reject variables whose pair-wise correlation

coefficient with the selected columns exceeds rsqmax3 Select the next variable to have the smallest squared

multiple correlation coefficient with those previously selected

4 Reject variables with squared multiple correlation coefficients greater than rsqmax

5 Repeat 3 - 4 until all variables are selected or rejected

Continuum Regression• A regression procedure with the generalized

criterion function)21(422 )''()''(

2 XcXcyXcF

• Varying the continuous parameter 0 1.5 adjusts the balance between the covariance of the response with the descriptors and the variance of the descriptors, so that = 0 is equivalent to ordinary least squares = 0.5 is equivalent to partial least squares = 1.0 is equivalent to principal components regression

Model Selection Strategy

• For = 0.0, 0.1, …, 1.5 build a CR model for the set of variables selected by UFS with rsqmax = 0.1, 0.2, …, 0.9, 0.99

• Select the model with rsqmax and maximizing Q2 (leave-one-out cross-validated R2)– Apply n-fold cross-validation to check predictive

ability– Apply a randomization test (1000 permutations of the

response scores) to guard against chance correlation

Pyrethroid Data Set

• 70 physicochemical descriptors to predict killing activity (KA) of 19 pyrethroid insecticides

• Only 6 descriptors are correlated with KA at the 5% level

• Optimal models– 4-variable, 2-component model with R2 = 0.775,

Q2 = 0.773 obtained when rsqmax = 0.7, = 1.2

– 3-variable, 1-component model with R2 = 0.81, Q2 = 0.76 obtained when rsqmax = 0.6, = 0.2

Optimal Model I

• Standard errors are bootstrap estimates based on 5000 bootstraps

• Randomization test tail probabilities below 0.0003 for fit and 0.0071 for prediction

DVXMIZAAKA 037.000024.08044.0564.931.2 )80.2( )98( )000083.0( )11.0(

Optimal Model II

• Standard errors are bootstrap estimates based on 5000 bootstraps

• Randomization test tail probabilities below 0.0001 for fit and 0.0052 for prediction

DVXMIZAKA 20.000019.0567.880.1 )00.2( )000055.0( )08.0(

N-Fold Cross-Validation

3 variable model4 variable model

Feature Recognition

• Important explanatory variables may not be selected for inclusion in the model– force some variables in, then continue UFS algorithm

• The component loadings for the original variables can be examined to identify variables highly correlated with the components in the model

Loadings for the 1-component pyrethroid model with tail probability < 0.01

variable loading

A5 0.756

A3 0.723

A8 0.619

NS16 - 0.605

DVX - 0.603

ES12 - 0.584

MIZ 0.567

Steroid Data Set

• 21 steroid compounds from SYBYL CoMFA tutorial to model binding affinity to human TBG

• Initial data set has 1248 variables with values below 30 kcal/mol

• Removed 858 variables not significantly correlated with response (5% level)

• Removed 367 variables with variance below 1.0 kcal/mol

• Leaving 23 variables to be processed by UFS/CR

Optimal models

• UFS/CR produces a 3-variable, 1-component model with R2 = 0.85, Q2 = 0.83 at rsqmax = 0.3, = 0.3

• CoMFA tutorial produces a 5-component model with R2 = 0.98, Q2 = 0.6

N-Fold Cross-Validation

CoMFA tutorial model UFS/CR model

Putative Pharmacophore

Selwood Data Set

• 53 descriptors to predict biological activity of 31 antifilarial antimycin analogues

• 12 descriptors are correlated with the response variable at the 5% level

• Optimal models– 2-variable, 1-component model with R2 = 0.42,

Q2 = 0.41 obtained when rsqmax = 0.1, = 1.0

– 12-variable, 1-component model with R2 = 0.85, Q2 = 0.5 obtained when rsqmax = 0.99, = 0.0 (omitting compound M6)

N-Fold Cross-Validation

2-variable model 12-variable model

Summary

• Multicollinearity is a potential cause of poor predictive power in regression.

• The UFS algorithm eliminates redundancy and reduces multicollinearity, thus improving the chances of obtaining robust, low-dimensional regression models.

• Chance correlation can be addressed by eliminating variables that are uncorrelated with the response.

Summary

• UFS can be used to adjust the balance between reducing multicollinearity and including relevant information.

• Case studies show that leave-one-out cross-validation should be supplemented by n-fold cross-validation, in order to obtain accurate and precise estimates of predictive ability (Q2).

Acknowledgements

• Astra Zeneca• GlaxoSmithKline• MSI• Unilever

BBSRC Cooperation with Industry Project: Improved Mathematical Methods for Drug Design

Reference

D. C. Whitley, M.G. Ford and D. J. Livingstone Unsupervised forward selection: a method for eliminating redundant variables.J. Chem. Inf. Comp. Sci., 2000, 40, 1160-1168.

UFS software available from: http://www.cmd.port.ac.uk

CR is a component of Paragon (available summer 2001)