Random Forests: some methodological insights - arXiv ?· Centre de recherche INRIA Saclay – Île-de-France…

Download Random Forests: some methodological insights - arXiv ?· Centre de recherche INRIA Saclay – Île-de-France…

Post on 28-Sep-2018




0 download

Embed Size (px)


  • arX




    v1 [







    appor t de r ech er ch e














    Thme COG


    Random Forests: some methodological insights

    Robin Genuer Jean-Michel Poggi Christine Tuleau

    N 6729

    Novembre 2008

  • Centre de recherche INRIA Saclay le-de-FranceParc Orsay Universit

    4, rue Jacques Monod, 91893 ORSAY CedexTlphone : +33 1 72 92 59 00

    Random Forests: some methodological insights

    Robin Genuer , Jean-Michel Poggi , Christine Tuleau

    Theme COG Systemes cognitifsEquipes-Projets Select

    Rapport de recherche n 6729 Novembre 2008 32 pagesAbstract: This paper examines from an experimental perspective randomforests, the increasingly used statistical method for classification and regres-sion problems introduced by Leo Breiman in 2001. It first aims at confirming,known but sparse, advice for using random forests and at proposing some com-plementary remarks for both standard problems as well as high dimensionalones for which the number of variables hugely exceeds the sample size. But themain contribution of this paper is twofold: to provide some insights about thebehavior of the variable importance index based on random forests and in addi-tion, to propose to investigate two classical issues of variable selection. The firstone is to find important variables for interpretation and the second one is morerestrictive and try to design a good prediction model. The strategy involves aranking of explanatory variables using the random forests score of importanceand a stepwise ascending variable introduction strategy.

    Key-words: Random Forests, Regression, Classification, Variable

    Importance, Variable Selection.

    Universite Paris-Sud, Mathematique, Bat. 425, 91405 Orsay, France Universite Paris Descartes, France Universite Nice Sophia-Antipolis, France

  • Forets aleatoires : remarques methodologiques

    Resume : On sinteresse a la methode des forets aleatoires dun point de vuemethodologique. Introduite par Leo Breiman en 2001, elle est desormais large-ment utilisee tant en classification quen regression avec un succes spectaculaire.On vise tout dabord a confirmer les resultats experimentaux, connus mais epars,quant au choix des parametres de la methode, tant pour les problemes dits stan-dards que pour ceux dits de grande dimension (pour lesquels le nombre devariables est tres grand vis a vis du nombre dobservations). Mais la contributionprincipale de cet article est detudier le comportement du score dimportance desvariables base sur les forets aleatoires et dexaminer deux problemes classiquesde selection de variables. Le premier est de degager les variables importantes ades fins dinterpretation tandis que le second, plus restrictif, vise a se restreindrea un sous-ensemble suffisant pour la prediction. La strategie generale procedeen deux etapes : le classement des variables base sur les scores dimportancesuivi dune procedure dintroduction ascendante sequentielle des variables.

    Mots-cles : Forets aleatoires, Regression, Classification, Impor-

    tance des Variables, Selection des Variables.

  • Random Forests: some methodological insights 3

    1 Introduction

    Random forests (RF henceforth) is a popular and very efficient algorithm, basedon model aggregation ideas, for both classification and regression problems, in-troduced by Breiman (2001) [8]. It belongs to the family of ensemble methods,appearing in machine learning at the end of nineties (see for example Dietterich(1999) [15] and (2000) [16]). Let us briefly recall the statistical framework byconsidering a learning set L = {(X1, Y1), . . . , (Xn, Yn)} made of n i.i.d. obser-vations of a random vector (X, Y ). Vector X = (X1, ..., Xp) contains predictorsor explanatory variables, say X Rp, and Y Y where Y is either a class labelor a numerical response. For classification problems, a classifier t is a mappingt : Rp Y while for regression problems, we suppose that Y = s(X) + ands is the so-called regression function. For more background on statistical learn-ing, see Hastie et al. (2001) [24]. Random forests is a model building strategyproviding estimators of either the Bayes classifier or the regression function.

    The principle of random forests is to combine many binary decision treesbuilt using several bootstrap samples coming from the learning sample L andchoosing randomly at each node a subset of explanatory variables X . Moreprecisely, with respect to the well-known CART model building strategy (seeBreiman et al. (1984) [6]) performing a growing step followed by a pruning one,two differences can be noted. First, at each node, a given number (denoted bymtry) of input variables are randomly chosen and the best split is calculatedonly within this subset. Second, no pruning step is performed so all the treesare maximal trees.

    In addition to CART, another well-known related tree-based method mustbe mentioned: bagging (see Breiman (1996) [7]). Indeed random forests withmtry = p reduce simply to unpruned bagging. The associated R1 packagesare respectively randomForest (intensively used in the sequel of the paper),rpart and ipred for CART and bagging respectively (cited here for the sake ofcompleteness).

    RF algorithm becomes more and more popular and appears to be very pow-erful in a lot of different applications (see for example Daz-Uriarte and Alvarezde Andres (2006) [14] for gene expression data analysis) even if it is not clearlyelucidated from a mathematical point of view (see the recent paper by Biauet al. (2008) [5] and Buhlmann, Yu (2002) [11] for bagging). Nevertheless,Breiman (2001) [8] sketches an explanation of the good performance of randomforests related to the good quality of each tree (at least from the bias point ofview) together with the small correlation among the trees of the forest, wherethe correlation between trees is defined as the ordinary correlation of predictionson so-called out-of-bag (OOB henceforth) samples. The OOB sample which isthe set of observations which are not used for building the current tree, is usedto estimate the prediction error and then to evaluate variable importance.

    Tuning method parameters

    It is now classical to distinguish two typical situations depending on n thenumber of observations, and p the number of variables: standard (for n >> p)and high dimensional (when n

  • 4 Genuer et. al

    for the two main parameters of the method. Essentially, the study carried outin the two papers [8] and [14] give interesting insights but Breiman focuses onstandard problems while Daz-Uriarte and Alvarez de Andres concentrate onhigh dimensional classification ones.

    So the first objective of this paper is to give compact information aboutselected bench datasets and to examine again the choice of the method param-eters addressing more closely the different situations.

    RF variable importance

    The quantification of the variable importance (VI henceforth) is an impor-tant issue in many applied problems complementing variable selection by inter-pretation issues. In the linear regression framework it is examined for exampleby Gromping (2007) [22], making a distinction between various variance de-composition based indicators: dispersion importance, level importance ortheoretical importance quantifying explained variance or changes in the re-sponse for a given change of each regressor. Various ways to define and computeusing R such indicators are available (see Gromping (2006) [23]).

    In the random forests framework, the most widely used score of importanceof a given variable is the increasing in mean of the error of a tree (MSE forregression and misclassification rate for classification) in the forest when theobserved values of this variable are randomly permuted in the OOB samples.Often, such random forests VI is called permutation importance indices in op-position to total decrease of node impurity measures already introduced in theseminal book about CART by Breiman et al. (1984) [6].

    Even if only little investigation is available about RF variable importance,some interesting facts are collected for classification problems. This index canbe based on the average loss of another criterion, like the Gini entropy used forgrowing classification trees. Let us cite two remarks. The first one is that the RFGini importance is not fair in favor of predictor variables with many categorieswhile the RF permutation importance is a more reliable indicator (see Strobl etal. (2007) [36]). So we restrict our attention to this last one. The second oneis that it seems that permutation importance overestimates the variable impor-tance of highly correlated variables and they propose a conditional variant (seeStrobl et al. (2008) [37]). Let us mention that, in this paper, we do not noticesuch phenomenon. For classification problems, Ben Ishak, Ghattas (2008) [4]and Daz-Uriarte, Alvarez de Andres (2006) [14] for example, use RF variableimportance and note that it is stable for correlated predictors, scale invariantand stable with respect to small perturbations of the learning sample. But thesepreliminary remarks need to be extended and the recent paper by Archer et al.(2008) [3], focusing more specifically on the VI topic, do not answer some cru-cial questions about the variable importance behavior: like the importance ofa group of variables or its behavior in presence of highly correlated variables.This one is the second goal of this paper.

    Variable selection

    Many variable selection procedures are based on the cooperation of variableimportanc


View more >