Decision Tree Learning

Presented by Ping Zhang

Nov. 26th, 2007

Introduction Decision tree learning is one of the most

widely used and practical method for inductive inference

Decision tree learning is a method for approximating discrete-valued target functions, in which the learned function is represented by a decision tree

Decision tree learning is robust to noisy data and capable of learning disjunctive expressions

Decision tree representation Decision tree classify instances by

sorting them down the tree from the root to some leaf node, which provides the classification of the instance

Each node in the tree specifies a test of some attribute of the instance, and each branch descending from that node corresponds to one of the possible values for this attributes

Decision Tree for PlayTennis

When to Consider Decision Trees Instances describable by attribute-value

pairs Target function is discrete valued Disjunctive hypothesis may be required Possibly noisy training data

Examples (Classification problems): Equipment or medical diagnosis Credit risk analysis

Top-Down Induction of Decision Trees

Entropy (1)

Entropy (2)

Information Gain

Training Examples

Selecting the Next Attribute

Which attribute should be tested here?

Hypothesis Space Search by ID3 Hypothesis space is complete Target function surely in there Only outputs a single hypothesis No back tracking Local minima Statically-based search choices Robust to noisy data Inductive bias: “prefer shortest tree”

From ID3 to C4.5 C4.5 made a number of improvements to ID3. Some of these are: Handling both continuous and discrete attributes Handling training data with missing attribute value Handling attributes with differing costs Pruning trees after creation

Overfitting in Decision Trees

Reduced-Error Pruning

Rule Post-Pruning Convert tree to equivalent set of rules Prune each rule by removing any preconditions that result in

improving its estimated accuracy Sort the pruned rules by their estimated accuracy, and consider

them in this sequence when classifying subsequent instance

Perhaps most frequently used method

Continuous Valued Attributes Create a discrete attribute to test continuous

There are two candidate thresholds The information gain can be computed for each of the candidate attributes, Temperature>54 and Temperature>85, and the best can be selected(Temperature>54)

Attributes with many ValuesProblems: If attribute has many values, Gain will select it Imagine using the attribute Data. It would have the

highest information gain of any of attributes. But the decision tree is not useful.

Missing Attribute Values

Attributes with Costs Consider Medical diagnosis, BloodTset has cost 150 dallors How to learn a consistent tree with low expected cost?

Conclusion Decision Tree Learning is Simple to understand and interpret Requires little data preparation Able to handle both numerical and

categorical data Use a white box model Possible to validate a model using

statistical tests Robust, perform well with large data in

a short time