poggi analytics - star - 1a

Buenos Aires, mayo de 2016Eduardo Poggi

Agenda

Reglas de generalización Algoritmo Star de Michalsky Algoritmo de Vere Learning First Order Rules

Agenda

Reglas de generalización Algoritmo Star de Michalsky Algoritmo de Vere Learning First Order Rules

Reglas de generalización

Eliminación de conjunción P(X) <- A(X) ^ B(X) ^ C(X) < P(X) <- A(X) ^ B(X)

Adición de disyunción P(X) <- A(X) < P(x) <- A(X) v B(X)

Conjunciones por disyunciones P(X) <- A(X) ^ B(X) < P(x) <- A(X) v B(X)

Reglas de generalización

Ampliación de rango de valores P(v/v in R1) < P(v/v in R2) iif R1<R2

Constantes por variables P(a) <- < P(X) <-

Resolución inductiva { P(X) <- A(X) ^ B(X); P(X) <- -A(X) ^ C(X) } < P(X) <- B(X) v C(X)

Reglas de generalización

Escalar en árbol de generalización

P(v) < P(t(v)) iif v<t(v)

DistanciasProducto

Comestibles Limpieza Indumentaria

Animal Vegetal Mineral

Lácteos Cárnicos

Leche liquida Leche fermentada Quesos Manteca

Yogurt entero Yogurt descremado

Yogurt natural Yogurt saborizado

Reglas de generalización constructivas

Reemplazo de términos P(X) <- A(X) ^ B(X) < P(X) <- A(X) ^ C(X) iff B(X) < C(X)

Agenda

Reglas de generalización Algoritmo Star de Michalsky Algoritmo de Vere Learning First Order Rules

STAR (Michalsky)

Hasta condición de terminación Seleccionar un ejemplo Obtener el árbol de generalización (STAR) a partir de

aplicar al ejemplo todas las reglas de generalización (especialización) posibles que no cubran contra-ejemplos.

Evaluar la lista de generalización y ordenar. Eliminar los ejemplos ya cubiertos.

STAR (Michalsky)

venenoso(X) <- color(X,marron)

forma(X,alargado) tierra(X,humeda)

ambiente(X,humedo) bajo_arbol(X,a33) arbol(a33,fresno)

venenoso(X) <- color(X,marron)

forma(X,alargado) tierra(X,humeda)

ambiente(X,humedo) bajo_arbol(X,Y) arbol(Y,fresno)

venenoso(X) <- color(X,marron)

forma(X,alargado) tierra(X,humeda) bajo_arbol(X,a33) arbol(a33,fresno)

venenoso(X) <- color(X,marron)

forma(X,alargado) tierra(X,humeda)

ambiente(X,humedo)

venenoso(X) <- …

STAR (Michalsky)

venenoso(X) <- color(X,[marron,verde])

forma(X,alargado) tierra(X,humeda)

bajo_arbol(X,Y) arbol(Y,[fresno,laurel])

venenoso(X) <- forma(X,alargado) tierra(X,humeda)

ambiente(X,humedo)

venenoso(X) <- color(X,marron)

forma(X,alargado) tierra(X,humeda)

ambiente(X,humedo) bajo_arbol(X,Y) arbol(Y,Z)

Agenda

Reglas de generalización Algoritmo Star de Michalsky Algoritmo de Vere Learning First Order Rules

Abstracción y GME

Abstracción como sustitución inductiva GCME = Generalización Común Máximalmente

Específica Acoplamiento y residuo

Vere

P = GCME de los ejemplos N = GCME de los contraejemplos C = P & -N Continuar iterativamente hasta: C = P & -(N1 & -(N2 & -(… & Nk) …))

Vere

P1 = color(X,marron) ^ forma(X,alargado) ^ tierra(X,humeda)

^ (ambiente(X,humedo) v ambiente(X,semi_humedo)^ bajo_arbol(X,Y) ^ arbol(Y,Z)

N1 = color(X,verde) ^ forma(X,redondo) ^ ambiente(X,semi_humedo)^ bajo_arbol(X,Y) ^ arbol(Y,Z)

C1 = P1 ^ -N2 = ?

Vere

C1 = P1 ^ -N2 = color(X,marron) ^ forma(X,alargado) ^ tierra(X,humeda) ^ (ambiente(X,humedo) v ambiente(X,semi_humedo)^ bajo_arbol(X,Y) ^ arbol(Y,Z) ^ - [ color(X,verde) ^ forma(X,redondo) ^ ambiente(X,semi_humedo) ^ bajo_arbol(X,Y) ^ arbol(Y,Z) ]

= color(X,marron) ^ forma(X,alargado) ^ tierra(X,humeda) ^ (ambiente(X,humedo) v ambiente(X,semi_humedo) ^ bajo_arbol(X,Y) ^ arbol(Y,Z) ^ -color(X,verde) ^ -forma(X,redondo) ^ -ambiente(X,semi_humedo) ^ - bajo_arbol(X,Y) ^ - arbol(Y,Z)

Vere

≈ … color(X,marron) ^ -color(X,verde) forma(X,alargado) ^ -forma(X,redondo) tierra(X,humeda) ^ ambiente(X,humedo ^ bajo_arbol(X,Y) ^ arbol(Y,Z)

ML como BH

K = ? Lista = {semilla} Hasta condición de terminación

Nodo = primero de la lista Seleccionar reglas de generalización aplicables al Nodo Aplicar reglas al Nodo y generar nuevos Nodos Calcular Performance de Nodos Agregar Nodos a la Lista Ordenar Lista según Performance Truncar Lista en los k mejores

Agenda

Reglas de generalización Algoritmo Star de Michalsky Algoritmo de Vere Learning First Order Rules

Learning set of rules

Learning sets of rules has the advantage that the hypothesis is easy to interpret.

Sequential covering algorithm to learn first-order rules.

Learning rules

First-order rule sets contain rules that have variables. This enables us to have stronger representational power.

Example: If Parent(x,y) then Ancestor(x,y) If Parent(x,z) and Ancestor(z,y) then Ancestor(x,y)

How would you represent this using a decision tree or predicate calculus?

Sequential Covering

General idea: Learn one rule that covers certain number of positive

examples Remove those examples covered by the rule Repeat until no positive examples are left.

Rule 1 Rule 2

Accuracy vs Coverage

We ask that each rule has high accuracy but not necessarily high coverage, for example:

Rule 1

Rule 1 has 90% accuracy and 50% coverage. In general thecoverage may be low as long as accuracy is high.

Sequential Covering Algorithm

Sequential-Covering (class,attributes,examples,threshold T)

RuleSet = 0 Rule = Learn-one-rule(class,attributes,examples) While (performance(Rule) > T) do

RuleSet += Rule Examples = Examples \ {ex. classified correctly by Rule} Rule = Learn-one-rule(class,attributes,examples)

Sort RuleSet based on the performance of the rules Return RuleSet

Sequential Covering Algorithm

Observations: It performs a greedy search (no backtracking); as such it

may not find an optimal rule set. It learns a disjunctive set of rules by learning each

disjunct (conjunction of att.values) at a time. It sequentially covers the set of positive examples until

the performance of a rule is below a threshold.

Learn One Rule

How do we learn each individual rule?One approach is to proceed as in decision tree learning but by following the branch with best score in terms of splitting function:

Luminosity

Mass

Type A Type B

Type C

> T1<= T1

> T2<= T2

If Luminosity <= T1 andMass > T2 then class is Type B

Learn One Rule

Observations: We greedily choose the attribute that most improves rule

performance over the training set. We perform a greedy depth first search with no backtracking. The algorithm can be extended using a beam-search:

We keep a list of the best k attributes at each step. For each attribute we generate descendants. In the next step we take the best k attributes and continue.

Algorithm

LearnOneRule(class,attributes,examples,k): Best-hypothesis = 0 Candidate-hypotheses = {Best-hypothesis} While Candidate-hypotheses is not empty do

Generate the next more specific candidate hypotheses Update Best-hypothesis

For all h in new-candidates if (Performance(h) > Performance(Best-hypothesis)) Best-hypothesis = h

Update Candidate-hypotheses = best k members of new-candidates Return rule: If Best-hypothesis then prediction (most frequent class

of examples covered by Best-hypothesis)

Algorithm

Generate the next more specific candidate hypotheses: Values = the set of all attribute values, e.g., color = blue For each rule h in Candidate-hypotheses do

For each attribute-value v do Add to h value v new-candidates += h

Remove from new-candidates hypotheses that are duplicates, inconsistent or not maximally specific.

Return new-candidates

Example

Astronomy problem: classifying objects as stars of different types.

Attributes: luminosity, mass, temperature, size.

Assume the set of possible values are as follows:

Luminosity <= T1 = l1 Luminosity > T1 = l2 Mass <= T2 = m1 Mass > T2 = m2 Temperature <= T3 = c1 Temperature > T3 = c2 Size <= T4 = s1 Size > T4 = s2

Running Algorithm on Example

Most specific hypotheses: l1, l2, m1, m2, c1, c2, s1, s2

Assume Performance = P P(c1) > P(x) for all x different than c1

Then best-hypothesis = c1

Assume k = 4

Best possible hypotheses: l1, m2, s1, c1

Running Algorithm on Example

Candidate hypotheses: l1, m2, and c1 New candidates:

l1 & l1 (*) l1 & L2 (^) m2 & l1 c1 & l1 l1 & m1 m2 & l2 c1 & l2 l1 & m2 m2 & m1 (^) … etc l1 & c1 m2 & m2 (*) l1 & c2 m2 & c1 l1 & s1 … etc l1 & s2

(*) duplicate (^) inconsistent

Running Algorithm on Example

Compute the performance of each new candidate. Update best-hypothesis to the best new candidate Example: Best–hypothesis = l1 & c2

Now take the best k = 3 new candidates And continue generating new candidates:

l1 & c2 & s1 l1 & c2 & s2 … etc

Performance Evaluation

The performance of a new candidate can be computed using information-theoretic measures like entropy:

Performance(h, examples, class) h_examples = the subsets of examples covered by h Return Entropy(h_examples)

Considerations

The best-hypothesis is the hypothesis with highest performance value, and not necessarily the last hypothesis:

Space:

L1, m2, c1 L1&m2, l2&s1, m2&c1 L1&m2&s1, l2&s1&c2, m2&c1&s2 …

Possible best hypothesis: l2&s1

Variations

What happens if the proportion of examples of a class is low?In other words what happens if the a priori probability of a classof examples is very low?

Example: patients with a very strange disease.

In that case we can modify the algorithm to learn only from those rare examples, and to classify anything outside the rule set as negative.

Variations

A second variation is used in the popular AQ and CN2 algorithms. General idea:

Choose one seed positive example Look for the most specific rule that covers the positive example

and has high performance Repeat with another seed example until no more improvement

is seen on the rule set

Variations

Rule 1 Rule 2

seed1seed2

Final Points for Consideration

A search can be done in a general-to-specific fashion. But one can also use a specific-to-general fashion. Which one is best?

Here we use a generate-then-test strategy. How about using an example-driven strategy like the candidate elimination algorithm? (this last type is more easily fooled by noise in the data)

When and how should we prune rules? Different performance metrics exist:

Relative frequency Accuracy Entropy

Rule learning and decision trees

What is the difference between both?

Decision trees: Divide and conquer Rule Learning: Separate and conquer

eduardopoggi@yahoo.com.ar

eduardo-poggi

http://ar.linkedin.com/in/eduardoapoggi

https://www.facebook.com/eduardo.poggi

@eduardoapoggi

Bibliografía