review

Review

Belief and Probability• The connection between toothaches and cavities is not a logical

consequence in either direction.

• However, we can provide a degree of belief on the sentences.– We usually get this belief from statistical data.

• Assigning probability 0 to a sentence correspond to an unequivocal belief that the sentence is false.

• Assigning probability 1 to a sentence correspond to an unequivocal belief that the sentence is true.

Syntax• Basic element: random variable• Possible worlds defined by assignment of values to random variables.

• Boolean random variables

Cavity (do I have a cavity?)• Discrete random variables

Weather is one of <sunny,rainy,cloudy,snow>• Domain values must be exhaustive and mutually exclusive

• Elementary propositions e.g.,

– Weather = sunny (abbreviated as sunny) – Cavity = false (abbreviated as cavity)

• Complex propositions formed from elementary propositions and standard logical connectives e.g., – Weather = sunny Cavity = false– sunny cavity

Atomic events• Atomic event: A complete specification of the state of the world

E.g., if the world is described by only two Boolean variables Cavity and Toothache, then there are 4 distinct atomic events:

Cavity = false Toothache = false

Cavity = false Toothache = true

Cavity = true Toothache = false

Cavity = true Toothache = true

• Atomic events are mutually exclusive and exhaustive

Joint probability• Joint probability distribution for a set of random variables gives the

probability of every atomic event on those random variables.

• If we consider all the variables then the joint probability distribution is called full joint probability distribution.

– A full joint distribution specifies the probability of every atomic event.

• Any probabilistic question about a domain can be answered by the full joint distribution.

Prior and Conditional probability• Prior or unconditional probability associated with a proposition is the degree

of belief accorded to it in the absence of any other information.

P(Cavity = true) = 0.1 (or P(cavity) = 0.1)

P(Weather = sunny) = 0.7 (or P(sunny) = 0.7)

• Conditional or posterior probabilities

P(cavity | toothache) = 0.8

i.e., given that toothache is all I know

• Definition of conditional probability:

• Product rule:

0)( if )(

)()|(

bP

bP

baPbaP

)()|()()|()( aPabPbPbaPbaP

Chain rule• Chain rule is derived by successive application of product rule:

n

i ii

nnnnn

nnnn

xxxP

xxxPxxxPxxP

xxxPxxPxxP

1 11

1121121

11111

),...,|(

...

),...,|(),...,|(),...,(

),...,|(),...,(),...,(

)...( means ),...,( 111 nnn xXxXPxxP

Inference by enumeration• Suppose we are given a proposition φ. Start with the joint

probability distribution:

• Sum up the probabilities of the atomic events where φ is true.

Inference by enumeration

Inference by enumeration• We can also compute conditional probabilities:

Normalization Constant

12.0 0.012)(0.108

)(

)(

)()|(

toothachecavityP

toothacheP

toothachecavityPtoothachecavityP

08.0 0.064)(0.016

)(

)(

)()|(

toothachecavityP

toothacheP


508.012.0

1

108.012.0

1)|()|(


Hidden Variables• What does it mean to compute

)( toothachecavityP

• We sum up: 0.108 + 0.012• I.e. the probabilities of atomic events that make the proposition

cavity toothache true. So,

)()(

)(

catchtoothachecavityPcatchtoothachecavityP

toothachecavityP

• The variable “Catch” is a called a hidden variable.

Independence• We can write:

P(toothache, catch, cavity, cloudy) =P(cloudy | toothache, catch, cavity) P(toothache, catch, cavity)

=P(cloudy) P(toothache, catch, cavity)

• Thus, the 32 element table for four variables can be constructed from one 8-element table and one 4-element table!!

• A and B are independent if for each a, b in the domain of A and B respectively, we have

P(a|b) = P(a) or

P(b|a) = P(b) or

P(a, b) = P(a) P(b)

• Absolute independence powerful but rare

Bayes' Rule• Product rule

• Bayes' rule:

• Useful for assessing diagnostic probability from causal probability:

)()|()()|()( aPabPbPbaPbaP

)(

)()|()|(

bP

aPabPbaP

)(

)()|()|(

effectP

causePcauseeffectPeffectcauseP

Bayes’ rule (cont’d)Let s be the proposition that the patient has a stiff neck

m be the proposition that the patient has meningitis,

P(s|m) = 0.5

P(m) = 1/50000

P(s) = 1/20

P(m|s) = P(s|m) P(m) / P(s)

= (0.5) x (1/50000) / (1/20)

= 0.0002

That is, we expect only 1 in 5000 patients with a stiff neck to have meningitis.

More than two variables

)()|,...,(

),...,(

)()|,...,(

),...,|(

1

1

1

1

causePcauseeffecteffectP

effecteffectP


effecteffectcauseP

n

n

n

n

Now, the Naïve Bayes model makes the following assumption:Although Effect1, …,Effectn might not be independent in general,

they are independent given the value of Cause. This is called conditional independence.

E.g. If I have a cavity, the probability that the probe catches in it doesn't depend on whether I have a toothache.

We write this as: P(toothache, catch | cavity) = P(toothache | cavity) . P(catch | cavity) . P(cavity)

Naïve Bayes

)()|(

)()|()|(

)()|,...,(

),...,|(

1

1

1

1

causePcauseeffectP

causePcauseeffectPcauseeffectP


effecteffectcauseP

n

i i

n

n

n

For finding the alpha we need to compute also:

)()|(

),...,|(

1

1

causePcauseeffectP

effecteffectcausePn

i i

n

Then the alpha is:

),...,|(),...,|(

1

11 nn effecteffectcausePeffecteffectcauseP

What about when the Naïve Bayes assumption doesn’t hold?

Instead we have a network of inter-dependencies.

Let’s, first review the conditional independence.

Conditional Independence (cont’d)• We can write out full joint distribution using chain rule, e.g.:

P(toothache, catch, cavity)

= P(toothache | catch, cavity) P(catch, cavity)

= P(toothache | catch, cavity) P(catch | cavity) P(cavity)

= P(toothache | cavity) P(catch | cavity) P(cavity)

• In most cases, the use of conditional independence reduces the size of the representation of the joint distribution from exponential in n to linear in n.

• Conditional independence is our most basic and robust form of knowledge about uncertain environments.

Because of Cond. Indep.

Bayesian Networks: Motivation

• The full joint probability can be used to answer any question about the domain, – but intractable as the number of variables grow.

• Furthermore specifying probabilities of atomic events is rather unnatural and can be very difficult.

Bayesian networks• Syntax:

– a set of nodes, one per variable

– a directed, acyclic graph (a link means: "directly influences")

– a conditional distribution for each node given its parents:

P(Xi | Parents (Xi))

• The conditional distribution is represented as a conditional probability table (CPT) giving the distribution over Xi for each

combination of parent values.

Example• Topology of network encodes conditional independence

assertions:

• Weather is independent of the other variables• Toothache and Catch are conditionally independent given

Cavity, which is indicated by the absence of a link between them.

Another Example

The topology shows that burglary and earthquakes directly affect the probability of alarm, but whether Mary or John call depends only on the alarm.

Thus, our assumptions are that they don’t perceive any burglaries directly, and they don’t confer before calling.

Semantics• The full joint distribution is

defined as the product of the local conditional distributions:

P(x1, … ,xn) =

i = 1 P(xi | parents(xi))

e.g.,

P(j m a b e)

= P(j | a) P(m | a) P(a | b, e) P(b) P(e)

= …

Inference in Bayesian Networks• The basic task for any probabilistic inference system is to

compute the posterior probability for a query variable, given some observed events (or effects) – that is, some assignment of values to a set of evidence variables.

• A typical query:

P(x|e1,…,em)

• We could ask: What’s the probability of a burglary if both Mary and John calls

P(burglary | johncalls, marycalls)?

Inference by enumerationSum out variables from the joint without actually

constructing its explicit representation

e amjaebP

mjbP

mjP

mjbP

mjbP

),,,,(

),,(

),(

),,(

),|(

e (earthquake) and a (alarm)are values of the hidden variables.All the possible e’s and a’s have to be considered.

Now, rewrite full joint entries using product of CPT entries:

)|()|(),|()()(

),,,,(

),|(

amPajPebaPePbP

mjaebP

mjbP

e a

e a

Machine Learning

Pseudo-code for 1RFor each attribute,

For each value of the attribute, make a rule as follows:

count how often each class appears

find the most frequent class

make the rule assign that class to this attribute-value

Calculate the error rate of the rules

Choose the rules with the smallest error rate

Evaluating the weather attributes

Arbitrarily breaking the tie between the firstand third rule sets we pick the first.Oddly enough the game is played when it’s overcast and rainy but not when it’s sunny. Perhaps it’s an indoor pursuit.

Statistical modeling: Probabilities for the weather data

Naïve Bayes for classification• Classification learning: what’s the probability of the class

given an instance?– e = instance

– h = class value for instance

• Naïve Bayes assumption: evidence can be split into independent parts (i.e. attributes of instance!)

P(h | e) = P(e | h) P(h) / P(e)

= P(e1|h) P(e2|h)… P(en|h) P(h) / P(e)

The weather data example

P(Play=yes | e) =

P(Outlook=Sunny | Play=yes) *

P(Temp=Cool | Play=yes) *

P(Humidity=High | Play=yes) *

P(Windy=True | Play=yes) *

P(Play=yes) / P(e) =

= (2/9) * (3/9) * (3/9) * (3/9) *(9/14) / P(e) = 0.0053 / P(e)

Don’t worry for the 1/P(E); It’s alpha, the normalization constant.

Normalization constant = 1/P(e) = 1/(0.0053 + 0.0206)

So,

P(Play=yes | e) = 0.0053 / (0.0053 + 0.0206) = 20.5%

P(Play=no | e) = 0.0206 / (0.0053 + 0.0206) = 79.5%

The “zero-frequency problem”• What if an attribute value doesn’t occur with every class

value (e.g. “Humidity = High” for class “Play=Yes”)?– Probability P(Humidity=High | play=yes) will be zero!

• A posteriori probability will also be zero! – No matter how likely the other values are!

• Remedy: add 1 to the count for every attribute value-class combination (Laplace estimator)– I.e. initialize the counters to 1 instead of 0.

• Result: probabilities will never be zero! (stabilizes probability estimates)

Constructing Decision Trees• Normal procedure: top down in recursive divide-and-

conquer fashion

– First: an attribute is selected for root node and a branch is

created for each possible attribute value

– Then: the instances are split into subsets (one for each

branch extending from the node)

– Finally: the same procedure is repeated recursively for each

branch, using only instances that reach the branch

• Process stops if all instances have the same class

Which attribute to select?

(a)(b)

(c) (d)

A criterion for attribute selection• Which is the best attribute?

• The one which will result in the smallest tree– Heuristic: choose the attribute that produces the “purest”

nodes

• Popular impurity criterion: entropy of nodes– Lower the entropy purer the node.

• Strategy: choose attribute that results in lowest entropy of the children nodes.

Example: attribute “Outlook”

The final decision tree

• Note: not all leaves need to be pure; sometimes identical instances have different classes

Splitting stops when data can’t be split any further

Numerical attributes• Tests in nodes can be of the form xj > constant

• Divides the space into rectangles.

Considering splits• The only thing we really need to do differently in our algorithm

is to consider splitting between each data point in each dimension.

• So, in our bankruptcy domain, we'd consider 9 different splits in the R dimension – In general, we'd expect to

consider m - 1 splits, if we have m data points;

– But in our data set we have some examples with equal R values.

Considering splits II• And there are another 6 possible splits in the L dimension

– because L is an integer, really, there are lots of duplicate L values.

Bankruptcy Example

Bankruptcy Example• We consider all the possible splits in each dimension, and

compute the average entropies of the children.

• And we see that, conveniently, all the points with L not greater than 1.5 are of class 0, so we can make a leaf there.

Bankruptcy Example• Now, we consider all the splits of the remaining part of space.

• Note that we have to recalculate all the average entropies again, because the points that fall into the leaf node are taken out of consideration.

Bankruptcy Example• Now the best split is at R > 0.9. And we see that all the points

for which that's true are positive, so we can make another leaf.

Bankruptcy Example• Continuing in this way, we finally obtain:

Rules: Covering• Strategy for generating a rule set directly is based on

covering.

• A rule “lhs then rhs” covers a data instance, if the tests in “lhs” are true for that instance.

Example: contact lenses data

Example: contact lenses data

The numbers on the right show the fraction of “correct” instances in the set singled out by that choice (second numbers show covering).In this case, correct means that their recommendation is “hard.”

Selecting a Test for the LHS of a Rule• Goal: to maximize accuracy

– t: total number of instances covered by rule

– p: positive examples of the class covered by rule

– t-p: number of errors made by rule

Select test that maximizes the ratio p/t

• We are finished when p/t = 1 or the set of instances can’t be split any further

Modified rule and resulting data

The rule isn’t very accurate, getting only 4 out of 12 that it covers. So, it needs further refinement.

Further refinement

Modified rule and resulting data

Should we stop here? Perhaps. But let’s say we are going for exact rules, no matter how complex they become.So, let’s refine further.

Further refinement

The result

Linear Hypothesis Class• Equation of a hyperplane in the feature space

• w, b are to be learned

• In two dimensions, we can see the geometric interpretation of w and b. – The vector w is perpendicular to the linear

separator; such a vector is known as the normal vector.

– The scalar b, which is called the offset, is proportional to the perpendicular distance from the origin to the linear separator.

– The constant of proportionality is the negative of the magnitude of the normal vector.

Hyperplane: Geometry

Linear classifier• We can now exploit the sign of this distance to define a linear

classifier, one whose decision boundary is a hyperplane.

• Instead of using 0 and 1 as the class labels (which was an arbitrary choice anyway) we use the sign of the distance, either +1 or -1 as the labels (that is the values of the yi ’s).

Margin• A variant of the signed

distance of a training point to a hyperplane is the margin of the point.

Margin: i = yi(w.xi+b) =

proportional to perpendicular distance of point xi to hyperplane.

i > 0 : point is correctly classified (sign of distance = yi)

i < 0 : point is incorrectly classified (sign of distance yi)

iiy xw

Training: Perceptron algorithm

Artificial Neural Networks• The basic idea in neural nets is to define

interconnected networks of simple units (let's call them "artificial neurons") in which each connection has a weight.

– Weight wij is the weight of the ith input into unit j.

– The networks have some inputs where the feature values are placed and they compute one or more output values.

– Each output unit corresponds to a class. The network prediction is the output whose value is highest.

• The learning takes place by adjusting the weights in the network so that the desired output is produced whenever a sample in the input data set is presented.

Single Perceptron Unit• We start by looking at a simpler

kind of "neural-like" unit called a perceptron.

– This is where the perceptron algorithm that we saw earlier came from.

– Perceptrons antedate the modern neural nets.

• A perceptron unit basically compares a weighted combination of its inputs against a threshold value and then outputs a 1 if the weighted inputs exceed the threshold.

• Trick: we treat the (arbitrary) threshold as if it were a weight w0 on a constant input x0 whose value is –1.

• In this way, we can write the basic rule of operation as computing the weighted sum of all the inputs and comparing to 0.

Linear Classifier: Single Perceptron Unit

)())1(()( 0 xwxwx whb z

where

else

zz

0

01)(

Multi-Layer Perceptron• Yes. The introduction of

"hidden" units into these networks make them much more powerful: – they are no longer limited to

linearly separable problems.

• Earlier layers transform the problem into more tractable problems for the latter layers.

Check XOR example in the full slides.

Multi-Layer Perceptron Learning• Any set of training points can be separated by a three-layer

perceptron network.

• “Almost any” set of points is separable by two-layer perceptron network.

• However, the presence of the discontinuous threshold in the operation means that there is no simple local search for a good set of weights; – one is forced into trying possibilities in a combinatorial way.

• The limitations of the single-layer perceptron and the lack of a good learning algorithm for multilayer perceptrons essentially killed the field for quite a few years.

Sigmoid Unit• The key property of the

sigmoid is that it is differentiable. – The output of this

function (y) varies smoothly with changes in the weights.

– So, we can use gradient.

Gradient Descent

E

w

y

w

yy

yyyE

yyE

n

m

mmm

mm

w

w

ww

ww

w

wx

wxwx

wx

follows as change We

,...,),( where

),()),((

)),((2

1

1

2

Here we see the gradient of the training error as a function of the weights.

The descent rule is basically to change the weights by taking a small step (determined by the learning rate ) in the directionopposite this gradient.

Online version: We consider each time only the error for one data item

Had we a single unit…

i

i

ii

z

n

iii

n

i

xz

zs

w

z

z

zs

w

z

z

y

w

y

ezsyxwz

w

y

w

yy

)(

)(

1

1)(

,...,),(1

wxw

ii

m

im

ii

xwz

zsyy

z

E

xz

zsyyww

)()(

)()(

Delta rule

Substituting in the equation of previous slide we get (for the arbitrary ith element):

Derivative of the sigmoid

)1(

))(1)((

11

1

]][)1([

)1()(

1

1)(

2

1

yy

zszs

e

e

e

ee

edz

d

dz

zdse

zs

z

z

z

zz

z

z

Generalized Delta Rule

))(1( 646545444

2442

1441

wwyy

yw

yw

)(

)1(jdownstreamk

kjkjjj

ijji

wyy

yw

In general, for a hidden unit j we have

Immediate downstream only.

Generalized Delta RuleFor an output unit we have

))(1( mnnnn yyyy

inni yw

Where

yn is the output of this output unit.

y is the real class of the on focus data instance.

Backpropagation Algorithm1. Initialize weights to small random values2. Choose a random sample training item, say (xm, ym)

3. Compute total input zj and output yj for each unit (forward prop)

4. Compute n for output layer n = yn(1-yn)(yn-ynm)

5. Compute j for all preceding layers by backprop rule6. Compute weight change by descent rule (repeat for all weights)

• Note that each expression involves data local to a particular unit, we don't have to look around summing things over the whole network.

• It is for this reason, simplicity, locality and, therefore, efficiency that backpropagation has become the dominant paradigm for training neural nets.

Backpropagation Example

y3

z3

y1

z1

y2

z2

1 2

3

-1 -1x1x2

w01

w03

w02w11 w22

w21

w13 w23

w12

-1

133111

233222

3333

)1(

)1(

))(1(

wyy

wyy

yyyy m

First do forward propagation:Compute zi’s and yi’s.

232323

131313

30303 )1(

yww

yww

rww

222222

121212

20202 )1(

xww

xww

rww

212121

111111

10101 )1(

xww

xww

rww

review

Documents

true toothache

false toothache

cavity ptoothache

boolean variables cavity

pcloudy toothache

unconditional probability

joint distribution

probabilities of atomic