version spaces

Machine LearningMachine Learning

Version Spaces LearningVersion Spaces Learning

22

Neural Net approachesNeural Net approaches

Symbolic approaches:Symbolic approaches: version spacesversion spaces decision treesdecision trees knowledge discovery knowledge discovery data miningdata mining speed up learningspeed up learning inductive learninginductive learning … …

Introduction:Introduction: Very many approaches to machine learning:Very many approaches to machine learning:

Version SpacesVersion Spaces

A A concept learningconcept learning technique based on technique based on refining modelsrefining models of the world. of the world.

44

Concept LearningConcept Learning Example:Example:

a student has the following observations about a student has the following observations about having an allergic reaction after meals:having an allergic reaction after meals:

RestaurantRestaurant MealMeal DayDay Cost Cost ReactionReaction

Alma 3Alma 3 breakfastbreakfast FridayFriday cheapcheapYesYesDe MoeteDe Moete lunchlunch FridayFriday expensiveexpensive NoNoAlma 3Alma 3 lunchlunch SaturdaySaturday cheapcheapYesYesSedesSedes breakfastbreakfast SundaySunday cheapcheap NoNoAlma 3Alma 3 breakfastbreakfast SundaySunday expensiveexpensiveNoNo

++--++----

Concept to learnConcept to learn: under which circumstances do : under which circumstances do I get an allergic reaction after meals ??I get an allergic reaction after meals ??

55

In generalIn general There is a set of all possible events:There is a set of all possible events:

Example:Example:

RestaurantRestaurant MealMeal DayDay Cost Cost

3 X 3 X 7 X 2 = 1263 X 3 X 7 X 2 = 126

Reaction:Reaction: Restaurant X Meal X Day X Cost Restaurant X Meal X Day X Cost --> Bool--> Bool

There is a boolean function (implicitly) defined There is a boolean function (implicitly) defined on this set.on this set. Example:Example:

Find an inductive ‘guess’ of the concept, thatFind an inductive ‘guess’ of the concept, thatcovers all the examplescovers all the examples!!

We have the value of this function for SOME We have the value of this function for SOME examples only.examples only.

66

Pictured:Pictured:

Set of all possibleSet of all possible eventsevents

Given some Given some examples:examples:

++++ ++

++--

--

--

--

Find a concept that covers all the positive Find a concept that covers all the positive examples and none of the negative ones!examples and none of the negative ones!

77

Non-determinismNon-determinism

Many different ways to solve this!Many different ways to solve this!


++++ ++

++--

--

--

--

How to choose ?How to choose ?

88

An obvious, but bad choice:An obvious, but bad choice:

The concept IS:The concept IS:

Alma 3Alma 3 andand breakfast breakfast andand Friday Friday andand cheap cheap OROR Alma 3 Alma 3 andand lunch lunch andand Saturday Saturday andand cheap cheap

Does NOT generalize the examples any !!Does NOT generalize the examples any !!



--

----

++

++

99

Pictured:Pictured:

Only the positive examples are positive !Only the positive examples are positive !


++++ ++

++--

--

--

--

1010

Equally bad is:Equally bad is:

The concept is anything EXCEPTThe concept is anything EXCEPT::

De Moete De Moete andand lunch lunch andand Friday Friday and and expensiveexpensiveANDAND

Sedes Sedes andand breakfast breakfast andand Sunday Sunday andand cheap cheapANDAND

Alma 3 Alma 3 andand breakfast breakfast and and Sunday Sunday andand expensiveexpensive



++--++----

1111

Pictured:Pictured:

Everything except the negative examples are Everything except the negative examples are positive:positive:


++++ ++

++--

--

--

--

1212

Solution: Solution: fix a language of hypotheses:fix a language of hypotheses:

We introduce a fix language of concept We introduce a fix language of concept descriptions.descriptions.

= hypothesis space= hypothesis space

The concept can only be identified as being one of The concept can only be identified as being one of the hypotheses in this languagethe hypotheses in this language

avoids the problem of having ‘useless’ conclusionsavoids the problem of having ‘useless’ conclusions forces some generalization/induction to cover more forces some generalization/induction to cover more

than just the given examples. than just the given examples.

1313

most general hypothesis: most general hypothesis: [ ?, ?, ?, ?][ ?, ?, ?, ?]

Reaction - Example:Reaction - Example:

RestaurantRestaurant MealMeal DayDay Cost Cost Reaction Reaction

Alma 3Alma 3 breakfastbreakfast FridayFriday cheapcheap YesYesDe MoeteDe Moete lunchlunch FridayFriday expensiveexpensive NoNoAlma 3Alma 3 lunchlunch SaturdaySaturday cheapcheap YesYesSedesSedes breakfastbreakfast SundaySunday cheapcheap NoNoAlma 3Alma 3 breakfastbreakfast SundaySunday expensiveexpensive NoNo

++--++----

Every hypothesis is a 4-tuple:Every hypothesis is a 4-tuple:

maximally specific: ex.: maximally specific: ex.: [ Sedes, lunch, Monday, cheap][ Sedes, lunch, Monday, cheap] combinations of ? and values are allowed: ex.:combinations of ? and values are allowed: ex.:

[ De Moete, ?, ?, expensive][ De Moete, ?, ?, expensive] or or [ ?, lunch, ?, ?][ ?, lunch, ?, ?]

One more hypothesis: One more hypothesis: (bottom = denotes no example) (bottom = denotes no example)

1414

Hypotheses relate to Hypotheses relate to sets of possible events sets of possible events

EventsEvents

x2x2

x1x1

x1 = x1 = < < Alma 3, lunch, Monday, expensive>Alma 3, lunch, Monday, expensive>x2 =x2 =< Sedes, lunch, Sunday, cheap>< Sedes, lunch, Sunday, cheap>

h1 h1 = [?, lunch, Monday, ?]= [?, lunch, Monday, ?]h2 h2 = [?, lunch, ?, cheap]= [?, lunch, ?, cheap]h3 h3 = [?, lunch, ?, ?]= [?, lunch, ?, ?]

GeneralGeneral

SpecificSpecific

HypothesesHypotheses

h2h2

h3h3

h1h1

1515

Expressive power of this Expressive power of this hypothesis language:hypothesis language:

Conjunctions of explicit, individual propertiesConjunctions of explicit, individual properties

Examples:Examples: [?, lunch, Monday, ?][?, lunch, Monday, ?] : Meal = lunch : Meal = lunch Day = Monday Day = Monday

[?, lunch, ?, cheap][?, lunch, ?, cheap] : Meal = lunch : Meal = lunch Cost = cheap Cost = cheap [?, lunch, ?, ?][?, lunch, ?, ?] : Meal = lunch : Meal = lunch

In addition to the 2 special hypotheses:In addition to the 2 special hypotheses: [ ?, ?, ?, ?][ ?, ?, ?, ?] : anything : anything

: nothing: nothing

1616

Other languages of hypotheses Other languages of hypotheses are allowedare allowed

Example:Example: identify the color of a given sequence identify the color of a given sequence of colored objects.of colored objects. the examples:

redred : : ++ purplepurple : : -- blueblue : : ++

any_colorany_color

mono_colormono_color polyo_colorpolyo_color

redred blue blue greengreen orangeorange purplepurple

A useful language of hypotheses:A useful language of hypotheses:

1717

Important about hypothesis Important about hypothesis languages:languages:

They should have a specific <-> general They should have a specific <-> general ordering.ordering.

Corresponding to the set-inclusion Corresponding to the set-inclusion of the events they cover.of the events they cover.

1818

Defining Concept Learning:Defining Concept Learning: Given:Given:

A set A set XX of possible events : of possible events : Ex.: Eat-events: <Restaurant, Meal, Day, Cost>Ex.: Eat-events: <Restaurant, Meal, Day, Cost>

An (unknown) target functionAn (unknown) target function c:c: X X -> {-, +}-> {-, +} Ex.: Reaction: Eat-events -> { -, +}Ex.: Reaction: Eat-events -> { -, +}

A language of hypothesesA language of hypotheses H H Ex.: conjunctions: [ ?, lunch, Monday, ?]Ex.: conjunctions: [ ?, lunch, Monday, ?]

A set of training examplesA set of training examples DD, with their value under, with their value under cc Ex.: (<Alma 3, breakfast,Friday,cheap>,+) , …Ex.: (<Alma 3, breakfast,Friday,cheap>,+) , …

Find:Find: A hypothesisA hypothesis h h inin H H such that for allsuch that for all xx in in DD::

xx is covered byis covered by h h c(c(xx) = + ) = +

1919

The inductive learning The inductive learning hypothesis:hypothesis:

IfIf a hypothesis approximates the target function a hypothesis approximates the target functionwell well over a sufficiently large number of examplesover a sufficiently large number of examples,,

thenthen the hypothesis will also approximate the the hypothesis will also approximate the target function well target function well on other unobserved exampleson other unobserved examples..

2020

Find-S:Find-S:a naïve algorithma naïve algorithm

InitializeInitialize: : h := h :=

ForFor each each positivepositive training example training example xx in in DD Do:Do:IfIf hh does does notnot cover cover xx::

Replace Replace hh by a minimal generalization by a minimal generalizationof of hh that covers that covers xx

ReturnReturn hh

2121

Initially:Initially: h = h =

Reaction example:Reaction example:

Example 1:Example 1: h = h = [Alma 3,breakfast,Friday,cheap][Alma 3,breakfast,Friday,cheap]

minimal generalizationsminimal generalizations= the individual events= the individual events

Example 2:Example 2: h = h = [Alma 3, [Alma 3, ??, , ??,cheap],cheap]

Generalization =Generalization =replace something by ?replace something by ?

no more positive examples: return no more positive examples: return hh


Alma 3Alma 3 breakfastbreakfast FridayFriday cheapcheap YesYesDe MoeteDe Moete lunchlunch FridayFriday expensiveexpensive NoNoAlma 3Alma 3 lunchlunch SaturdaySaturday cheapcheapYesYesSedesSedes breakfastbreakfast SundaySunday cheapcheap NoNoAlma 3Alma 3 breakfastbreakfast SundaySunday expensiveexpensive NoNo

++--++----

2222

Properties of Find-SProperties of Find-S

BethBeth JoJo AlexAlex

HackerHacker ScientistScientist FootballplayerFootballplayer

HH

Non-deterministicNon-deterministic:: Depending on H, there maybe several minimal Depending on H, there maybe several minimal

generalizations: generalizations:

Beth can be minimally generalized in 2 ways to includeBeth can be minimally generalized in 2 ways to includea new example Jo.a new example Jo.

2323

Properties of Find-S (2)Properties of Find-S (2)

May pick incorrect hypothesis (w.r.t. the May pick incorrect hypothesis (w.r.t. the negative examples):negative examples):

D :D : BethBethAlexAlexJoJo

++--++

BethBeth JoJo AlexAlex

HackerHacker ScientistScientist FootballplayerFootballplayer

HH

Is a wrong solutionIs a wrong solution

2424

Properties of Find-S (3)Properties of Find-S (3)

Cannot detect inconsistency of the training data:Cannot detect inconsistency of the training data:

D :D : BethBethJo Jo BethBeth

++++--

D :D : BethBethAlex Alex JoJo

++--++ JoJo AlexAlex

ScientistScientist

HH

BethBeth

Nor inability of the language H to learn the concept:Nor inability of the language H to learn the concept:

2525

Nice about Find-S:Nice about Find-S: It doesn’t have to remember previous examples ! It doesn’t have to remember previous examples !

If If hh already covered the 20 first examples, already covered the 20 first examples, then then h’h’ will as well will as well

hh

XX HH

If the previousIf the previous h h already covered all previous examples, already covered all previous examples, then a minimal generalization then a minimal generalization h’h’ will too ! will too !

h’h’

2626

Dual Find-S:Dual Find-S:

InitializeInitialize: : h := h := [ ?, ?, .., ?][ ?, ?, .., ?]

ForFor each each negativenegative training example training example xx in in DD Do:Do:IfIf hh does cover does cover xx::

Replace Replace hh by a minimal specialization by a minimal specializationof of hh that does that does notnot cover cover xx

ReturnReturn hh

2727

Reaction example:Reaction example:



++--++----

Initially:Initially: h = [ ?, ?, ?, ?]h = [ ?, ?, ?, ?]

Example 1:Example 1:

h= [ ?, breakfast, ?, ?]h= [ ?, breakfast, ?, ?]


h= [ Alma 3, breakfast, ?, ?]h= [ Alma 3, breakfast, ?, ?]


h= [ Alma 3, breakfast, ?, cheap]h= [ Alma 3, breakfast, ?, cheap]

2828

Perform both Find-S and Dual Find-S: Perform both Find-S and Dual Find-S: Find-S deals with positive examplesFind-S deals with positive examples Dual Find-S deals with negative examplesDual Find-S deals with negative examples

Version Spaces: the idea: Version Spaces: the idea:

BUT:BUT:do do NOTNOT select 1 minimal generalization or select 1 minimal generalization or

specialization at each step,specialization at each step,

but keep track of but keep track of ALLALL minimal minimal generalizations generalizations or specializationsor specializations

2929

Version spaces:Version spaces:initializationinitialization

The version spaces The version spaces GG and and SS are initialized to be are initialized to be the smallest and the largest hypotheses only.the smallest and the largest hypotheses only.

G = { [ ?, ?, …, ?] }G = { [ ?, ?, …, ?] }

S = {S = { } }

3030

Negative examples:Negative examples:

Replace the top hypothesis by ALL minimal Replace the top hypothesis by ALL minimal specializations that specializations that DO NOTDO NOT cover the negative cover the negative example.example.

InvariantInvariant: only the hypotheses : only the hypotheses more specificmore specific than the than the ones of ones of GG are still possible: are still possible:

they don’t cover the negative examplethey don’t cover the negative example

S = {S = { } }

G = { [ ?, ?, …, ?] }G = { [ ?, ?, …, ?] }

G = {h1, h2, …, hn}G = {h1, h2, …, hn}

3131

Positive examples:Positive examples:

Replace the bottom hypothesis by ALL minimal Replace the bottom hypothesis by ALL minimal generalizations that generalizations that DODO cover the positive example. cover the positive example.

InvariantInvariant: only the hypotheses : only the hypotheses more generalmore general than the than the ones of ones of SS are still possible: are still possible:

they do cover the positive examplethey do cover the positive example

G = {h1, h2, …, hn}G = {h1, h2, …, hn}

S = {S = { }}

S = {S = {h1’, h2’, …,hm’ h1’, h2’, …,hm’ } }

3232

Later: negative examplesLater: negative examples

Replace the all hypotheses in Replace the all hypotheses in GG that cover a next that cover a next negative example by ALL minimal specializations negative example by ALL minimal specializations that that DO NOTDO NOT cover the negative example. cover the negative example.

InvariantInvariant: only the hypotheses : only the hypotheses more specificmore specific than the than the ones of ones of GG are still possible: are still possible:

they don’t cover the negative examplethey don’t cover the negative example

S = {S = {h1’, h2’, …,hm’ h1’, h2’, …,hm’ } }

G = {h1, h2, …, hn}G = {h1, h2, …, hn}

G = {h1, h21, h22, …, hn}G = {h1, h21, h22, …, hn}

3333

Later: positive examplesLater: positive examples

Replace the all hypotheses in Replace the all hypotheses in SS that do not that do not cover a next positive example by ALL minimal cover a next positive example by ALL minimal generalizations that generalizations that DODO cover the example. cover the example.

InvariantInvariant: only the hypotheses : only the hypotheses more generalmore general than the than the ones of ones of SS are still possible: are still possible:

they do cover the positive examplethey do cover the positive example

G = {h1, h21, h22, …, hn}G = {h1, h21, h22, …, hn}

S = {S = {h1’, h2’, h1’, h2’, …,hm’ …,hm’ } }

S = {S = {h11’, h12’, h13’, h2’, …,hm’ h11’, h12’, h13’, h2’, …,hm’ } }

3434

Optimization: negative:Optimization: negative:

Only consider specializations of elements in Only consider specializations of elements in GG that are still more general than some specific that are still more general than some specific hypothesis (in hypothesis (in SS))

G = {h1, h21, …, hn}G = {h1, h21, …, hn}

S = {S = {h1’, h2’, …,hm’ h1’, h2’, …,hm’ } }

InvariantInvariant: on : on SS used ! used !

… … are more general than ...are more general than ...

3535

Optimization: positive:Optimization: positive:

Only consider generalizations of elements in Only consider generalizations of elements in SS that are still more specific than some general that are still more specific than some general hypothesis (in hypothesis (in GG))

InvariantInvariant: on : on GG used ! used !

G = {h1, h21, h22, …, hn}G = {h1, h21, h22, …, hn}

S = {S = {h13’, h2’, …,hm’ h13’, h2’, …,hm’ } }

… … are more specific than ...are more specific than ...

3636

Pruning: negative examples Pruning: negative examples

The new negative example can also be used to The new negative example can also be used to prune all the prune all the SS - hypotheses that cover the - hypotheses that cover the negative example.negative example.

G = {h1, h21, …, hn}G = {h1, h21, …, hn}

S = {S = {h1’, h3’, …,hm’ h1’, h3’, …,hm’ } }

InvariantInvariant only works for the previous examples, not the last one only works for the previous examples, not the last one

Cover the last negative example!Cover the last negative example!

3737

Pruning: positive examples Pruning: positive examples

The new positive example can also be used to The new positive example can also be used to prune all the prune all the GG - hypotheses that do not cover - hypotheses that do not cover the positive example.the positive example.

G = {h1, h21, …, hn}G = {h1, h21, …, hn}

S = {S = {h1’, h3’, …,hm’ h1’, h3’, …,hm’ } }

Don’t cover the last positive exampleDon’t cover the last positive example

3838

Eliminate redundant hypothesesEliminate redundant hypotheses

If a hypothesis from If a hypothesis from GG is more specific than is more specific than another hypothesis from another hypothesis from GG: eliminate it !: eliminate it !

G = {h1, h21, …, hn}G = {h1, h21, …, hn}

S = {S = {h1’, h3’, …,hm’ h1’, h3’, …,hm’ } }

More specific than another general hypothesisMore specific than another general hypothesis

Reason:Reason: Invariant acts as a wave frontInvariant acts as a wave front: anything above : anything above GG is not is notallowed. The most general elements of allowed. The most general elements of GG define the real boundary define the real boundary

Obviously also for Obviously also for SS ! !

3939

Convergence:Convergence:

Eventually, if Eventually, if GG and and SS MAY get a common element: MAY get a common element: Version Version Spaces has Spaces has converged to a solutionconverged to a solution..

Remaining examples need to be verified for the solutionRemaining examples need to be verified for the solution..

G = {h1, h21, h22, …, hn}G = {h1, h21, h22, …, hn}

S = {S = {h13’, h2’, …,hm’ h13’, h2’, …,hm’ } }

4040

Reaction exampleReaction example

Initialization:Initialization:

[ ?, ?, ?, ?][ ?, ?, ?, ?]

MostMostgeneralgeneral

MostMostspecificspecific

4141

Alma3, breakfast, Friday, cheap: Alma3, breakfast, Friday, cheap: ++

Positive example: minimal generalization ofPositive example: minimal generalization of

[ ?, ?, ?, ?][ ?, ?, ?, ?]

[Alma3, breakfast, Friday, cheap][Alma3, breakfast, Friday, cheap]

4242

DeMoete, lunch, Friday, expensive: DeMoete, lunch, Friday, expensive: - - Negative example: minimal specialization of Negative example: minimal specialization of [ ?, ?, ?, ?][ ?, ?, ?, ?]

15 possible specializations !! 15 possible specializations !!

[[Alma3Alma3, ?, ?, ?], ?, ?, ?][[DeMoeteDeMoete, ?, ?, ?] , ?, ?, ?] [[SedesSedes, ?, ?, ?], ?, ?, ?][?, [?, breakfastbreakfast, ?, ?], ?, ?][?, [?, lunchlunch, ?, ?], ?, ?][?, [?, dinnerdinner, ?, ?], ?, ?][?, ?, [?, ?, MondayMonday, ?], ?][?, ?, [?, ?, TuesdayTuesday, ?], ?][?, ?, [?, ?, WednesdayWednesday, ?], ?][?, ?, [?, ?, ThursdayThursday, ?], ?][?, ?, [?, ?, FridayFriday, ?], ?][?, ?, [?, ?, SaturdaySaturday, ?], ?][?, ?, [?, ?, SundaySunday, ?], ?][?, ?, ?, [?, ?, ?, cheapcheap]][?, ?, ?, [?, ?, ?, expensiveexpensive]]

Matches the negative exampleMatches the negative exampleXX

XX

XX

XX

Does not generalize the specific modelDoes not generalize the specific model

Specific model:Specific model:[Alma3, breakfast, Friday, cheap][Alma3, breakfast, Friday, cheap]

XX

XXXXXXXXXX

XXXX

Remain !Remain !

4343

Result after example 2:Result after example 2:

[ ?, ?, ?, ?][ ?, ?, ?, ?]


[[Alma3Alma3, ?, ?, ?], ?, ?, ?] [?, [?, breakfastbreakfast, ?, ?], ?, ?] [?, ?, ?, [?, ?, ?, cheapcheap]]

4444

Alma3, lunch, Saturday, cheap: Alma3, lunch, Saturday, cheap: ++

Positive example: minimal generalization of Positive example: minimal generalization of [Alma3, [Alma3, breakfast, Friday, cheap]breakfast, Friday, cheap]


[Alma3, ?, ?, ?][Alma3, ?, ?, ?] [?, breakfast, ?, ?][?, breakfast, ?, ?] [?, ?, ?, cheap][?, ?, ?, cheap]

[Alma3, ? , ? , cheap][Alma3, ? , ? , cheap]

does not match new exampledoes not match new example

4545

Sedes, breakfast, Sunday, cheap: Sedes, breakfast, Sunday, cheap: - - Negative example: minimal specialization of the Negative example: minimal specialization of the

general models:general models:

[Alma3, ?, ?, ?][Alma3, ?, ?, ?] [?, ?, ?, cheap][?, ?, ?, cheap]


[[Alma3Alma3, ?, ?, cheap], ?, ?, cheap]

The only specializationThe only specializationthat is introduced is that is introduced is pruned, because it ispruned, because it ismore specific than more specific than

another general hypothesisanother general hypothesis

4646

Alma 3, breakfast, Sunday, Alma 3, breakfast, Sunday, expensive: -expensive: - Negative example: minimal specialization of Negative example: minimal specialization of

[Alma3, ?, ?, ?][Alma3, ?, ?, ?]

[Alma3, ?, ?, ?][Alma3, ?, ?, ?]


[Alma3, ?, ?, [Alma3, ?, ?, cheapcheap]]

Same hypothesis !!!Same hypothesis !!!

Cheap food at Alma3Cheap food at Alma3produces the allergy !produces the allergy !

4747

Version Space Algorithm:Version Space Algorithm:

InitiallyInitially: : G := { the hypothesis that covers everything}G := { the hypothesis that covers everything} S := {S := {}}

For each new For each new positivepositive example: example:

GeneralizeGeneralize all hypotheses in all hypotheses in SS that do not cover the that do not cover the example yet, but ensure the following:example yet, but ensure the following:

- Only introduce - Only introduce minimal changesminimal changes on the hypotheses. on the hypotheses.

- - EachEach new specific hypothesis is a new specific hypothesis is a specialization ofspecialization of some general hypothesissome general hypothesis..

- - NoNo new specific hypothesis is a new specific hypothesis is a generalization ofgeneralization of some other specific hypothesissome other specific hypothesis..

Prune awayPrune away all hypotheses in all hypotheses in GG that do not cover the that do not cover the example.example.

4848

Version Space Algorithm (2):Version Space Algorithm (2):......For each new For each new negativenegative example:example:SpecializeSpecialize all hypotheses in all hypotheses in GG that cover the example, that cover the example,

but ensure the following:but ensure the following:

- Only introduce - Only introduce minimal changesminimal changes on the hypotheses. on the hypotheses.

- - EachEach new general hypothesis is a new general hypothesis is a generalization ofgeneralization of some specific hypothesissome specific hypothesis..

- - NoNo new general hypothesis is a new general hypothesis is a specialization ofspecialization of some other general hypothesissome other general hypothesis..

Prune awayPrune away all hypotheses in all hypotheses in SS that cover the example. that cover the example.

Until there are no more examples:Until there are no more examples: report report SS and and GG

OR OR SS or or GG become empty become empty: report failure: report failure

4949

Properties of VS:Properties of VS:

Symmetry:Symmetry: positive and negative examples are dealt with in a positive and negative examples are dealt with in a

completely dualcompletely dual way. way.

Does not need to remember previous examples.Does not need to remember previous examples.

Noise:Noise: VS VS cannotcannot deal with noise ! deal with noise !

If a positive example is given to be negativeIf a positive example is given to be negativethen VS then VS eliminateseliminates the desired the desired

hypothesis hypothesis from the Version Space G !from the Version Space G !

5050

Termination:Termination: If it terminates because of “no more examples”:If it terminates because of “no more examples”:

[Alma3, ?, ?, ?][Alma3, ?, ?, ?] [?, ?, Monday,?][?, ?, Monday,?]

[Alma3, ?, Monday, cheap][Alma3, ?, Monday, cheap]

[Alma3, ?, ?, cheap][Alma3, ?, ?, cheap] [?, ?, Monday, cheap][?, ?, Monday, cheap][Alma3, ?, Monday, ?][Alma3, ?, Monday, ?]

Then Then all these hypothesesall these hypotheses, and , and all intermediate hypothesesall intermediate hypotheses,,are still correct descriptions covering the test data.are still correct descriptions covering the test data.

VS makes NO unnecessary choices !VS makes NO unnecessary choices !

ExampleExample (spaces on termination): (spaces on termination):

5151

Termination (2):Termination (2): If it terminates because of If it terminates because of SS or or GG being empty: being empty:

Then either:Then either: The data is inconsistent (noise?)The data is inconsistent (noise?) The target concept cannot be represented in The target concept cannot be represented in

the the hypothesis-language H.hypothesis-language H.

[Alma3, breakfast, ?, cheap] [Alma3, breakfast, ?, cheap] [Alma3, lunch, ?, cheap] [Alma3, lunch, ?, cheap]

<Alma3, dinner, Sunday, cheap> <Alma3, dinner, Sunday, cheap> --<Alma3, breakfast, Sunday, cheap> <Alma3, breakfast, Sunday, cheap> ++<Alma3, lunch, Sunday, cheap> <Alma3, lunch, Sunday, cheap> ++

Example:Example: target concept is:target concept is:

given examples like: given examples like:

this cannot be learned in our language H.this cannot be learned in our language H.

5252

Which example next?Which example next? VS can decide itself which example would be VS can decide itself which example would be

most useful next.most useful next.




It can ‘query’ a user for the most relevant It can ‘query’ a user for the most relevant additional classification !additional classification !

Example:Example:

<Alma3,lunch,Monday,<Alma3,lunch,Monday,expensiveexpensive>> classified negative by 3 hypothesesclassified negative by 3 hypothesesclassified positive by 3 hypothesesclassified positive by 3 hypotheses

is the most informative new exampleis the most informative new example

5353

Use of partially learned Use of partially learned concepts:concepts:

Example:Example: <Alma3,lunch,Monday,cheap><Alma3,lunch,Monday,cheap>




can be classified as positivecan be classified as positive is is covered by all remaining hypothesescovered by all remaining hypotheses ! !

it is enough to check that it is covered by the it is enough to check that it is covered by the hypotheses in hypotheses in SS ! (all others generalize these) ! (all others generalize these)

5454

Use of partially learned concepts Use of partially learned concepts (2):(2):

Example:Example: <Sedes,lunch,Sunday,cheap><Sedes,lunch,Sunday,cheap>




can be classified as negativecan be classified as negative is is not covered by any remaining hypothesisnot covered by any remaining hypothesis ! !

it is enough to check that it is not covered by any it is enough to check that it is not covered by any hypothesis in hypothesis in GG ! (all others specialize these) ! (all others specialize these)

5555


Example:Example: <Alma3,lunch,Monday,expensive><Alma3,lunch,Monday,expensive>




can not be classifiedcan not be classified is covered by 3, not covered 3 hypotheses is covered by 3, not covered 3 hypotheses

no conclusionno conclusion

5656


Example:Example: <Sedes,lunch,Monday,expensive><Sedes,lunch,Monday,expensive>




probably does not belong to the concept : ratio : 1/6probably does not belong to the concept : ratio : 1/6

can only be classified with a certain degree of can only be classified with a certain degree of precisionprecision

is covered by 1, not covered 5 hypotheses is covered by 1, not covered 5 hypotheses

5757

The relevance of inductive BIAS: The relevance of inductive BIAS: choosing Hchoosing H

Our hypothesis language L fails to learn some Our hypothesis language L fails to learn some concepts.concepts.

See example :See example :[Alma3, breakfast, ?, cheap] [Alma3, breakfast, ?, cheap] [Alma3, lunch, ?, cheap] [Alma3, lunch, ?, cheap]

What about choosing a more expressive language H?What about choosing a more expressive language H? Assume H’:Assume H’:

allows conjunctions (as before)allows conjunctions (as before) allows allows disjunctiondisjunction and and negationnegation too ! too !

Example:Example:– Restaurant = Alma3 Restaurant = Alma3 ~(Day = Monday) ~(Day = Monday)

5858

Inductive BIAS (2):Inductive BIAS (2): This language H’ allows to represent ANY This language H’ allows to represent ANY

subset of the complete set of all events Xsubset of the complete set of all events X

RestaurantRestaurant MealMeal DayDay Cost Cost

3 X 3 X 7 X 2 = 1263 X 3 X 7 X 2 = 126

But X has 126 elementsBut X has 126 elements we can express 2we can express 2126126 different hypotheses now !different hypotheses now !

5959

Inductive BIAS (3)Inductive BIAS (3) Version Spaces using H’:Version Spaces using H’:



++--++----

??

{~[DeMoete. Lunch, Friday, expensive] {~[DeMoete. Lunch, Friday, expensive] ~[Sedes, breakfast, Sunday, cheap]} ~[Sedes, breakfast, Sunday, cheap]}

{~[DeMoete. Lunch, Friday, expensive] {~[DeMoete. Lunch, Friday, expensive] ~[Sedes, breakfast, Sunday, cheap] ~[Sedes, breakfast, Sunday, cheap] ~[Alma 3, breakfast, Sunday, expensive]}~[Alma 3, breakfast, Sunday, expensive]}

{~[DeMoete. Lunch, Friday, expensive]}{~[DeMoete. Lunch, Friday, expensive]}

{[Alma 3, breakfast, Friday, cheap]{[Alma 3, breakfast, Friday, cheap] [Alma 3, lunch, Saturday, cheap]} [Alma 3, lunch, Saturday, cheap]}

{[Alma 3, breakfast, Friday, cheap]}{[Alma 3, breakfast, Friday, cheap]}

6060

Inductive BIAS (3)Inductive BIAS (3) Resulting Version Spaces:Resulting Version Spaces:

G = {~[DeMoete. Lunch, Friday, expensive] G = {~[DeMoete. Lunch, Friday, expensive] ~[Sedes, breakfast, Sunday, cheap]~[Sedes, breakfast, Sunday, cheap] ~[Alma 3, breakfast, Sunday, expensive]}~[Alma 3, breakfast, Sunday, expensive]}

S = {[Alma 3, breakfast, Friday, cheap]S = {[Alma 3, breakfast, Friday, cheap] [Alma 3, lunch, Saturday, cheap]}[Alma 3, lunch, Saturday, cheap]}

We haven’t learned anythingWe haven’t learned anything merely restated our positive and negative examples !merely restated our positive and negative examples !

In general: in order to be able to learn, we need an inductive BIAS (= assumption):In general: in order to be able to learn, we need an inductive BIAS (= assumption): Example:Example: “ The desired concept CAN be described “ The desired concept CAN be described as a conjunction of features “as a conjunction of features “

6161

Shift of BiasShift of Bias

Practical approach to the Bias problem: Practical approach to the Bias problem:

Start VS with a very weak hypothesis language.Start VS with a very weak hypothesis language.

IfIf the concept is learned: o.k. the concept is learned: o.k. ElseElse

Refine your language and restart VSRefine your language and restart VS

Avoids the choice.Avoids the choice.

Gives the most general concept that can be learned. Gives the most general concept that can be learned.