machine learning in practice lecture 2 carolyn penstein rosé language technologies institute/...

Machine Learning in PracticeLecture 2

Carolyn Penstein RoséLanguage Technologies Institute/ Human-Computer Interaction Institute

Plan for the Day Any questions? Announcements:

First homework assigned

Machine Learning process overview Learn how to use weka Introduce assignment Introduction to Cross-Validation

Overview of Machine Learning Process Skills

Naïve Approach: When all you have is a hammer…

TargetRepresentationData

Naïve Approach: When all you have is a hammer…

TargetRepresentation

Problem: there isn’t one universally best approach!!!!!

Data

Slightly less naïve approach: Aimless wandering…




Problem 1: It takes too long!!!

Data



Problem 2: You might not realize all of the options that are available to you!

Data

Expert Approach: Hypothesis driven




You might end up with the same solution in the end, but you’ll get there faster.

Data



Today we’ll start to learn how!

Data

Warm Up Exercise

Every combination of featurevalues is represented.

Warm Up Exercise


What will happen if youtry to predict HairColor

from the other features?

Warm Up Exercise

Warm Up Exercise

If you don’t have good features,even the most powerful algorithmwon’t be able to learn an accurate

prediction rule.


What will happen if youtry to predict HairColor

from the other features?

Warm Up Exercise

If you don’t have good features,even the most powerful algorithmwon’t be able to learn an accurate

prediction rule.

But that doesn’t mean thisdata set is a hopeless case!

For example, maybe the people wholike red and have brown hair like a different shade of red than the ones

who have blond hair.

So ask yourself: what information might be hidden

or implicit that might allow me to learna rule?

Getting a bit more sophisticated…

Example Data Set

Example Data Set

We’re going to consider a new algorithm

Example Data Set

We’re going to consider a new algorithm

We’re also going to considerdata representation issues

More Complex Algorithm… Two simple algorithms

last time0R – Predict the majority

class1R – Use the most

predictive single feature Today – Intro to

Decision TreesToday we will stay at a

high levelWe’ll investigate more

details of the algorithm next time

* Only makes 2 mistakes!

More Complex Algorithm… Two simple algorithms

last time0R – Predict the majority

class1R – Use the most

predictive single feature Today – Intro to

Decision TreesToday we will stay at a

high levelWe’ll investigate more

details of the algorithm next time

* Only makes 2 mistakes!

What will it do with this example?

Why is it better? Not because it is more complex

Sometimes more complexity makes performance worse

What is different in what the three rule representations assume about your data?0R1RTrees

The best algorithm for your data will give you exactly the power you need





Let’s say you know the rule you are trying to learnis a circle and you have these points. What rulewould you learn?






Now lets say you don’t know the shape, now what would you learn?






Now lets say you don’t know the shape, now what would you learn?If you know the shape, you have fewer degrees

of freedom – less room to make a mistake.

Back to the Opinion Poll Data Set

From http://www.swivel.com/ Example of the kind of data set you could

use for your course projectBetter to find a larger data set

http://www.swivel.com/




Who ran theopinion poll





When the pollwas conducted





Who the Democraticcandidate would be

:





Who the Republicancandidate would be

:





Who is runningagainst who




use for your course projectBetter to find a larger data set Which party

will win





This is what we wantto predict


Do you see any redundant information?

Do you see any missing or hidden information?

How could you expand on what’s here?

Add features thatdescribe the source

How could you expand on what’s here?

Add features that describethings that were going on

during the time when the pollwas taken

How could you expand on what’s here? Add features that

describe personalcharacteristics of the

candidates

What do you think would be the best rule?

What would Weka do with this data?

Using Weka

Start Weka Open up the

Explorer interface

Using Weka Click on Open File

Open OpinionPoll.csv from the Lectures folder

You can save it as a .arff file

Using Weka Click on Open File

Open OpinionPoll.csv from the Lectures folder

You can save it as a .arff file

Summary stats for selected attributes are displayed

Using Weka

Observe interaction between attributes by selecting on interface

Select oneattribute here

Select anotherattribute here

Using Weka

Observe interaction between attributes by selecting on interface

Select oneattribute here

Select anotherattribute here

Based on what you see, do you think thesources of the opinion polls were biased?

Using Weka Go to

Classify Panel

Select a classifier

Using Weka Select a

classifier Select the

predicted value

Using Weka Select a


predicted value

Start the evaluation

Using Weka Select a


predicted value

Start the evaluation

Observe the results

Looking at the Results

Percent correct

Percent correct,controlling for correct by chance

Performance onindividual categories

Confusion matrix

* Right click in Result list and select Save Result Buffer to save performance stats.

Notice the shape of the tree(although the text is too small to read!)

Notice the shape of the tree(although the text is too small to read!)

It’s making its decisionbased only on who the

Republican candidate is.

Why did it do that?

Where will it make mistakes?

Notice the more complex rule if we force binary splits

…

Note that the more complexrule performs worse!!!

More representation issues…

“Gyre” by Eric Rosé

Low resolution image gives some information

Higher resolution image gives more information

But not if the accuracy is bad

But not if the accuracy is bad

Question: Whenmight that happen?

Low resolution gives more information if the accuracy is higher

Assignment 1

Assignment 1

Make sure Weka is set up properly on your machine

Know the basics of using Weka

Information about you…

Information about You Learning goals Priority on learning activities Project goals Programming competence

Cross-Validation

If Outlook = sunny, no else if Outlook = overcast, yes else if Outlook = rainy and Windy = TRUE, no else yes

Performance ontraining data?


Performance ontraining data?

Performance ontesting data?


IMPORTANT!If you evaluate the performanceof your rule on the same data

you trained on, you won’tget an accurate estimate of

how well it will do on new data.

What is cross validation?

What is cross validation?

Notice thatCross validationis for testingonly! Not forbuilding the rule!

But then…..

If we are satisfied with the performance estimate we get

Then we build the model with the WHOLE SET

Now let’s see how it works…

But then…..

If we are satisfied with the performance estimate we get


Now let’s see how it works…

If you are not satisfied with the performance you get,

then you should try to determine what went wrong,

and then evaluate a different model that compensates.

Simple Cross Validation Let’s say your data has

attributes A, B, and C

You want to train a rule to predict D

First train on 2, 3, 4, 5, 6,7 and apply trained model to

1 The results is Accuracy1

1

2

3

4

5

6

7

TEST

TRAIN

TRAIN

TRAIN

TRAIN

TRAIN

TRAIN

Fold: 1






1

2

3

4

5

6

7

TRAIN

TRAIN

TRAIN

TRAIN

TRAIN

TRAIN

TEST

Fold: 2






1

2

3

4

5

6

7

TRAIN

TRAIN

TRAIN

TRAIN

TEST

TRAIN

TRAIN

Fold: 3




First train on 1,2, 3, 5, 6,7 and apply trained model to


1

2

3

4

5

6

7

TRAIN

TRAIN

TRAIN

TEST

TRAIN

TRAIN

TRAIN

Fold: 4






1

2

3

4

5

6

7

TRAIN

TRAIN

TEST

TRAIN

TRAIN

TRAIN

TRAIN

Fold: 5




First train on 1, 2, 3, 4, 5, 7 and apply trained model to


1

2

3

4

5

6

7

TRAIN

TEST

TRAIN

TRAIN

TRAIN

TRAIN

TRAIN

Fold: 6




First train on 1, 2, 3, 4, 5, 6 and apply trained model to 7 The results is Accuracy7 Finally: Average Accuracy1

through Accuracy7

1

2

3

4

5

6

7

TRAIN

TRAIN

TRAIN

TRAIN

TRAIN

TEST

TRAIN

Fold: 7

Remember!

If we are satisfied with the performance estimate we get using cross-validation


We don’t use cross-validation to build the model

Why do we do cross validation? Use cross-validation when you do not have

enough data to have completely independent train and test sets

We are trying to estimate what performance would you get if you trained over your whole set and applied that model to an independent set of the same size

We compute that estimate by averaging over folds

Do we have to do all of the folds?

Yes! The test set on each fold is too small to

give you an accurate estimate of performance alone

Variation across folds Evaluation over part of the data is likely to

be misleading

Why do we do cross validation? Makes the most of your data – large

portion used for training Avoids testing on training data

Testing on training data will over estimate your performance!!!

But if you do multiple iterations of cross-validation, in some ways you are using insights from your testing data in building your model

Questions about cross-validation from in-person students…

How do you decide how many folds? How is data divided between folds? Don’t you need to have a hold-out set to

be totally sure you have a good estimate of performance?

Other questions from in-person students…

Do our class projects have to be classification problems per se?Clustering of pen stroke data

Will we learn to work with time series data in this course?

Questions?

machine learning in practice lecture 2 carolyn penstein rosé language technologies institute/...

Documents