Rugby players, Ballet dancers,

and the

Nearest Neighbour Classifier

COMP24111 lecture 2

A Problem to Solve with Machine Learning

Distinguish rugby players from ballet dancers.

You are provided with a few examples.Fallowfield rugby club (16).Rusholme ballet troupe (10).

TaskGenerate a program which will correctly classify ANY player/dancer in the world.

HintWe shouldn’t “fine-tune” our system too much so it only works on the local clubs.

Taking measurements….

We have to process the people with a computer, so it needs to be in a computer-readable form.

What are the distinguishing characteristics?

1. Height2. Weight3. Shoe size4. Sex

Class, or “label”

Terminology

“Features”

“Examples”

The Supervised Learning Pipeline

ModelModelTesting

Data(no labels)

Testing Data

(no labels)

Training dataand labels

Training dataand labels

Predicted Labels

Learning algorithm

Taking measurements….

Weight

63kg55kg75kg50kg57kg…85kg93kg75kg99kg100kg…

Height

190cm185cm202cm180cm174cm

150cm145cm130cm163cm171cm

height

weight

Person

12345…1617181920…

The Nearest Neighbour RuleWeight

63kg55kg75kg50kg57kg…85kg93kg75kg99kg100kg…

Height

190cm185cm202cm180cm174cm

150cm145cm130cm163cm171cm

height

weight

Person

12345…1617181920…

“TRAINING” DATA

Who’s this guy? - player or dancer?

height = 180cmweight = 78kg

“TESTING” DATA

The Nearest Neighbour RuleWeight

63kg55kg75kg50kg57kg…85kg93kg75kg99kg100kg…

Height

190cm185cm202cm180cm174cm

150cm145cm130cm163cm171cm

height

weight

Person

12345…1617181920…

height = 180cmweight = 78kg

1. Find nearest neighbour

2. Assign the same class

“TRAINING” DATA

Model(memorize the training data)

Model(memorize the training data)

Testing Data

(no labels)

Testing Data

(no labels)

Training dataTraining data

Predicted Labels

Learning algorithm(do nothing)

Supervised Learning Pipeline for Nearest Neighbour

The K-Nearest Neighbour ClassifierWeight

63kg55kg75kg50kg57kg…85kg93kg75kg99kg100kg…

Height

190cm185cm202cm180cm174cm

150cm145cm130cm163cm171cm

Person

12345…1617181920…

Testing point x

For each training datapoint x’

measure distance(x,x’)

End

Sort distances

Select K nearest

Assign most common class

“TRAINING” DATA

height

weight

Quick reminder: Pythagoras’ theorem

. . .

measure distance(x,x’)

. . .

height

weight

i

ii xxxx 2)'()',(distance

a.k.a. “Euclidean” distancea

cb 22

222

.... bacSo

cba

The K-Nearest Neighbour ClassifierWeight

63kg55kg75kg50kg57kg…85kg93kg75kg99kg100kg…

Height

190cm185cm202cm180cm174cm

150cm145cm130cm163cm171cm

Person

12345…1617181920…

Testing point x

For each training datapoint x’

measure distance(x,x’)

End

Sort distances

Select K nearest

Assign most common class

“TRAINING” DATA

height

weight

Seems sensible.

But what are the disadvantages?

The K-Nearest Neighbour ClassifierWeight

63kg55kg75kg50kg57kg…85kg93kg75kg99kg100kg…

Height

190cm185cm202cm180cm174cm

150cm145cm130cm163cm171cm

Person

12345…1617181920…

“TRAINING” DATA

height

weight

Here I chose k=3.

What would happen if I chose k=5?

What would happen if I chose k=26?

The K-Nearest Neighbour ClassifierWeight

63kg55kg75kg50kg57kg…85kg93kg75kg99kg100kg…

Height

190cm185cm202cm180cm174cm

150cm145cm130cm163cm171cm

Person

12345…1617181920…

“TRAINING” DATA

height

weight

Any point on the left of this “boundary” is closer to the red circles.

Any point on the right of this “boundary” is closer to the blue crosses.

This is called the “decision boundary”.

Where’s the decision boundary?

height

weight

Not always a simple straight line!

Where’s the decision boundary?

height

weight

Not always contiguous!

The most important concept in Machine Learning

Looks good so far…

The most important concept in Machine Learning

Looks good so far…

Oh no! Mistakes!What happened?

The most important concept in Machine Learning

Looks good so far…

Oh no! Mistakes!What happened?

We didn’t have all the data.

We can never assume that we do.

This is called “OVER-FITTING”to the small dataset.

The most important concept in Machine Learning

So, we have our first “machine learning” algorithm… ?

Testing point x

For each training datapoint x’

measure distance(x,x’)

End

Sort distances

Select K nearest

Assign most common class

The K-Nearest Neighbour Classifier

Make your own notes on its advantages / disadvantages.

I will ask for volunteers next time we meet…..

Model(memorize the training data)

Model(memorize the training data)

Testing Data

(no labels)

Testing Data

(no labels)

Training dataTraining data

Predicted Labels

Learning algorithm(do nothing)

Pretty dumb! Where’s the learning!

Now, how is this problem like

handwriting recognition?

height

weight

Let’s say the measurements are pixel values.

pixel 2 value

pixel 1 value

25500

255

(190, 85)

A two-pixel image

Three dimensions…A three-pixel image

This 3-pixel image is represented bya SINGLE point in a 3-D space.

pixel 2

pixel 1

pixel 3

(190, 85, 202)

pixel 2

pixel 1

pixel 3

(190, 85, 202)

A three-pixel image

(25, 150, 75)

Another 3-pixel image

Straight line distance between them?

Distance between images

pixel 2

pixel 1

pixel 3

(190, 85, 202)

A three-pixel image

A four-pixel image.

A five-pixel image

4-dimensional space? 5-d? 6-d?

A four-pixel image. A different four-pixel image.

Assuming we read pixels in a systematic manner, we can now represent any image as a single point in a high dimensional space.

(190, 85, 202, 10)Same 4-dimensional vector!

(190, 85, 202, 10)

16 x 16 pixel image. How many dimensions?

?

We can measure distance in 256 dimensional space.

256

1

2)'()',(distancei

iii xxxx

maybe

maybe

probablynot

Which is the nearest neighbour to our ‘3’ ?

AT&T Research LabsThe USPS postcode reader – learnt from examples.

FAST recognition

NOISE resistant, can generalise to future

unseen patterns

Your lab exercise

Use K-NN to recognise handwritten USPS digits.

Biggest mistake by students in last year’s class?

Final lab session before reading week.(see website for details)

i.e. you have 4 weeks.

…starting 2 weeks from now!