Module #1 contdCenter of a distributionSpread of a distribution

Quartiles5-Number Summary and Boxplot

Outliers

Learning Objectives

By the end of this lecture, you should be able to:

– Recognize how scales, mislabeled axes, etc on charts can be misleading – Describe the two most common statistics to describe the center of a

dataset, and when they should be used– Describe two common statistics used to describe the spread of a

dataset, and when they should be used– Understand boxplots and the 5-number summary– Describe what is meant by an outlier and describe two techniques for

identifying outliers. – Describe and apply the 1.5*IQR rule for outliers

3

Misleading chart through poor choice of scale/axis

4

Death rates from cancer (US, 1945-95)

0

50

100

150

200

250

1940 1950 1960 1970 1980 1990 2000

Years

Death

rate

(per

thousand)

Death rates from cancer (US, 1945-95)

0

50

100

150

200

250

1940 1960 1980 2000

Years

Dea

th r

ate

(per

thou

sand

)

Death rates from cancer (US, 1945-95)

0

50

100

150

200

250

1940 1960 1980 2000

Years

Death

rate

(per

thousand)

A picture is worth a thousand words,

BUT

There is nothing like hard numbers.

Look at the scales.

Scales matterHow you stretch the axes and choose your scales can give a different impression.

Death rates from cancer (US, 1945-95)

120

140

160

180

200

220

1940 1960 1980 2000

Years

Death

rate

(pe

r th

ousan

d)

Outliers• This is a very important topic. • Outliers refer to values that seem somehow ‘extreme’ or well

outside the typical range of values in your dataset.• How to deal with outliers is a very involved subject, and while it

certainly merits much discussion, we will not delve into it too much today.

• Your goal for today is to identify outliers. That is, to develop some ability to look at a number and make a reasonably educated decision as to whether or not that value is an outlier.

• We will discuss two techniques for doing so shortly:– Examination of a histogram– Using the “1.5 * IQR” Rule

5

Describing the center and spread of a distribution

• A distribution is best described through a combination of visuals (e.g. graphs), and numbers.

• Two key numeric descriptions are:– Center: e.g. the mean– Spread (aka Variation)

• Center:– Statistics for describing the center: Mean, Median, Mode

• Mean: Most of us are familiar with the ‘mean’ (average). However, we should typically only use the mean if the dataset has no outliers, and is not highly skewed.

• Median: a better choice for the center of a distribution that has outliers, or is skewed• Mode: Will discuss later

• Spread (Variation)– Statistics for describing the spread: Percentiles, Quartiles, Standard Deviation– We will discuss these shortly

6

7

The mean or arithmetic average

To calculate the average, or mean, add

all values, then divide by the number of

individuals. It is the “center of mass.”

Sum of heights is 1598.3

divided by 25 women = 63.9 inches

58.2 64.059.5 64.560.7 64.160.9 64.861.9 65.261.9 65.762.2 66.262.2 66.762.4 67.162.9 67.863.9 68.963.1 69.663.9

Measure of center: the mean

Heights of 25 women in inches

8

Another measure of center: the median

The median is the midpoint of a distribution—the number such

that half of the observations are smaller and half are larger.

1. Sort observations by size.n = number of observations

______________________________

1 1 0.62 2 1.23 3 1.64 4 1.95 5 1.56 6 2.17 7 2.38 8 2.39 9 2.510 10 2.811 11 2.912 3.313 3.414 1 3.615 2 3.716 3 3.817 4 3.918 5 4.119 6 4.220 7 4.521 8 4.722 9 4.923 10 5.324 11 5.6

n = 24 n/2 = 12

Median = (3.3+3.4) /2 = 3.35

2.b. If n is even, the median is the mean of the two middle observations.

1 1 0.62 2 1.23 3 1.64 4 1.95 5 1.56 6 2.17 7 2.38 8 2.39 9 2.510 10 2.811 11 2.912 12 3.313 3.414 1 3.615 2 3.716 3 3.817 4 3.918 5 4.119 6 4.220 7 4.521 8 4.722 9 4.923 10 5.324 11 5.625 12 6.1

n = 25 (n+1)/2 = 26/2 = 13 Median = 3.4

2.a. If n is odd, the median is observation (n+1)/2 down the list

Survival years for Disease X

9

10

‘Resistant’ is an important term. We say that the median is ‘resistant’ to outliers because the presence of 1 or 2 outliers does not affect the median dramatically. Conversely, the mean is not resistant to outliers.

Consider a series of incomes (in thousands) taken from a graduate classroom:18, 24, 37, 41, 62, 63, 2000

The median income is the middle value in the dataset: $41,000

However, the mean is dramatically higher: $320,000 since the one individual who made $2 million dollars pulls the mean disproportionally in the high direction. As a result, we end up with a ‘center’ value that probably does not truly represent the ‘average’ income of our sample.

So we say that:•The median is resistant to outliers•The mean is not resistant to outliers

11

P

erc

en

t o

f p

eo

ple

dyi

ng

Effect of outliers on the mean and median

4.3x

Without the outliers

2.4x

With the outliers

Note the presence of outliers – those two fortunate people who managed to live several years longer than the others. These two large values moved the mean up from 3.4 to 4.2However, the median , the number of years it takes for half the people to die only went from 3.4 to 3.6. Note that this says that the median is fairly resistant, but not 100% resistant. The median is not sensitive to the size of the outlier, rather, iIt is sensitive to the number of outliers.This is typical behavior for the mean and median. The mean is sensitive to outliers, because when you add all the values up to get the mean the outliers are weighted disproportionately by their large size.However, when you get the median, they are just another two points to count –the actual size of those values does not affect things.

12

Measures of spread / variation Most people intuitively ‘get’ the benefit of knowing the center of a distribution

(e.g. the ‘average’ salary of first-year doctors). However, a piece of data that is sadly neglected but is EVERY bit as important, is the spread of the data (also known as the variation).

Just as there are different ways of describing the center of a distribution (e.g. mean, median, mode), there are different techniques for describing the spread of a distribution.

As with the center, you must know which description of the spread is the best of the most accurate tool for describing the spread.

Common techniques for describing the variation in a dataset: Range: the highest and lowest values in the dataset. Important, but

outliers can give people a highly inaccurate picture (imagine if you looked at the range of salaries).

Quartiles – dividing the range into four Standard Deviation / Variance: this is one of the most effective means

of describing the spread, and a tool that we will come back to constantly throughout this course.

Percentiles and Quartiles

• The xth percentile (e.g. the 38th percentile) is the value at which ‘x’ percent of observations fall below it. – Example: If your height is said to be in the 80th percentile, it means that 80%

of the people measured were shorter than you.

• Two commonly used percentiles are the first quartile and the third quartile. These refer to the 25th and 75th percentiles respectively. – Q1 (first quartile): Refers to the 25th percentile. Ie: 25% of observations are

below this value.– Q2 (second quartile): Refers to the 50th percentile. In other words, the

median!– Q3 (third quartile): Refers to the 75th percentile. Ie: 75% of observations fall

below this value.

13

5-Number Summary and Box Plot

• Once you have divided your dataset into quartiles, you now have one technique for creating a neat little summary. It is called the ‘5 Number Summary’ and is made up of:

– Lowest number– First (lower) quartile– Median (not the mean!)– Third (upper) quartile– Highest number

• Once you have this summary in hand, you can even ‘draw’ it using a simple (but very convenient) plot known as a box plot.

15

M = median = 3.4

Q1= first quartile = 2.2

Q3= third quartile = 4.35

1 1 0.62 2 1.23 3 1.64 4 1.95 5 1.56 6 2.17 7 2.38 1 2.39 2 2.510 3 2.811 4 2.912 5 3.313 3.414 1 3.615 2 3.716 3 3.817 4 3.918 5 4.119 6 4.220 7 4.521 1 4.722 2 4.923 3 5.324 4 5.625 5 6.1

Determining the quartiles:

Start by finding the median. (This is Q2).

Then find the middle value between the lowest

number and the median (excluding the median

itself). This is the first quartile, Q1. It is the

value in the sample that has 25% of the

observations (data points) at or below it.

Then find the middle value between the

median and the highest number. This is the

third quartile, Q3. It is the value in the sample

that has 75% of the data at or below it. (It is

the median of the upper half of the sorted data,

excluding M).

Survival time (years)n=25

16

Determining the Five Number Summary

The five number summary is made up of:1. Minimum number2. Q13. Median (Q2)4. Q35. Maximum number

For this dataset, the summary is: 0.6, 2.2, 3.4, 4.35, 6.1

Again, the five number summary is a good tool for summarizing the center and spread of skewed distributions.

M = median = 3.4

Q1= first quartile = 2.2

Q3= third quartile = 4.35

1 1 0.62 2 1.23 3 1.64 4 1.95 5 1.56 6 2.17 7 2.38 1 2.39 2 2.510 3 2.811 4 2.912 5 3.313 3.414 1 3.615 2 3.716 3 3.817 4 3.918 5 4.119 6 4.220 7 4.521 1 4.722 2 4.923 3 5.324 4 5.625 5 6.1

M = median = 3.4

Q3= third quartile = 4.35

Q1= first quartile = 2.2

25 6 6.124 5 5.623 4 5.322 3 4.921 2 4.720 1 4.519 6 4.218 5 4.117 4 3.916 3 3.815 2 3.714 1 3.613 3.412 6 3.311 5 2.910 4 2.89 3 2.58 2 2.37 1 2.36 6 2.15 5 1.54 4 1.93 3 1.62 2 1.21 1 0.6

Largest = max = 6.1

Smallest = min = 0.6

Disease X

0

1

2

3

4

5

6

7

Yea

rs u

nti

l dea

th

Five-number summary:

min Q1 M Q3 max

The boxplot is a graph of the 5-Number summary

BOXPLOT

0123456789

101112131415

Disease X Multiple Myeloma

Yea

rs u

ntil

deat

h

Comparing box plots for a normal and a right-skewed distribution

Boxplots for skewed data

Boxplots remain

true to the data and

depict clearly

symmetry or skew.

OUTLIERS – Identification of the Outlier

At what point do we typically label a datapoint as an outlier? We will discuss

two methods here:

1.One way is to look at a chart and see if any values appear to be “off the

chart” relative to the large majority of values.

2.Another tool is the “1.5 IQR” Rule for outliers.

20

Alaska Florida

Identifying outlier(s) on a histogram

The overall pattern is fairly

symmetrical except for 2

states that clearly do not

belong to the main trend.

Alaska and Florida have

unusual representation of

the elderly in their

population.

A large gap in the

distribution is typically a

sign of an outlier.

Again, we are NOT currently

interested in what to do with

outliers; merely in how to

identify them.

Identification of outliers using the 1.5 IQR Rule

1. Determine the distance between Q1 and Q3 – this is called the

Interquartile Range, or IQR.

2. Multiply by 1.5

3. Determine the distance from the suspicious data point to the nearest

quartile (Q1 or Q3).

4. Determine the distance between Q1 and Q3, called the interquartile

range, or IQR.

5. We call an observation a suspected outlier if it falls more than 1.5 times

the size of the interquartile range (IQR) below the first quartile or above

the third quartile.

This technique is called the “1.5 * IQR rule for outliers.”

22

Example of the 1.5 IQR Rule

Here is the 5-number summary for the dataset discussed earlier: 0.6, 2.2, 3.4, 4.35, 6.1

Would a value of 7.5 be an outlier? What about 8?•IQR = 4.35-2.2 = 2.15•1.5*IQR = 3.23•For a number to be an outlier on the high side, it must be greater than 4.35 +3.23: 7.58•So, 7.5 would not be considered an outlier by this criteria. However, 8 would.

Q1= first quartile = 2.2

Q3= third quartile = 4.35

1 1 0.62 2 1.23 3 1.64 4 1.95 5 1.56 6 2.17 7 2.38 1 2.39 2 2.510 3 2.811 4 2.912 5 3.313 3.414 1 3.615 2 3.716 3 3.817 4 3.918 5 4.119 6 4.220 7 4.521 1 4.722 2 4.923 3 5.324 4 5.625 5 6.1

23

Remember that a histogram does not give you ALL the data - it is merely a summary (albeit a good one!) of the distribution.

However, to be able to do statistics using specific numbers (e.g. to calculate a 5-number summary) you wold need to see the actual dataset.

For this example, I will provide you with Q1 and Q3:Q1: 19.27Q3: 45.40

IQR = 45.40 – 19.27 = 26.131.5*IQR = 39.2

Any amount more than 84.60 is a suspected outlier.

How to deal with OUTLIERSOutliers are data points that require some thought. The first step is to decide whether a data point should indeed

be labeled as an outlier. We will discuss this momentarily. Once you have decided that it is an outlier, the next

question is what you want to do with it.

There are two options for dealing with outliers – you can include them in your analysis, or you can leave them

out.

•Exclude outliers: Suppose you have a datapoint that is extremely high – and you think it was recorded in error.

In this case, you would not want to include this value in your calculations since values like mean and standard

deviation would be thrown off by this bad datapoint.

•However, if you choose to leave out a datapoint, you MUST include in your paper a discussion of your reasons

for doing so.

•Include outliers: The other option, of course, is to include the outlier(s) in your calculations and analysis. In this

case, you have to decide which statistics to use (mean vs median, etc)

•Discussion question: Suppose we wanted to determine the average height of DePaul students and we use our

class as a sample. However, that particular day, we are being visited by an incoming freshman who just happens

to be the tallest person in the world. Would you include him/her in your analysis?

– I would probably leave him out of the analysis since he does not represent the ‘typical’ DePaul student.

– However, when reporting my decision, I MUST report that I did so, and explain my decision.