6.hypothesis testing and the comparison of 2 or more populations asw chapter 9 + chapter 10

6. Hypothesis Testing and the Comparison of 2 or More

Populations

ASW Chapter 9 + Chapter 10

2

A) Introduction Estimating parameters of population

→ hypothesis testing on our model.

Model or Theory Deductions/Analysis

Testable Hypothesis

Data

Reject Theory

Accept Theory(For Now)

Modify Theory

Empirical Result

1

2

34

5

Ch. 1-8, 12, 13

Economic Theory

Ch. 9-11, 12, 13

Tests

3

Confidence Intervals and Hypothesis Testing Confidence intervals → range that μ falls into.

NOW: is μ > 0, or > 1, etc. OR: is μ1 > μ2?

Testing for specific values of μ. We have a confidence interval for Saskatchewan

female wages by age. What could we test here?

We will have a confidence interval for Bachelor’s salaries in Saskatchewan. What could we test here?

Others: gasoline prices? Stock Market fluctuations?

4

B) Developing Null and Alternative Hypotheses Start with a testable hypothesis.

Point of interest: do older women get paid more? Economic theory: is 0 < MPC < 1 and constant?

Define it’s opposite: Older women’s salaries are < average. MPC is > 1.

One is the Null hypothesis, the other is the Alternative hypothesis. Use sample data to test the Null hypothesis.

What if it is not that simple to have 2 opposites?

5

Which is the Null? General rule: the hypothesis with the = sign or the

< or the > sign is the Null. OR: the Null is something we assume is true

unless contradicted by the sample.

6

1. Research hypotheses: Testing an exception to the general rule, so it goes in the

alternative. E.g, testing if older women’s salaries (μ) > average:

H0: μ < μ(average)

HA: μ > μ(average)

Results will tell us either: If testing shows H0 cannot be rejected (“accepted”) →

implies that older women’s salaries are not higher, but we cannot be sure.

If testing shows H0 can be rejected → we can infer HA is true, μ > μ(average).

7

2. Testing the Validity of a Claim Assume claim is true until disproven. E.G.: manufacturer’s claim of weight/container.

H0: μ > 100 grams.

HA: μ < 100 grams.

Results will tell us either: If testing shows H0 cannot be rejected (“accepted”)

→ manufacturers claim not challenged. If testing shows H0 can be rejected → we can infer

manufacturer is lying.

8

3. Testing in Decision-Making Here, if either too high or too low, need to do

something. E.G.: is class length 75 minutes?

H0: μ = 75 minutes.

HA: μ ≠ 75 minutes.

If H0 not rejected (accepted), no change in behaviour.

If H0 rejected –> change behaviour.

9

C) Type I and Type II Errors Sample data → could have errors.

10

Type I and Type II Errors

CorrectCorrectDecisionDecision Type II ErrorType II Error

CorrectCorrectDecisionDecisionType I ErrorType I ErrorRejectReject HH00

AcceptAccept HH00

HH0 0 TrueTrue HH0 0 FalseFalseConclusionConclusion

Population Condition Population Condition

False Negative False Positive

11

False Positive or False Negative?

New claims over bungled shooting of BrazilianBy Mark Sellman, Times Online, and Daniel McGrory

U.K. police defend shoot-to-kill after mistake…Blair said Menezes had emerged from an apartment block in south London that had been under surveillance in connection with Thursday’s attacks, and refused police orders to halt. Menezes had also been wearing an unseasonably heavy coat, further raising police suspicions.MSNBC, July 24th.

12

Other Type I and Type II Errors Sampling songs. Health tests. Pregnancy tests. Jury decisions.

13

Level of Significance Hypothesis testing is really designed to control the

chance of a Type I error. Probability of Type I error = the level of significance.

Selecting (= level of significance ) select probability of Type I error.

What is the level of significance for Jury trials?

We do not control for Type II errors

–> except by our language of stating “do not reject H0”.

14

Level of Significance cont’d is picked by researcher –> normally 5%? = 5% → type I error happens only 5% of the time.

15

D) The probability value (p-value) approach

1. Develop null and alternative hypothesis.

2. Select level of significance .

3. Collect data, calculate sample mean and test statistic.

4. Use test statistic to calculate p-value.

5. Compare: reject H0 if p-value < .

The sample implies that the alternative (your research hypothesis) is true.

p-value approach

16

Hypothesis Testing: The Critical Value Approach

4. Use to determine critical value and rejection rule.

5. Compare: if |test statistic| > |critical value|, reject H0.

The sample implies that the alternative (your research hypothesis) is true.

Critical-value approach

1. Develop null and alternative hypothesis.2. Select level of significance .3. Collect data, calculate sample mean and test

statistic.

17

Hypothesis Testing cont’d… This is essentially inverting our confidence index.

→ Is more than 2 standard deviations away from some benchmark?

18

E) Population Mean, σ Known, One-Tailed Test Same hypothesis

p-value method and critical value method. Example: a new employment program initiative has been

introduced to reduce time spent being unemployed. Goal: 12 weeks or less unemployed.

Population standard deviation believed to be 3.2 weeks. Sample of 40 unemployed workers, average time

unemployed 13.25 weeks. Assuming a level of significance () of .05, is the

program goal being met?

19

First (Common) Steps Hypothesis:

H0: μ < 12HA: μ > 12

Clearly 13.25 > 12. This casts doubt on our program goal (the null), and

whether we should continue it. Key: is it enough more, given sample size and

standard deviation, or is it just a (small) random fluctuation?

20

Computing the Test Statistic Under our assumptions: use the standard normal. Use sample mean to calculate test statistic

13.25 12 2.47

/ 3.2/ 40x

zn

13.25 12 2.47

/ 3.2/ 40x

zn

Is this z big enough to reject the null hypothesis? Next, go our two routes:

Calculate p-value OR z-critical.

21

Calculating the p-value Given the z-value, what is the corresponding

probability?

0 Z=2.47 z

p=??

This is the probability that 13.25 > 12 by chance.

22

Calculating the p-value cont’d Find 2.47 on the Standard Normal Distribution tables:

z .00 .01 .02 .03 .04 .05 .06 .07 .08 .09

…

2.3

2.4 .4918 .4920 .4922 .4925 .4927 .4929 .4931 .4932 .4934 .4936

2.5

…

23

Calculating the p-value cont’d .4932 is the probability of being between 0 and z=2.47.

p = 0.5 – 0.4932 = 0.0038.

0 Z=2.47 z

0.4932

24

Should We Reject the Hypothesis? This says that the probability of getting a sample mean

of 13.25 when the true mean is 12 = .0038 or less than ½ of 1 percent. Our significance level was only 5%, so we reject the

null. We are 99.62% certain that the program has failed.

25

Rejection of the Null

0 Z=2.47 z

0.0038

Z.05

Sometimes we say: “significant at the 0.38% level”.

26

Critical Value Approach This is an alternative you often see in textbooks or

articles. Find the value of z.05, and compare it to the test value of z (2.47).

From the tables, z.05 = 1.645.

Because 2.47 > 1.645, reject H0.

27

Rejection

0 Z=2.47 z

Z.05= 1.645

28

Excel Let’s do this example in Excel.

Look at Appendix 9.2 in text, especially Figure 9.8

29

F) Population Mean, σ Known, Two-Tailed Test Null: μ = μ0 → Alternative is μ ≠ μ0.

Must examine two areas of the distribution. Example:

Price/earnings ratios for stocks. Theory: stable rate of P/E in market = 13.

If P/E (market) < 13, you should invest in the stock market.

If P/E (market) > 13, you should take your money out.

30

Estimate Steps Can we estimate if the population P/E is 13 or not? Common steps:

Set hypothesis:H0: μ = 13HA: μ ≠ 13

Select = .05. Calculate standard error. Calculate z-value.

31

Calculating Test Statistic We have a sample of 50

= 12.1. Historical σ = 3.0456.

4307.050

0456.3Error Standard x

n

0896.24307.0

131.12-x value

x

0

z

32

p-value Approach Calculate the p-value.

We will calculate for the lower tail→ then make an adjustment for the upper tail.

33

p-value, Two-Tailed Test

0 Z=2.09 zZ=–2.09

p(z < –2.09) = ??

p(z > 2.09) = ??

We can just calculate one value, and double it.

34

Calculating the p-value cont’d Find 2.090 on the Standard Normal Distribution tables:

z .00 .01 .02 .03 .04 .05 .06 .07 .08 .09

…

1.9

2.0 .4772 .4778 .4783 .4788 .4793 .4798 .4803 .4808 .4812 .4817

2.1

…

35

p-value, Two-Tailed Test

0 Z=2.09 zZ=–2.09

p(z < –2.09) = ??

p(z > 2.09) = 0.5 -.4817= 0.0183

Doubling the value, we find the p-value = 0.0366

0.4817

36

Should We Reject the Null Hypothesis? Yes!

p-value = 0.0366 < = 0.05. There is only a 3.66% chance that the measured

price/earnings ratio sample mean of 12.1 is not equal to the stable rate of 13 by random chance.

37

Critical Value Approach Reject the null hypothesis if:

test z-value > critical value or if

test z-value < critical value Two tailed test: = 0.05 → need critical value for

/2 = 0.025. The tables tell us that this is 1.96.

38

G) Population Mean, σ Unknown σ unknown → must estimate it with our sample too

→ use t-distribution, n – 1 degrees of freedom.

n

sxError Standard

x

0-x value

t

39

One-tailed Test, p-value Approach Steps:

1. Set up hypothesis.2. Decide on level of significance ()3. Collect data, calculate sample mean and test statistic.4. Use test statistic & t-table/Excel to calculate p-value.5. Compare: reject H0 if p-value < .

40

The RCMP periodically samplesThe RCMP periodically samples

vehicle speeds at various locationsvehicle speeds at various locations

on a particular roadway. on a particular roadway.

The sample of vehicle speedsThe sample of vehicle speeds

is used to test the hypothesisis used to test the hypothesis

Example: Highway Patrol

The locations where The locations where HH00 is rejected are deemed is rejected are deemed

the best locations for radar traps.the best locations for radar traps.

HH00: : << 65 65

41

ExampleExample: Highway Patrol: Highway Patrol

Outside Lumsden:Outside Lumsden:A sample of 64 vehiclesA sample of 64 vehicles

––> average speed = 65.5 mph> average speed = 65.5 mph

––> standard deviation = 4.2 mph. > standard deviation = 4.2 mph.

Use Use = .05 to test the hypothesis. = .05 to test the hypothesis.

42

Common to Both ApproachesCommon to Both Approaches

1. Determine the hypotheses.1. Determine the hypotheses.

2. Specify the level of significance.2. Specify the level of significance.

3. Compute the value of the test statistic.3. Compute the value of the test statistic.

= .05= .05

HH00: : << 65 65

HHaa: : > 65 > 65

9524.64/2.4

655.65-x value

x

0

t

43

4. Estimate the p-value From t-Distribution Table Must “interpolate” the value of tt = 0.9524, df = 65 = 0.9524, df = 65

Degrees of Freedom

Area in Upper Tail

0.20 0.10 0.05 0.025 0.01 0.005

1

…

50 .849 1.299 1.676 2.009 2.403 2.678

60 .848 1.296 1.671 2.000 2.390 2.660

80 .846 1.292 1.664 1.990 2.374 2.639

…

0.9524 .10 < .10 < pp–value < .20–value < .20

44

pp –Value Approach –Value Approach

5. Determine whether to reject 5. Determine whether to reject HH00..

Because Because pp–value –value >> = .05, we do NOT reject = .05, we do NOT reject HH00..

We are at least 95% confident that the mean We are at least 95% confident that the mean speedspeedof vehicles outside Lumsden is LESS than OR of vehicles outside Lumsden is LESS than OR EQUAL TO 65 mph.EQUAL TO 65 mph.

45

5. Determine whether to reject 5. Determine whether to reject HH00..

Because 0.9524 Because 0.9524 << 1.669, we do NOT reject 1.669, we do NOT reject HH00..

Critical Value ApproachCritical Value Approach

For For = .05 and d.f. = 64 – 1 = 63, = .05 and d.f. = 64 – 1 = 63, tt.05.05 = 1.669 = 1.669

4. Determine the critical value and rejection rule.4. Determine the critical value and rejection rule.

Reject Reject HH00 if if tt >> 1.669 1.669

46

H) Introduction: Comparing Population Differences Do men get paid more than women?

$46,452 for men vs. $35,122 for women (bachelors). Do more 100-level Economics courses help you in Econ

201? Has the crime rate risen? Are there more hurricanes

recently?

47

Key point: the role of standard deviation

1x 2x

1x 2x

Probably the same

Probably different

48

Comparing 2 Populations True population means: 1 and 2.

Random sample of n1 –> 1.

Versus random sample of n2 –> 2.

Transform into problem: is 1 – 2 = 0?

Assuming 1 and 2 known –> use z-test.

If unknown –> estimate ’s from sample s’s, and use t-test.

49

I) Confidence Intervals, 2 Means: ’s Unknown How important is an extra introductory course in

determining your grade in Economics 201? Data:

Natural experiment. 59 students. 43 had only one 10x course. 16 had two 10x courses.

Final exam grades: One 10x: average = 61.69%, s1 = 22.65, n1 = 43.

Two 10x: average = 75.11%, s2 = 12.80, n2 = 16.

50

Confidence Interval Estimation Point estimator: 1 – 2.

Standard error of 1 – 2 is:

2

22

1

21

21 n

s

n

sxx

51

Confidence Interval cont’d Confidence interval of difference in means:

1 – 2 + Margin of Error

Typically use α = 0.05. Margin of error:

2

22

1

21

2/21 n

s

n

stxx

52

Degrees of Freedom One UGLY formula:

22 21 2

1 22 22 2

1 2

1 1 2 2

1 11 1

s sn n

dfs s

n n n n

22 21 2

1 22 22 2

1 2

1 1 2 2

1 11 1

s sn n

dfs s

n n n n

In this example: df = 47.36 → round down to 47. 95% confidence interval → t0.025.

For 47 degrees of freedom, table says: 2.012.

53

Confidence Interval cont’d.

or2

22

1

21

2/21 n

s

n

stxx

or16

79.163

43

17.513012.211.7586.61

54

Confidence Interval cont’d.

or

9.47513.25-

or

4.709*2.01213.25-

22.725- to3.775-

55

Confidence Interval We are 95% confident that students with only one 10x

course scored between 3.775% and 22.725% lower than students with two 10x courses.

Next step would be: why, how??

56

J) Hypothesis Tests, 2 Means: ’s Unknown Two datasets –> is the mean value of one larger than

the other? Is it larger by a specific amount?

μ1 vs. μ2 –> μ1 – μ2 vs. D0.

Often set D0 = 0 –> is μ1 = μ2?

57

Example: Female vs. Male Salaries Saskatchewan 2001 Census data:

- only Bachelor’s degrees- aged 21-64- work full-time- not in school

Men: M = $46,452.48, sM = 36,260.1, nM = 557.

Women: W = $35,121.94, sW = 20,571.3, nW = 534.

M – W = $11,330.44 } our point estimate. Is this an artifact of the sample, or do men

make significantly more than women?

58

Hypothesis, Significance Level, Test Statistic We will now ONLY use the p-value approach, and

NOT the critical value approach. Research hypothesis: men get paid more:

1. H0: μM – μW < 0H1: μM – μW > 0

2. Select = 0.05

3. Compute test t-statistic:

375.6

534426178351

5571314794928

0)94.3512148.46452()(22

0

W

W

M

M

WM

ns

ns

Dxxt

59

4. a. Compute the Degrees of Freedom Can compute by hand, or get from Excel:

22 21 2

1 22 22 2

1 2

1 1 2 2

1 11 1

s sn n

dfs s

n n n n

22 21 2

1 22 22 2

1 2

1 1 2 2

1 11 1

s sn n

dfs s

n n n n

= 888

60

4. b. Computing the p-value Degrees of Freedom 0.20 0.10 0.02 0.025 0.01 0.005

…

100 .845 1.290 1.660 1.984 2.364 2.626

.842 1.282 1.645 1.960 2.326 2.576

6.375 up here somewhere

The p-value <<< 0.005.

61

5. Check the Hypothesis Since the p-value is <<< 0.05, we reject H0.

We conclude that we can accept the alternative hypothesis that men get paid more than women at a very high level of confidence (greater than 99%).

62

Excel t-Test: Two-Sample Assuming Unequal Variances

  Male Female

Mean 46452.47935 35121.94195

Variance 1314794928 423178351.3

Observations 557 534

Hypothesized Mean Diff. 0

df 888

t Stat 6.381034789

P(T<=t) one-tail 1.41441E-10 .00000000014

t Critical one-tail 1.646571945

P(T<=t) two-tail 2.82883E-10

t Critical two-tail 1.962639544  

63

Summary Hypothesis tests on comparing two populations.

Convert to a comparison of the difference to a standard.

More complex standard deviation and degrees of freedom.

Same methodology as comparing other hypothesis tests.

64

K) Statistical vs. Practical Significance Our tests: statistically significantly Real world interest:practical significance.

Men vs. women: the difference is statistically significant AND practically:

$46,452.48 vs $35,121.94 Saskatchewan, full-time, Bachelor’s:

Women make only 75.6% of men, same average education level.

65

Source: Leader-Post, Oct. 31, 2008

66

L) Matched Samples Controlled experiment –> match individuals in each

group. Matched samples –> each individual tries each method

in turn. Variation between samples not a problem. Focus on difference data.

Independent samples –> the norm in economics. Regression analysis.

67

M) Introduction to ANOVA What if we want to compare 3 or more sample means

(treatment means)? Example: total income, Saskatchewan females

employed full-time and full-year, by age, 2003 (Source: See Oct. 8th lectures)

Age group Income in thousands of dollars

Sample size

Mean Standard deviation

25-34 33.3 13.5 55

35-44 40.3 20.7 57

45-54 45.1 25.9 37

55-64 40.1 25.9 31

Overall weighted average = 38.2

68

ANOVA’s Hypotheses

4 different populations.

equal are means population allNot :

: 43210

AH

H

There is one true population mean and 4 sample variations.

69

N) Steps of ANOVA

1. Set up the Hypothesis Statements

H0: μ1 = μ2 = μ3 = μ4 = … = μk

HA: Not all population means are equal

2. Collect your sample data:

Means: 1, 2, 3, 4, … k

Variances: s21, s2

2, s23, s2

4 ,… s2k

Sample Sizes: n1, n2, n3, n4,… nk

70

Steps of ANOVA Continued

4. Calculate the overall average:

k

kk

nnnn

xnxnxnxnx

...

...

321

332211

5. Create our two estimates of 2.

71

Step 5 a) Estimating 2 via SSTR Between-treatments estimate of 2 or

sum of squares due to treatments (SSTR). This compares to , and constructs an estimate of 2

based on the assumption the Null Hypothesis is true:

2222

211

1

2

)(...)()(

)(SSTR

xxnxxnxxn

xxn

kk

k

j jj

sx j ' x

72

Step 5 b) Estimating 2 via SSE Within-treatments estimate of 2 or

the sum of squares due to error (SSE). This takes the weighted average of the sample sj2 as an

estimate of 2 and is a good estimate regardless of whether the Null is true:

2222

211

1

2

)1(...)1()1(

)1(SSE

kk

k

jjj

snsnsn

sn

73

Step 6: Testing The Null If Null true, both estimates should be similar, and

SSTR ≈ 1.SST

If ratio >>> 1 reject the Null, accept the Alternative that there is multiple population distributions.

74

Steps of ANOVA

1. Set up the Hypothesis Statement. (Null: all means are equal)

2. Collect the sample data.

3. Select level of significance –> α = 0.05.

4. Calculate the overall average.

5. a) Estimate 2 via sum of squares due to treatments (SSTR).b) estimate of 2 via sum of squares due to error (SSE).

6. If Null true, both estimates should be similar, and STR ≈ 1.

SST

75

MSTR and MSE MSTR = sum of squares due to treatment

numerator degrees of freedom

= sum of squares due to treatmentno. of treatments – 1

= SSTRk-1

MSE = Sum of squares due to errordenominator degrees of freedom

= Sum of squares due to error

total no. of obs. – no. of treatments

= SSEnT – k

df1

df2

76

F-test F-statistic = MSTR k-1 degrees of freedom (df1)

MSE nT – k degrees of freedom (df2)

If H0 is true, MSTR ≈ MSE → F-statistic ≈ 1.

If H0 is false→ p-value is < level of significance (α).→ F-statistic is higher than critical value from the table/Excel.

77

F-Distribution

MSE

MSTRF

0 Ftest-value

1MSE

MSTRy that Probabilit

78

O) Saskatchewan Female Wages Example Example: total income, Saskatchewan females

employed full-time and full-year, by age, 2003 (Source: See Oct. 8th lectures)

Age Group Mean Income Variance =(St. Dev.)2

Sample Size

25-34 33.3 (13.5)2 = 182.25 55

35-44 40.3 (20.7)2 = 428.49 57

45-54 45.1 (25.9)2 = 670.81 37

55-64 40.1 (25.9)2 = 670.81 31

Overall weighted average = 38.2

79

Calculating the MSTR, MSE

5.11483/4.3445)1/(

4.3445

)2.381.40(31

)2.381.45(37)2.383.40(57)2.383.33(55

)()()()(

)(SSTR

2

222

244

233

222

211

1

2

kSSTRMSTR

xxnxxnxxnxxn

xxnk

j jj

8.443)4180/(4.78110)/(

4.78110

81.670*3081.670*3649.428*5625.182*54

)1()1()1()1(

)1(SSE

244

233

222

211

1

2

kNSSEMSE

snsnsnsn

sn

T

k

jjj

80

Calculating F-Stat and p-value

Ftest-value = MSTR = 1148.5 = 2.59 MSE 443.8

MSE

MSTRF

0 2.59

? value-P

81

F-Table for df1 = 3 and df2 = 176

Denomina-tor degrees of freedom (df2) (MSE)

Area in Upper Tail

Numerator degrees of freedom (df1)

(MSTR)

2 3 4

100 .10

.05

.025

.01

… 2.14

2.70

3.25

3.98

1000 .10

.05

.025

.01

… 2.09

2.61

3.13

3.80

Clearly the p-value > 0.05–> accept the Null of one distribution

176 degrees of freedom, F=2.59in here.

82

Excel F-test formula =FDIST(F-value, df1, df2) –> yields value of .0544.

MSE

MSTRF

0 2.59

0.0544 value-P

83

P) Econometrics for Dummies… Instead of ANOVA, economists tend to use Regression

analysis + “dummy” variables. Gives us the direction and size of the differences in

mean values. But ANOVA is a useful first step.