Chapter 21

More About Tests

More About P-values The p-value is not the probability that the null hypothesis is true.

The p-value is the probability of the observed statistic given that the null hypothesis is true.

More About P-values Interpretation: If the null hypothesis is true, the probability of getting the observed statistic, or one more extreme, is the p-value.

A small p-value means that the observed statistic is unusual if the null hypothesis is true. So, most likely the null hypothesis is incorrect.

How Small is Small? This is determined by an alpha level.

This is our “threshold” or “cut-off”

The most common alpha levels are 0.1, 0.05, and 0.01

How Small is Small? The alpha level used sometimes depends on the situation. Assessing the reliability of parachutes. We want a very, very small occurrence of failures, so we might choose an alpha level or 0.01, or even smaller.

Pizza: Pepperoni or not? Not a life or death situation so may just be happy with an alpha level of 0.1.

Rejecting the Null Hypothesis When do we reject Ho?

We reject Ho if our p-value is less than the significance level that we choose. If p-value < , there is sufficient evidence to reject the null hypothesis

If p-value > , there is not sufficient evidence to reject the null hypothesis.

Different Alpha Levels Different alpha levels can lead to different conclusions.

Consider a p-value of 0.045. 0.045 < = 0.05 and 0.045 < = 0.1 – for both of these alpha levels we would reject Ho

What if we use = 0.01? •Not reject the null hypothesis because p-value >

Different Alpha Levels If a p-value is statistically significant at a certain level, then it is also significant at all higher levels.

If a p-value is statistically significant at a certain level, it is not necessarily significant at lower levels.

Different Alpha Levels - Example

A researcher developing scanners to search for hidden weapons at airports has concluded that a new device is significantly better than the current scanner. He made this decision based on a test using = 0.05. Would he have made the same decision at = 0.10? How about = 0.01?

Different Alpha Levels - Example

Ho: no difference between scanners

Ha: new scanner is better

Conclusion: new is better, so Ho was rejected

p-value < = 0.05, so p-value < = 0.10

p-value ? = 0.01 The same decision would be made at = 0.10 but we don’t know what decision would be made with = 0.01.

Errors

Reject HO Do not reject HO

HO is trueOops, I made a Type I error.

I got it right!!

HO is false I got it right!! Oops, I made a Type II

error.

Errors Probability of Type I error

Significance level α. α% of the time our decision to reject Ho will be wrong.

Probability of Type II error β% of the time our decision not to reject Ho will be wrong.

Error depends on true value of p. Want this error to be small when po is far away from p.

Practical Significance Often, if n is large, we will reject Ho. The actual difference between po and the true value of p could be very small.

po and the true value of p could be practically the same.

Statistical significance is not the same as practical significance.

Using Tests Wisely How small should be?

How plausible is Ho?•If the null hypothesis is strongly held belief, you need a lot of evidence against the null hypothesis to convince people it’s wrong. Would need a small p-value.

Using Tests Wisely What are the consequences of rejecting Ho.

•Change in manufacturer procedures, etc. are expensive. We need a lot of evidence against the null hypothesis to convince people to spend a large amount of money.

Using Tests Wisely - Example A company developed a program to reduce the number of elementary school students who read below their grade level. They supplied materials and teacher training for a large-scale test involving nearly 8500 children in different school districts. The percentage of students who did not attain the grade level standard was reduced from 15.9% to 15.1%. A hypothesis test was conducted; Ho was rejected with a p-value of 0.023.