Making Sense of Statistical Making Sense of Statistical SignificanceSignificance

Chapter 7Effect Size and Power

Effect SizeEffect SizeAn effect can be statistically significant

without having much practical significance.Effect Size

◦ It is a measure of the difference between populations.

◦ It tells us how much something changes after a specific intervention.

◦ It indicates the extent to which two populations do not overlap. how much populations are separated due to the

experimental procedure◦ With a smaller effect size, the populations will

overlap more.

Effect Size and Distribution Overlap

Figuring The Effect SizeFiguring The Effect SizeRaw Score Effect Size

calculated by taking the difference between the Population 1 mean and the Population 2 mean

Standardized Effect Sizecalculated by dividing the raw score effect size

for each study by each study’s population standard deviation

This standardizes the difference between means in the same way a Z-score gives us a way to compare two scores on different measures.

Effect Size ExampleEffect Size ExampleIf Population 1 had a mean of 90,

Population 2 had a mean of 50, and the population standard deviation was 20, the effect size would be:◦(90 – 50) / 20 = 2

This indicates that the effect of the experimental manipulation (e.g., reading program) is to increase the scores (e.g., reading level) by 2 standard deviations.

Copyright © 2011 by Pearson Education, Inc. All rights reserved

Formula for Calculating the Formula for Calculating the Effect SizeEffect Size

Effect Size = Population 1 M – Population 2 M

Population SD

◦ Population 1 M = the mean for the population that receives the experimental manipulation

◦ Population 2 M = the mean of the known population (the basis for the comparison distribution)

◦ Population SD = the standard deviation of the population of individuals

◦ A negative effect size would mean that the mean of Population 1 is lower than the mean of Population 2.

Copyright © 2011 by Pearson Education, Inc. All rights reserved

Example of Calculating the Example of Calculating the Effect SizeEffect Size

For the sample of 64 fifth graders, the best estimate of the Population 1 mean is the sample mean of 220.

The mean of Population 2 = 200 and the standard deviation is 48.

Effect Size = Population 1 M – Population 2 M Population SD

Effect Size = 220 – 200 48

Effect Size = .42

Effect Size ConventionsEffect Size ConventionsStandard rules about what to

consider a small, medium, and large effect size ◦based on what is typical in behavioral

and social science research Cohen’s effect size conventions for mean

differences:How Big? Effect Size (Cohen’s d)No practical effect Less than .20Small effect size .20-.49Medium effect size .50-.79Large effect size .80 or greater

A More General Importance of A More General Importance of Effect SizeEffect Size

Knowing the effect size of a study lets you compare results with effect sizes found in other studies, even when the other studies have different population standard deviations. Knowing what is a small or a large effect size helps you evaluate the overall importance of a result---PRACTICAL SIGNIFICANCE!

A result may be statistically significant without having a very large effect. Meta-Analysis

a procedure that combines results from different studies, even results using different methods or measurementsThis is a quantitative rather than a qualitative review of the literature.Effect sizes are a crucial part of this procedure.

Statistical Power-Statistical Power-The Ability to Achieve Your The Ability to Achieve Your Goals!Goals!

Probability that the study will produce a statistically significant result when the research hypothesis is really true◦When a study has only a small chance

of being significant even if the research hypothesis is true, the study has low power.

◦When a study has a high chance of being significant when the study hypothesis is actually true, the study has high power.

Remember….Remember….If the research hypothesis is false, we

do not want to get significant results.If we reject the null when the

research hypothesis is false, we commit a TYPE I ERROR.

But, even if the research hypothesis is true, we do not always get significant results. When we FAIL to reject the null hypothesis when the

Copyright © 2011 by Pearson Education, Inc. All rights reserved

What determines the Power What determines the Power of a Study?of a Study?Effect Size and PowerEffect Size and Power

If there is a is a mean difference in the population, you have more chance of getting a significant result in the study.

Since the difference between population means is the main component of effect size, the bigger the effect size, the greater the power.

Effect size is also determined by the standard deviation of a population.The smaller the standard deviation, the bigger the effect size.

The smaller the standard deviation, the greater the power.

Sample SizeSample SizeThe more people there are in the

study, the greater the power is.The larger the sample size, the smaller

the standard deviation of the distribution of means becomes.◦ The smaller the standard deviation of the distribution

of means, the narrower the distribution of means—and the less overlap there is between distributions leading to higher power. Remember that though sample size and effect size

both influence power, they have nothing to do with each other.

Figuring Needed Sample Size Figuring Needed Sample Size for a Given Level of Powerfor a Given Level of PowerThe main reason researchers consider

power is to help them decide how many people to include in their studies.◦Sample size has an important

influence on power.◦Researchers need to ensure that

they have enough people in the study that they will be able see an effect if there is one.

Copyright © 2011 by Pearson Education, Inc. All rights reserved

Other Influences on PowerOther Influences on PowerSignificance Level

◦ Less extreme significance levels (e.g., p < .10) mean more power because the shaded rejection area of the lower curve is bigger and more of the area in the upper curve is shaded.

◦ More extreme significance levels (e.g., p < .001) mean less power because the shaded region in the lower curve is smaller.

One- vs. Two-Tailed Tests◦ Using a two-tailed test makes it harder to get

significance on any one tail. Power is less with a two-tailed test than a one-tailed

test.

Copyright © 2011 by Pearson Education, Inc. All rights reserved

Statistical Significance vs. Statistical Significance vs. Practical SignificancePractical Significance Statistical Significance vs. Practical Significance

◦ It is possible for a study with a small effect size to be significant. Though the results are statistically significant ,

they may not have any practical significance. e.g., if you tested a psychological treatment and

your result is not big enough to make a difference that matters when treating patients

Evaluating the practical significance of study results is important when studying hypotheses that have practical implications.◦ e.g., whether a therapy treatment works, whether a

particular math tutoring program actually helps to improve math skills, or whether sending mailing reminders increases the number of people who respond to the Census

More things to think More things to think about….about….With a small sample size, if a result is

statistically significant, it is likely to be practically significant.

In a study with a large sample size, the effect size should also be considered.

Role of Power When a Result is Role of Power When a Result is Not Statistically SignificantNot Statistically Significant

A nonsignificant result from a study with low power is truly inconclusive.

A nonsignificant result from a study with high power suggests that:◦the research hypothesis is false or◦there is less of an effect than was

predicted when calculating power