Power Analysis

An Overview

Power Is

• The conditional probability• that one will reject the null hypothesis• given that the null is really false• by a specified amount• and given certain other specifications such

as sample size and the criterion of statistical significance (alpha).

A Priori Power Analysis• You want to find how many cases you will

need to have a specified amount of power given– a specified effect size– the criterion of significance to be employed– whether the hypotheses are directional or

nondirectional• A very important part of the planning of

research.

A Posteriori Power Analysis• You want to find out what power would be

for a specified– effect size– sample size– and type of analysis

• Best done as part of the planning of research.– could be done after the research to tell you

what you should have known earlier.

Retrospective Power Analysis

• Also known as “observed power.”• What would power be if I were to– repeat this research– with same number of cases etc.– and the population effect size were exactly

what it was in the sample in the current research

• Some stat packs (SPSS) provide this.

Hoenig and Heisey • The American Statistician, 2001, 55, 19-

24• Retrospective power asks a foolish

question.• It tells you nothing that you do not already

know from the p value.• After the research you do not need a

power analysis, you need confidence intervals for effect sizes.

One Sample Test of Mean

• Experimental treatment = memory drug• H0: µIQ 100; σ = 15, N = 25• Minimum Nontrivial Effect Size (MNES)

= 2 points.• Thus, H1: µ = 102.

325

15M

= .05, MNES = 2, Power = ?

• Under H0, CV = 100 + 1.645(3) = 104.935– will reject null if sample mean 104.935

• Power = area under H1 104.935• Z = (104.935  102)/3 = 0.98 • P(Z > 0.98) = .1635 • = 1 - .16 = .84• Hope you like making Type II errors.

= .05, ES = 5, Power = ?

• What if the Effect Size were 5?• H1: µ = 105• Z = (104.935  105)/3 = 0.02 • P(Z > 0.02) = .5080 • It is easier to find large things than small

things.

H0: µ = 100 (nondirectional)

• CVLower = 100 1.96(3) = 94.12 or less • CVUpper = 100 + 1.96(3) = 105.88 or more • If µ = 105, Z = (105.88  105)/3 = .29• P(Z > .29) = .3859• Notice the drop in power.• Power is greater with directional

hypotheses IF you can correctly PREdict the direction of the effect.

Type III Error

• µ = 105 but we happen to get a very low sample mean, at or below CVLower.• We would correctly reject H0

• But incorrectly assert the direction of effect.• P(Z < (94.12 105)/3) = P(Z < 3.63),

which is very small.

H0: µ = 100, N = 100

• Under H0, CV = 100 + 1.96(1.5) = 102.94• If µ = 105, Z = (102.94  105)/1.5 = -1.37• P(Z > -1.37) = .9147• Anything that reduces the SE increases

power (increase N or reduce σ)

5.110015

M

Reduce to .01

• CVUpper = 100 + 2.58(1.5) = 103.87 • If µ = 105, Z = (103.87  105)/1.5 = -0.75• P(Z > 0.75) = .7734• Reducing reduces power, ceteris

paribus.

z versus t• Unless you know σ (highly unlikely), you

really should use t, not z.• Accordingly, the method I have shown you

is approximate.• If N is not small, it provides a good

approximation.• It is primarily of pedagogical value.

Howell’s Method• The same approximation method, but– You don’t need to think as much– There is less arithmetic– You need his power table

H0: µ = 100, N = 25, ES = 5

• IQ problem, minimum nontrivial effect size at 5 IQ points, d  = (105  100)/15 = 1/3.• with N = 25, • = (1/3)5 = 1.67.• Using the power table in our text, for a .05

two-tailed test, power = 36% for a of 1.60 and 40% for a of 1.70

Nd

= .05, = 1.67

• power for = 1.67 is 36% + .7(40% 36%) = 38.8%

I Want 95% Power• From the table, is 3.60.

• If I get data on 117 cases, I shall have power of 95%.• With that much power, if I cannot reject the

null, I can assert its near truth.

64.1163/16.3 22

d

N

The Easy Way: GPower• Test family: t tests• Statistical test: Means: Difference from constant (one

sample case)• Type of power analysis: Post hoc: Compute achieved

power – given α, sample size, and effect size• Tails: Two• Effect size d: 0.333333 (you could click “Determine” and

have G*Power compute d for you)• α error prob: 0.05• Total sample size: 25• This is NOT an approximation, it uses the t distribution.

Significant Results, Power = 36%

• Bad news – you could only get 25 cases• Good news – you got significant results• Bad news – the editor will not publish it

because power was low.• Duh. Significant results with low power

speaks to a large effect size.• But also a wide confidence interval.

Nonsignificant Results

• Power = 36%– You got just what was to be expected, a Type

II error.• Power = 95%– If there was anything nontrivial to be found,

you should have found it, so the effect is probably trivial.– The confidence interval should show this.

I Want 95% Power• How many cases do I need?

Sensitivity Analysis

• I had lots of data, N = 1500, but results that were not significant.• Can I assert the range null that d 0.• Suppose that we consider d 0 if

-0.1 d +0.1.• For what value of d would I have had 95%

power?

• If the effect were only .093, I would have almost certainly found it.

• I did not find it, so it must be trivial in magnitude• I’d rather just compute a CI.

Two Independent Samples Test of Means

• Effective sample size, .

• The more nearly equal n1 and n2, the greater the effective sample size.• For n = 50, 50, it is 50. For n =10, 90, it is

18.

21

112~

nn

n

Howell’s Method: Aposteriori• n1 = 36, n2 = 48, effect size = 40 points,

SD = 98

• From the power table, power = 46%.

85.1214.41408.

2

nd

14.41

481

361

2~

n 408.98/4021

d

I Want 80% Power• For effect size d = 1/3.• From power table, = 2.8 with alpha .05• I plan on equal sample sizes.

• Need a total of 2(141) = 282 subjects.

1413/18.222

22

d

n

G*Power• We have 36 scores in one group and 48 in

another.• If µ1 - µ2 = 40, and σ = 98, what is power?

I Want 80% Power• n1 = n2 = ? for d = 1/3, = .05, power = .8.

• You need 286 cases.

Allocation Ratio = 9• n1/n2 = 9. How many cases needed now?

• You need 788 cases!

Two Related Samples, Test of Means

• Is equivalent to one sample test of null that mean difference score = 0.

• With equal variances, • The greater , the smaller the SE, the

greater the power.

211222

21 2 Diff

)1(2 Diff

dDiff

• Adjust the value of d to take into account the power enhancing effect of this design.

)1(2 12

21

ddDiff

Diff

Howell’s Method: A Posteriori• Effect size = 20 points:– Cortisol level when anxious vs. when relaxed

• σ1 = 108, σ2 = 114• = .75• N = 16• Power = ?

Howell’s Method• Pooled SD = • d = 20/111 = .18.

• From the power table, power = 17%.

111)114(5.)108(5. 22

255.)75.1(2

18.

Diffd

.02.116255.

I Want 95% Power

3.199255.

6.3 22

Diffdn

G*Power• Dependent means,

post hoc.• Set the total sample

size to 16. • Click on “Determine.”• Select “from group

parameters.”• Calculate and transfer

to main window.

Power = 16%

I Want 95% Power

• You need 204 subjects.

Type III Errors

• You have correctly rejected H0: µ1= µ2.• Which µ is greater?• You conclude it is the one whose sample

mean was greater.• If that is wrong, you made a Type III error.• This probability is included in power.• To exclude it, see http://core.ecu.edu/psyc/wuenschk/StatHelp/Type_III.htm

Bivariate Correlation/Regression

• H0: Misanthropy-AnimalRights = 0• For power = .95, = .05, = .2, N = ?

One-Way ANOVA, Independent Samples

• f is the effect size statistic. Cohen considered .1 to be small, .25 medium, and .4 large.• In terms of 2, this is 1%, 6%, 14%.

2

2

1

)(

error

j

k

j

kf

Comparing three populations on GRE-Q

• Minimum nontrivial effect size is if each ordered mean differs from the next by 20 points (about 1/5 SD), = 100, n = 11.• (µj - µ)2 = 202 + 02 + 202 = 800

163.010000/3/800f

Power is only .115

I Want 70% Power

Analysis of Covariance• Adding covariates to the ANOVA model

can increase power.• If they are well correlated with the

dependent variable.• Adjust the f statistic this way, where r is

the corr between covariate(s) and Y.

21 rff

k = 3, f = .1, power = .95, N = ?

• f = 1 is a small effect.• Ouch, that is a lot of data we need here.

Add a Covariate, r = .7

14.49.1

1.

f

Reduce the error df by 1 for each covariate

Factorial ANOVA, Independent Samples

• We plan a 3 x 4 ANOVA.• Want power = 80% for medium-sized

effect.• Sample sizes will be constant across cells• Will be three F tests, with df = – 2 (the three level factor)– 3 (the four level factor)– 6 (the interaction)

The Three-Level Factor

• For a medium effect, you need 158 cases, = 158/12 = 13.2 per cell. Bump N up to 14(12) = 168 cases.

The Four Level Factor

The Interaction

Which N to Obtain?• You will not have the same power for each

effect.• If only interested in main effects, get the N

required for them.• Suppose we are interested in the interaction.

225/12 = 18.75 cases/cell, bump up to 19(12) = 228 cases.

• This would give you 93% power for the one main effect and almost 90% for the other.

Let GPower Determine the f• What f corresponds to 2 of 6% ?• Click Determine and enter 2 and 1- 2

Adjusting f for Other Effects• That f ignores the fact

that other effects in the model reduce the error variance.

• Suppose that I expect other effects to account for 14% of the total variance.

• I enter 6% for the effect and (100-6-14) = 80% for error.

ANOVA With Related Factors

• For the univariate-approach analysis, you need add two more parameters– The correlation between scores in one

condition and those in another condition– Epsilon, if you suspect that correlation to differ

across pairs of conditions• k = 4, f = .25 (medium), power = .95,

r = .5, = 1.

Need only 36 Cases

Increase r to .75

Estimate to be .6

Multivariate Approach: No Sphericity Assumption

Contingency Table Analysis (Two-Way)

• Effect size =

• P0i is the population proportion in cell i under the null hypothesis. • P1i is the population proportion in cell i

under the alternative hypothesis.• .1 is small, .3 medium, .5 large• For a 2 x 2, w is identical to

k

i i

ii

PPPw

1 0

201 )(

2 x 4, 95% Power, w = .1:Need 1,717 Cases !

MANOVA and DFA• There will be one root (discriminant function,

canonical variate) for each treatment df.• Each is a weighted linear combination of the Y

variables.• Each maximizes the ratio of the among groups

SS to within group SS (the eigenvalue, ).• Within a set, each root is independent of the

others.

Test Statistics for a Given Effect

• For each df there will be one and

• Hotellings Trace:

• Wilks Lambda:

• Pillai’s Trace:

• Roy’s Greatest Root: for the first root

1

1

1

The Effect Size Parameter

• It is f.• .1 is small, .25 medium, .4 large.• GPower will convert from value of trace

to f if you wish.• We plan a one-way MANOVA, four

groups, two Y variables.• Want 95% power for a medium effect.

Planning the 1-Way MANOVA

Planning the Post-MANOVA

• What will do you do if the MANOVA is significant?• You decide to do two univariate ANOVAs,

one on each outcome variable.• How much power would you have for each

of those?

Oh My, Only 25% Power

But I Want 95% Power !

• You have it, for the canonical variate you have created, just not for the original variables.• Maybe you should just work with the

canonical variates.• But maybe you, or your editors, don’t

really understand canonical variates.

95% Power for the Post-MANOVA Analyses of Variance

• Does the significant MANOVA protect you from inflating familywise error?• You decide to employ the Bonferroni

correction.• To keep familywise error capped at .05,

you use a .025 criterion for each of the two ANOVAs.• How many cases do you need?

Need 320 Cases. Ouch !

The Type I Boogey Man

• Paranoid obsession with this creature can really mess up your research life.• If that univariate ANOVA is significant, you

plan to make, for each Y, six comparisons (1-2, 1-3, 1-4, 2-3, 2-4, 3-4).• Bonferroni per comparison alpha = .05/12

= .00416.• How many cases now?

Need 165 x 4 = 660 Cases !

• 330/2 groups = 165 per group.

Links

• Assorted Stats Links• G*Power 3 – download site– User Guide – sorted by type of analysis– User Guide – sort by test distribution

• Internet Resources for Power Analysis• List of the analyses available in G*Power