using and reporting measures of effect size

Using and Reporting Measures of Effect Size

Roger E. Kirk

Department of Psychology & NeuroscienceBaylor University

Three Categories of Measures of Effect Magnitude

1. Measures of effect size (typically, standardized mean differences)

2. Measures of strength of association

3. A large category of other kinds of measures

2

1. Estimate the sample size required to achieve an acceptable power

2. Integrate the results of empirical research studies in meta-analyses

3. Supplement the information provided by null hypothesis significance tests

4. Determine whether research results are practically significant

Four Purposes of Measures of Effect Magnitude

3

1. Answers the wrong question

What we want to know is the probability that the null hypothesis is true, given our data:

Null hypothesis significance testing tells us the probability of obtaining our data or more extreme data if the null hypothesis is true:

Four Criticisms of Null Hypothesis Significance Testing

4

2. Is a trivial exercise

According to John Tukey

“the effects of A and B are always different—in some

decimal place—for any A and B. Thus asking

‘Are the effects different?’ is foolish.”

Four Criticisms of Null Hypothesis Significance Testing (continued)

5

According to Bruce Thompson

“Statistical testing becomes a tautological search for

enough participants to achieve statistical significance.

If we fail to reject, it is only because we have been too

lazy to drag in enough participants”

Four Criticisms of Null Hypothesis Significance Testing (continued)

6

3. Requires us to make a dichotomous decision from a continuum of uncertainty

The adoption of .05 as as the dividing point between

significance and non-significance is quite arbitrary.

Four Criticisms of Null Hypothesis Significance Testing (continued)

7

4. Does not address the question of whether results are

important, valuable, or useful: that is, their practical

significance.

Four Criticisms of Null Hypothesis Significance Testing (continued)

8

1. Is an observed effect real or should it be attributed to chance?

2. If the effect is real, how large is it?

3. Is the effect large enough to be useful?

Three Basic Questions that Researchers Want to Answer from Their Research

9

“Because confidence intervals combine information

on location and precision and can often be used to

infer significance levels, they are, in general the best

reporting strategy . . . Multiple degree-of-freedom

indicators are often less useful than effect-size

indicators that decompose multiple degree-of-freedom

tests into one degree-of-freedom effects . . .

Recommendation of the APA Publication Manual

10

(1) Cohen’s

Three ways to estimate

Cohen

Glass

Hedges

Effect size

11

= 0.2 is a small effect

= 0.5 is a medium effect

= 0.8 is a large effect

Guidelines for Interpreting

12

(1)

Sample estimators of omega squared and the intraclass correlation

Strength of Association

13

= .001 is a small association

= .059 is a medium association

= .138 is a large association

Guidelines for Interpreting Omega Squared

14

Effect Size Strength of Association Other Measures

Cohen d, f, g, h, q,

Glass g’Hedges gMahalanobis DMean1 – Mean2 Mdn1 – Mdn2

Mode1 – Mode2

RosenthalTangThompson d* Wilcox

Measures of Effect Magnitude ________________________________________________________________

r, rpb, r2, R, R2, , , f2

Chamber re

Cohen f2Contingency coef CCramér VFisher ZFriedman rm

Goodman

Herzberg R2

Kelly

_______________________________________________________________

Abs. risk reduction ARRCliff pCohen U1, U2, U3

Shift functionDunlap CLR

Grisson PSLogit d’McGraw & Wong CLOdds ratioPreece ratio of successProbit d’Relative risk RRSánchez-Meca dCox15

Effect Size Strength of Association Other Measures

Wilcox & Muska

More Measures of Effect Magnitude ________________________________________________________________

Kendall WLord R2

Olejnikrequivalent

ralerting

rcontrast

reffect size

Tatsuoka Wherry R2

_______________________________________________________________

Rosenthal & Rubin BESDRosenthal & Rubin

EScounter null

Wilcoxon

16

(1)

(2)

Two Ways to Estimate the Denominator of Cohen’s

17

Effect of the Unreliability of the Dependent Variable, Y, On the

Proportion of Explained Variance

18

Aspirin group pA = .01259

Placebo group pP = .02166

pA – pP = .01259 – .02166 = –.009

Double-Blind Study of 22,071 Men Physicians

19

THE END

20