using and reporting measures of effect size
DESCRIPTION
Using and Reporting Measures of Effect Size. Roger E. Kirk Department of Psychology & Neuroscience Baylor University. Three Categories of Measures of Effect Magnitude. 1. Measures of effect size (typically, standardized mean differences) 2. Measures of strength of association - PowerPoint PPT PresentationTRANSCRIPT
Using and Reporting Measures of Effect Size
Roger E. Kirk
Department of Psychology & NeuroscienceBaylor University
Three Categories of Measures of Effect Magnitude
1. Measures of effect size (typically, standardized mean differences)
2. Measures of strength of association
3. A large category of other kinds of measures
2
1. Estimate the sample size required to achieve an acceptable power
2. Integrate the results of empirical research studies in meta-analyses
3. Supplement the information provided by null hypothesis significance tests
4. Determine whether research results are practically significant
Four Purposes of Measures of Effect Magnitude
3
1. Answers the wrong question
What we want to know is the probability that the null hypothesis is true, given our data:
Null hypothesis significance testing tells us the probability of obtaining our data or more extreme data if the null hypothesis is true:
Four Criticisms of Null Hypothesis Significance Testing
4
2. Is a trivial exercise
According to John Tukey
“the effects of A and B are always different—in some
decimal place—for any A and B. Thus asking
‘Are the effects different?’ is foolish.”
Four Criticisms of Null Hypothesis Significance Testing (continued)
5
According to Bruce Thompson
“Statistical testing becomes a tautological search for
enough participants to achieve statistical significance.
If we fail to reject, it is only because we have been too
lazy to drag in enough participants”
Four Criticisms of Null Hypothesis Significance Testing (continued)
6
3. Requires us to make a dichotomous decision from a continuum of uncertainty
The adoption of .05 as as the dividing point between
significance and non-significance is quite arbitrary.
Four Criticisms of Null Hypothesis Significance Testing (continued)
7
4. Does not address the question of whether results are
important, valuable, or useful: that is, their practical
significance.
Four Criticisms of Null Hypothesis Significance Testing (continued)
8
1. Is an observed effect real or should it be attributed to chance?
2. If the effect is real, how large is it?
3. Is the effect large enough to be useful?
Three Basic Questions that Researchers Want to Answer from Their Research
9
“Because confidence intervals combine information
on location and precision and can often be used to
infer significance levels, they are, in general the best
reporting strategy . . . Multiple degree-of-freedom
indicators are often less useful than effect-size
indicators that decompose multiple degree-of-freedom
tests into one degree-of-freedom effects . . .
Recommendation of the APA Publication Manual
10
(1) Cohen’s
Three ways to estimate
Cohen
Glass
Hedges
Effect size
11
= 0.2 is a small effect
= 0.5 is a medium effect
= 0.8 is a large effect
Guidelines for Interpreting
12
(1)
Sample estimators of omega squared and the intraclass correlation
Strength of Association
13
= .001 is a small association
= .059 is a medium association
= .138 is a large association
Guidelines for Interpreting Omega Squared
14
Effect Size Strength of Association Other Measures
Cohen d, f, g, h, q,
Glass g’Hedges gMahalanobis DMean1 – Mean2 Mdn1 – Mdn2
Mode1 – Mode2
RosenthalTangThompson d* Wilcox
Measures of Effect Magnitude ________________________________________________________________
r, rpb, r2, R, R2, , , f2
Chamber re
Cohen f2Contingency coef CCramér VFisher ZFriedman rm
Goodman
Herzberg R2
Kelly
_______________________________________________________________
Abs. risk reduction ARRCliff pCohen U1, U2, U3
Shift functionDunlap CLR
Grisson PSLogit d’McGraw & Wong CLOdds ratioPreece ratio of successProbit d’Relative risk RRSánchez-Meca dCox15
Effect Size Strength of Association Other Measures
Wilcox & Muska
More Measures of Effect Magnitude ________________________________________________________________
Kendall WLord R2
Olejnikrequivalent
ralerting
rcontrast
reffect size
Tatsuoka Wherry R2
_______________________________________________________________
Rosenthal & Rubin BESDRosenthal & Rubin
EScounter null
Wilcoxon
16
(1)
(2)
Two Ways to Estimate the Denominator of Cohen’s
17
Effect of the Unreliability of the Dependent Variable, Y, On the
Proportion of Explained Variance
18
Aspirin group pA = .01259
Placebo group pP = .02166
pA – pP = .01259 – .02166 = –.009
Double-Blind Study of 22,071 Men Physicians
19
THE END
20