Practical Closed Stepwise Procedures to Test Homogeneity

of All Subsets of Means

James Schmeidler

Departments of Psychiatry and Biomathematical Sciences

Mount Sinai School Of Medicine

New York, NY

Stepdown Procedures

• Assumptions as in one-way ANOVA

• k groups, ni per group

• When testing the null hypothesis that a set of p means is homogeneous, a step-down procedure requires rejection for every set including all p means

Homogeneity Test for a Subset

• Use pooled within-group mean square, s2

• Two procedures have been considered

– ANOVA (test F times numerator df)

– Studentized ranges: take maximum of

(xa - xb) /{s[(1/na + 1/nb)/2]1/2}

for all pairs of means in the subset

Significance Level for a Subset

• Experimentwise α level

• i means in the subset

• (i/k)α level (Bonferroni inequality)

• Better than Scheffe or Tukey procedures

– They test every subset at α level critical value for a sample of k means

Closed Procedure (Peritz and Gabriel)

• Consider partitions that include the set of p• For each partition, there is a composite null

hypothesis that all its sets are homogeneous• The partition of the set of p and singleton

sets tests only the set of p, at α level• Reject the null hypothesis for the set of p if

the composite hypothesis is rejected for every partition

Begun and Gabriel Algorithm

• Reject null hypothesis for the set of p if

– The set of p is significant at α level

– Every complementary set of i > 2 means significant at (i/k)α level

• Only necessary to test each set once, not all sets in every partition

Application of the Algorithm

• m = k - p complementary means have 2m - m - 1 complementary subsets of i > 2

• Many fewer complementary sets to test than number of partitions

• General strategy: test least likely sets first

• Specific strategy: test pairs of means adjacent in the complementary set

Conjectures About Pairs of Adjacent Complementary Means• If adjacent pairs of complementary means

differ significantly: xa << xb and xb << xc

– xa << xc

– The set of three means is heterogeneous• Do these results generalize to more means?• Do these results apply to both ANOVA and

Studentized ranges?• Do they apply in unbalanced designs, too?

Theorem 1. Testing Adjacent Pairs of Means

• For Studentized ranges and ANOVA

• If xa < xb < xc with both {xa, xb} and {xb,

xc} significant at the (2/k)α level

• {xa, xc}is significant at the (2/k)α level

Theorem 2. Testing Adjacent Sets of Means

• For Studentized ranges in balanced designs and all ANOVAs

• If sets of D means with ends xa < xb and of

E means with ends xb < xc are significant at

the (D/k)α and (E/k)α levels, respectively

• Their union is significant at the

[(D + E – 1)/k]α level

Theorem 3. Test Only Pairs of Adjacent Complementary Means• For Studentized ranges in balanced designs

and all ANOVAs

• If all adjacent pairs of mean in a set are significant at the (2/k)α level

• The set and all its subsets are significant at the respective (i/k)α levels

Counterexample for Theorem 2

• Studentized ranges; α = .05; k = 5; s = 1.00

• n1 = n2 = n3 = 10; n4 = n5 = 3500

• Means: 0.00, 1.00, 2.00, 2.74, 2.80

• Test of first two means using closure

• Each complementary pair is heterogeneous

• All three complementary means are not

Implications of the Counterexample

• For Theorem 2 to fail for unbalanced Studentized ranges, the sample sizes must be ridiculously discrepant

• Therefore Theorems 2 and 3 apply to Studentized ranges in almost all practical applications

Seaman, Levin, and Serlin (SLS) Improved Significance Levels

• If r means left after excluding singleton sets from the partition

– Test set of i means at the (i/r)α level

– If r < k, (i/r)α > (i/k)α, increasing power

• A set has different significance levels in different partitions

• Cannot use Begun and Gabriel algorithm

Problematic Sets and Partitions

• A “problematic” complementary set is one not known to be significant at the (i/k)α level by a direct test, Theorem 1, or Theorem 2

• A “problematic” partition consists of the set of p means, at least one problematic set, and perhaps singletons

Theorem 4. Using SLS, Test Only Problematic Partitions

• If any problematic partition has r = k, set of p not rejected using SLS

• Let u = largest integer for which the set of p means is rejected at the [(p/(u + p)]α level

• All problematic sets with i > u means must be rejected at the [(i/(i + p)]α level

• If r < u + p, no need to test a partition’s sets

Situations For Examining Only Problematic Sets, Not Partitions

• If every problematic partition has only one problematic set

• If every problematic set of i means is rejected at the [i/(v + p)]α level, for v means in the union of all problematic sets

Example: Fail to Reject Only One Complementary Pair

• Any set that includes the pair is problematic

• Every problematic partition has only one problematic set, so only examine the sets

• Only sets with i > u means need to be examined

• If a set of i means is rejected at (i/k)α level, Theorem 2 applies to all sets including it

Conclusions

• Closed stepdown procedures improve power, but at the cost of additional tests

• Can use Tukey or Scheffe, stepdown, closure, and SLS in succession

• Theorem 3 makes the Begun and Gabriel algorithm much more practical

• Theorem 4 simplifies the SLS procedure