Behavioral Assessment, Vol. 14, pp. 153-171. 1992 0191-5401192 $5.00 + .00 Printed in the USA. All rights reserved. Copyright © 1992 Pergamon Press Ltd.

Randomization Tests for Extensions and Variations of ABAB Single-Case Experimental

Designs: A Rejoinder

PATRICK ONGHENA

Katlwlieke Universiteit Leuven, Belgium

Randomization tests have been developed for several single-case experimental designs. It is argued, however, that the randomization tests developed by Levin, Marascuilo, and Hubert (1978) for the ABAB design and by Marascuilo and Busk (1988) for replicated ABAB designs across subjects are inappropriate. An alternative randomization procedure for the ABAB design is presented, and the appropriate corresponding randomization test is derived. It is shown how this alternative procedure has to be adapted to allow for statistical analyses of extended ABAB, parametric-variation, drug-evaluation, interaction, replication, and multiple-baseline designs. Finally, some limitations of the use of random­ization tests for single-case designs are discussed, with particular reference to changing-criterion designs.

Key Words: randomization tests; ABAB designs; within-subject designs; statistical analysis; changing criterion designs; nonparametric statistical tests; meta analysis; time series designs.

Randomization theory provides for a category of statistical tests that is valid for single-case experimental designs (Edgington, 1967, 1980c). Furthermore, these "randomization tests" are extremely versatile in this area as a consequence of their potential for tailoring the tests to the specific designs used (Edgington, 1987).

Randomization tests have been developed for (a) AB and ABA designs (Edgington, 1975a), (b) ABAB designs and extensions (Levin, Marascuilo, & Hubert, 1978), (c) alternating treatments designs (Edgington, 1967, 1980b), (d) multiple schedule designs (Edgington, 1982), (e) multiple-base­line designs (Wampold & Worsham, 1986), and (f) replicated AB and ABAB designs across subjects (Marascuilo & Busk, 1988).

The author wishes to thank Jan Beirlant, Paul De Boeck, Luc Delbeke, Eugene Edginton, Joel Levin, Gert Stonns, and three anonymous reviewers for their helpful comments on an earlier draft of the article. The author is Research Assistant of the National Fund for Scientific Research (Belgium).

Correspondence concerning this article should be addressed to Patrick Onghena. K.U. Leuven, Department of Psychology, Tiensestraat 102, B-3000 Leuven, Belgium.

153

154 ONGHENA

In the present article, it will be argued that the Levin et al. (1978) ran­domization test (in the following abbreviated as the LMH test) is inappro­priate for the ABAB design, and that the randomization test developed by Marascuilo and Busk (1988) is inappropriate for the replicated ABAB design across subjects because it is based on the LMH test. A more appro­priate, alternative test will be presented, and it will be shown how this alternative test has to be modified to analyze replication and multiple-base­line designs and single-case experimental designs which have hitherto received little attention in the randomization test literature: extended ABAB designs, parametric-variation designs, drug-evaluation designs, and interaction designs.

THE LMH TEST FOR ABAB DESIGNS

According to Kazdin (1982), ABAB designs are the most basic experi­mental designs in single-case research. They consist of a family of designs in which repeated measurements are taken for a single unit (e.g., a person) under two different alternating conditions in four phases, with several mea­surements in each phase. In the first phase, repeated measurements are taken under control conditions (baseline or first A phase); in the second, under experimental conditions (intervention or first B phase); in the third, under control conditions again (withdrawal or second A phase); and in the fourth, under experimental conditions again (second intervention or second B phase; see also Barlow & Hersen, 1984, pp. 157-166).

Because of the non-independence of the measurements, parametric sta­tistical tests based on random sampling are inappropriate for single-subject designs in general, and ABAB designs in particular (Edgington, 1967, 1975a; Levin et al., 1978). Therefore, Levin et al. (1978) proposed to use a nonparametric randomization test to analyze such designs. However, they wanted a close correspondence between the random sampling and the ran­domization model. They used phase mean scores to get "approximately independent, or at least uncorrelated, random variables that are identically distributed even though the population model itself generates dependent observations" (p. 177).

Consider, for example, the hypothetical data collected in a single-sub­ject ABAB design in Table 1. To perform the LMH test, the mean scores of each phase are regarded as a population. The null hypothesis that there is no difference between the A and the B condition is assessed by referring to the conditional sampling distribution of a test statistic associated with all theoretically possible assignments of these means to the conditions. For the data in Table 1, the mean values of each phase are 4.0, 2.0, 3.0, and 1.0, respectively. The number of possible assignments of them to Conditions A and B, under the assumption that two must go to each condition, is given

155 RANDOMIZATION TESTS FOR SINGLE-CASE DESIGNS

TABLE 1 Hypothetical Data Used to Illustrate the Levin, Marascuilo. and Hubert Test for an

ABAB Design With 24 Measurement Times (MT)

MT 1 2 3 4 5 6 7 8 9 10 11 12 13 14 IS 16 17 18 19 20 21 22 23 24

Condition A A A A A A B B B B B B A A A A A A B B B B B B

Score 625 344 1 2 3 322342430 202

by the binomial coefficient (V = 6. One could assign the following pairs to one of the conditions (let us say A), where order is not important: 4.0 and 2.0, 4.0 and 3.0,4.0 and 1.0, 2.0 and 3.0, 2.0 and 1.0, and 3.0 and 1.0. For each of the six possible assignments to a particular condition (here A), the sum of the two means is then computed as the test statistic, and based on these sums a sampling distribution is defined (Table 2).

For the observed assignment, the sum of the means associated with condition A was 4.0 + 3.0 = 7.0. The probability (p) that the value of the test statistic for one of the assignments is as large as the observed test statistic in the generated distribution is 1/6 = .1667 for a one-tailed test. If a significance level of a = .05 was chosen, the null hypothesis of no differ­ence between Conditions A and B is not rejected.

THE INAPPROPRIATENESS OF THE LMH TEST FOR ABAB DESIGNS

Although the LMH test has a certain appeal as a descriptive device and is relatively simple to perform, it is not an appropriate randomization test to allow for causal inferences. To the extent that the LMH test is presented as a randomization test for a systematic ABAB design (Le., flrst an A condi­tion, then a B condition, then an A condition, and finally a B condition), it is invalid because there has to be random assignment of measurement times

TABLE 2 Randomization Distribution for the Levin, Marascuilo. and Hubert Test Applied to the

Hypothetical Data of Table 1

Means assigned to Condition A Sum of Means Probability

4.0 and 3.0 7.0 1/6 4.0 and 2.0 6.0 1/6 4.0 and 1.0 5.0 1/6 3.0 and 2.0 5.0 1/6 3.0 and 1.0 4.0 1/6 2.0 and 1.0 3.0 1/6

156 ONGHENA

to conditions if a valid randomization test is to be performed (Edgington, 1980b, 198Oc, 1984, 1987).

Although Levin et al. (1978) defended the application of randomization tests to systematic ABAB designs, they also suggested to use the LMH test for randomized ABAB designs:

For example, in the baseline (A) versus treattnent (B) situation an initial A phase could be designated as an adaptation or "warm-up" phase, not to be included in the analysis. It might then be possible to randomly assign two A and two B conditions to four successive phases to constitute the analysis proper. In other instances, A and B may represent two different experimental treattnents (rather than a baseline and a treattnent), in which case random assignment may be more reasonable. (Note that under the proper randomization scheme, it is possible to come up with two consecutive phases of the same type.) (p. 175)

The last sentence in parentheses, however, is crucial. It implies, which Levin et al. (1978) seem to ignore, that the LMH test assumes random assignments that are inconsistent with the ABAB sequence, and hence, that it is not an appropriate randomization test for the randomized ABAB design. For a randomization test to be valid, the permutation of the data to obtain the distribution of test statistics must correspond to the particular type of random assignment that is actually used (Edgington, 1980b, 198Oc). This means that if a randomization procedure was performed that corre­sponded to the permutation of the means in the LMH test, an ABAB design would have been equally likely as an AABB, ABBA, BAAB, BABA, or BBAA design. Hence, an investigator who wanted to use an ABAB design may, as a result of this randomization procedure, have arrived at an AB (AABB) or an ABA (ABBA) design instead.

The LMH test is not well suited to analyze ABAB designs because it is an analogue of the randomization test that is appropriate for the alternat­ing treatments design in which there is one potential alternation of condi­tion per measurement time (Edgington, 1967, 1980b), It seems that Levin et al. (1978) and Marascuilo and Busk (1988) consider the ABAB design as if it were an alternating treatments design with two different conditions, four measurement times, and two measurement times for each condition, with the only exception being that for the ABAB design the phase means have to be used because there is slower alternation and more measure­ments per phase.

It could be argued that, while it is true that the LMH test is inappropri­ate for ABAB designs, it is a valid test for randomized XXXX designs, where X could be an A or B phase, and two A and two B phases can be assigned. Although this design has not been used yet, the argument is cor­rect. There is, however, still another problem: the LMH test has zero power for randomized XXXX designs at any reasonable significance level (ex = .10 or lower). By definition, a randomized XXXX design consists of

157 RANOOMIZATION TESTS FOR SINGLE-CASE DESIGNS

four phases, and consequently the p-value can never be smaller than .1667. Levin et al. (1978) and Marascuilo and Busk (1988) realized this, and they proposed to test at the a = .1667 level for a one-tailed test or at the a = .3333 level for a two-tailed test, to strengthen the alternative hypothe­sis (e.g., by making more specific predictions about the rank ordering of the mean values), to add phases to the design, or to add subjects to the study. These suggestions, however, only confmn that the basic LMH test has zero power at a =.10, a =.05, or lower.

The LMH test also lacks power for systematic ABAB designs, as in the example with the data in Table 1. The validity consideration is, however, more fundamental. Even if the LMH test were powerful (as it can be made in its extensions), there would still be the problem of its validity.

In sum, the LMH test is inappropriate for ABAB designs because, to the extent that it is proposed for systematic ABAB designs it is invalid, and to the extent that it is proposed for randomized designs it has zero power and is, strictly speaking, not even a test for ABAB designs.

AN APPROPRIATE RANDOMIZATION TEST FOR THE ABAB DESIGN

A valid randomization test is performed in nine consecutive steps: (a) choice of the alternative hypothesis, (b) specification of the level of signifi­cance a, (c) specification of the number of measurement times N, (d) ran­domization of some aspect of the design, (e) choice of a sensitive test statistic, (f) data collection, (g) computation of the test statistic from the obtained data, (h) formation of the randomization distribution, and (i) com­parison of the observed test statistic to the randomization distribution. It is essential for the validity of the test that the first five steps are perfonned before data are collected (Edgington, 1980b, 1980c, 1984, 1987; Wampold & Worsham, 1986). Because in the previous section it was shown that the invalidity of the LMH test is the most fundamental problem, the alternative test will be presented closely following these nine steps to assure its validi­ty. In addition, the alternative test will be described with the same data and level of significance that were used in the illustration of the LMH test, and it will be shown to be a powerful randomization test.

The Null and Alternative Hypotheses

The null hypothesis of a randomization test for single-case experiments is always that there is no differential effect of the conditions for any of the measurement times. The alternative hypothesis can be chosen. As in statis­tical tests for group experiments with two conditions, it is possible to have a directional (one-tailed) or a nondirectional (two-tailed) alternative. Furthermore, it is possible to make even stronger predictions about trends

158 ONGHENA

(Edgington, 1975b, 1987; Levin et al. 1978). Generally speaking, the more specific the predictions, the more powerful one's test becomes. The trade­off is, of course, that the test becomes completely inappropriate if the data are not in the predicted direction. As some critics of one-tailed t tests might say, one should only use a directional test if any treatment differences of a kind other than the predicted kind are regarded to be either irrational or irrelevant (Edgington, 1975b; Goldfried, 1959; Goldman, 1960; Kimmel, 1957; Levin et al., 1978).

In the following, the randomization test for the ABAB design will be described for a one-tailed alternative, parallel to the illustration of the LMH test (B > A). In applied behavior analysis, it is usually reasonable to specify the direction of behavior change (Kazdin, 1982; Wampold & Worsham, 1986).

The Level ofSignificance and the Number ofMeasurement Times

In the Neyman-Pearson decision-theoretical framework of hypothesis testing, the level of significance and the number of observations must be specified before the data are collected (Cohen, 1988). Together with the test statistic, the design, and the alternative hypothesis, they determine the power of the test.

In parametric statistical group analyses, the issue of Type II errors is frequently, wrongly, neglected (Cohen, 1962, 1990; Sedlmeier & Gigerenzer, 1989), and this neglect is even more prevalent in statistical analyses of single-case data. Occasionally, qualitative references are made to sensitivity considerations with respect to the choice of the test statistic, the randomization procedure, the alternative hypothesis, or the level of sig­nificance (e.g., Barlow & Hersen, 1984; Edgington, 1987; Kazdin, 1982; Levin et al., 1978), but there has never been an attempt to quantify the rela­tionship between the number of measurement times and the power of ran­domization tests for single-case designs.

This quantification is straightforward, however, if there exists a group design with the same randomization structure as the single-case design. The power of the randomization test for the single-case design is then equal to the power of the randomization test for the group design (Onghena, 1991; Onghena & Delbeke, 1992). If the same test statistic is used, the power of the randomization test for the group design itself can be approxi­mated using the standard tables of the corresponding random sampling test (May, Masson, & Hunter, 1990, pp. 313-314) or else can be derived by computer-intensive methods (Gabriel & Hall, 1983; Gabriel & Hsu, 1983; Kempthorne, 1952; Kempthorne & Doerfler, 1969; Onghena, 1992). For example, the randomization procedure for the alternating treatments design as presented by Edgington (1967) has the same structure as the completely randomized design (using Kirk's [1982] terminology), the only difference

159 RANDOMIZATION TESTS FOR SINGLE-CASE DESIGNS

being that for the first the random assignment to conditions concerns the measurement times and for the last it concerns the subjects.

If the randomization procedure of the single-case design has an unusual group design counterpart, then the power of the randomization test for the single-case design has to be derived directly by computer-intensive meth­ods (Onghena, 1991, 1992; Onghena & Delbeke, 1992), after specifying a more restrictive alternative hypothesis (e.g., a specific additive or multi­plicative effect for all measurement times under the experimental condition as proposed by Gabriel & Hall, 1983; Gabriel & Hsu, 1983; Kempthorne, 1952; Kempthorne & Doerfler, 1969; Onghena, 1992). For example, the randomization procedure for an AB design concerns the intervention point (Edgington, 1975a), and the group design counterpart would be to assign the first k selected subjects to the A condition, the others to the B condi­tion, and to determine k at random. This is unusual, and it would be very inefficient for a group design. Also the randomization procedure for an ABAB design, to be presented next, is of this type.

To make it possible to work with the data of Table 1, suppose that on the basis of practical and power considerations, a level of significance a =.05 and N =24 measurement times are chosen.

Randomization

The most appropriate aspect of the ABAB design that may be subject to randomization is the determination of the three points of change: the time of the first intervention (tl), the time of the withdrawal (t2), and the time of the second intervention (t3)' The possible times of the withdrawal are con­ditional upon the time of the first intervention, and the possible times of the second intervention are conditional upon the time of the withdrawal. Furthermore, the random selection of the three points of change has to be restricted to rule out the possibility of having too few (or even no) mea­surement times for one of the phases.

This randomization procedure makes a randomization test possible that radically differs from the LMH test. It follows the Edgington (1975a, 1980b) model for AB and ABA designs instead of the Edgington (1967) model for alternating treatments designs. In fact, Edgington (1975a, 1980b) already sug­gested this randomization procedure for extensions of AB and ABA designs.

To continue the illustration, suppose an ABAB design is chosen with at least n =4 measurements in each phase (N =24, detennined in the preceding step). Then it can be shown (see Appendix A, Equation 2) that there are 165 possible data divisions according to this randomization procedure. To make all data divisions equally probable, it is necessary to enumerate all possible triplets (tl,tz,t3) followed by random selection of a triplet. In the example, random selection of a triplet out of the set of 165 triplets {(5,9,13), (5,9,14), ... , (5,9,21), (5,10,14), ... , (5,10,21), ... , (6,10,14), ... , (6,10,21), ... ,

160 ONGHENA

(l3,17,21)} results in a probability of 1/165 for each of the data divisions. Suppose that, for the example, triplet (7,13,19) was selected at random. Notice that the resulting data division parallels the design in the illustration of the LMH test (Table 1).

The Test Statistic

Any test statistic that is sensitive to the expected effect of the treatment is acceptable. If one expects a difference in level between the two condi­tions, Condition A having the highest level, then the most straightforward test statistic will be T = (AI + Az)- (B1 + Bz), the difference between the sum of the means for the baseline and treatment phases. For the example, this is the statistic chosen.

For a difference in level, other interesting test statistics have been pro­posed for particular predictions: test statistics for immediate but decaying effects (Edgington, 1975a) and for delayed effects (Wampold & Furlong, 1981), test statistics corrected for trends (Edgington, 1980b, 198Oc, 1984, 1987) and for outliers (Marascuilo & Busk, 1988), and ordinal or nominal test statistics (Edgington, 1984; Marascuilo & Busk, 1988). It is also possi­ble to test for changes in slope (Wampold & Furlong, 1981), for specific linear or nonlinear trends (Edgington, 1975b; Levin et al. 1978), and for changes in variability (Edgington, 1975a).

Data Collection, the Observed Test Statistic, and the Randomization Distribution

Suppose the data in Table 1 were obtained. The test statistic T for these data (called the observed test statistic) is (4.0 + 3.0) - (2.0 + 1.0) =4.0. Then T is computed for all the data divisions that could have been selected following the randomization procedure. If the null hypothesis is true, each measurement is independent of the condition that is implemented, and any difference between the measurements under the two conditions is due sole­ly to a difference in factors associated with the time the two treatments were administered. Hence, each measurement time can be characterized by its own measurement, and consequently, under the null hypothesis, the ran­dom division of the measurement times in phases can be conceptualized as the random division of the measurements in phases. Because each division of the measurements is associated with a value for T, this results in a distri­bution of T under the null hypothesis (the randomization distribution). For the obtained data, the six lowest and the six highest values of T in this dis­tribution are shown in Table 3.

The Probability Value

As a last step, the observed test statistic is compared to the randomiza­tion distribution. If the proportion of test statistics that is as extreme as the

161 RANDOMIZATION TESTS FOR SINGLE-CASE DESIGNS

TABLE 3 Six Highest and Six Lowest Values of the Test Statistic Pin

the Randomization Distribution for the Alternative Test Applied to the Hypothetical Data of Table 1

Triplet of Measurement Tunes for Change T Probability

(7,14,19) 4.20 1/165 (7,11,19) 4.13 1/165 (7,15,19) 4.13 1/165 (7,13,19) 4.00 1/165 (7,14,18) 3.96 1/165 (6,14,19) 3.95 1/165

(12,17,21) 1.24 1/165 (5,10,14) 1.20 1/165 (11,17,21) 1.18 1/165 (12,16,21) 1.09 1/165 (11,16,21) 1.05 1/165 (13,17,21) 1.00 1/165

aT=(Aj +Az)-(Bj+Bz).

observed test statistic (the probability value or p-value) is smaller than or equal to a, the null hypothesis is rejected and the predicted effect in the alternative hypothesis is accepted.

In the example, the proportion of test statistics that is as large as (we had a directional alternative hypothesis) the observed test statistic is p =4/165 =.0242. This is smaller than the predetermined a =.05, and con­sequently the null hypothesis of no difference between the A and B condi­tions is rejected, in favor of the alternative hypothesis, that for some (per­haps all) of the measurement times the B condition has resulted in a higher score than if the A condition was given.

In sum, the alternative randomization procedure results in a valid ran­domization test for the ABAB design that has higher power than the LMH test with the fixed six randomization possibilities. The superior power was illustrated using the same data and level of significance, rejecting the null hypothesis with the alternative test while this was not possible with the LMH test.

RANDOMIZATION TESTS FOR EXTENDED ABAB, DRUG-EVALUATION, INTERACTION, AND PARAMETRIC­

VARIATION DESIGNS

The ABAB design can be extended by adding one or more phases after the basic ABAB sequence (Barlow & Hersen, 1984, p.175). A randomiza­

162 ONGHENA

tion test for such an extended ABAB design follows the nine steps described above. The randomization involves the k points of change; that is, the random sampling of a k-tuple of points of change out of the set of all possible k-tuples (see Appendix A for the computation of the cardinality of this set). Finally, a test statistic sensitive to the expected effect of the treat­ment (e.g., the difference between the sum of the means for the A condition and the sum of the means for the B condition) is calculated for the obtained data and compared with the randomization distribution of this test statistic to obtain the probability value.

The ABAB design can also be extended by comparing more than two conditions (e.g., in an ABABACA design; Barlow & Hersen, 1984, p. 177), and an appropriate randomization test can easily follow this extension. Besides the probable increment in k, the choice of the test statistic deserves particular attention. Edgington (1984, 1987) proposed to use the F-statistic from analysis of variance or an equivalent, but ordinal and nominal test statistics are also possible (Edgington, 1984; Marascuilo & Busk, 1988).

These same modifications apply to drug-evaluation designs and interac­tion designs. In a drug-evaluation design (Barlow & Hersen, 1984, p. 183) a no-drug condition (A), a drug condition (B), and a placebo condition (AI) are compared (e.g., in an AAIAAIABAIB design). For the randomization test, this is equivalent to an extended ABAB design with three conditions. In an interaction design (Barlow & Hersen, 1984, p. 193), a combination of conditions is present in some phases, allowing for an assessment of the additive and non-additive effects of the conditions (e.g., in an ABAC[BC]B[BC]B design, with [BC] representing a phase where both condition B and condition C are present). For the randomization test, this is equivalent to an extended ABAB design with four conditions, with the combination condition considered as a separate condition. Drug-evaluation and interaction designs can also be extended by adding several drugs or conditions in the comparison.

An interesting extension of the ABAB design is the parametric-varia­tion design, in which several levels of an independent variable are present­ed consecutively (e.g., in an ABABB(l)B(2)B(3) design, with B, B(l), B(2), and B(3), representing different levels of an independent variable; Barlow & Hersen, 1984, p. 179). The randomization procedure for a parametric­variation randomization test is equivalent to the randomization procedure of an extended ABAB design. Again, however, particular attention is need­ed for the choice of the test statistic. Parallel to the test statistic of the cor­relational trend test (Edgington, 1987), a conelation coefficient could be used. The correlation between the obtained data and coefficients assigned to the different levels of the independent variable might be calculated and compared to the randomization distIibution of this test statistic to get the probability value. The assignment of the coefficients to the levels could be perlormed using an ordinal, an interval, or a ratio weighting scheme.

163 RANOOMIZATION TESTS FOR SINGLE-CASE DESIGNS

RANDOMIZATION TESTS FOR REPLICATED ABAB DESIGNS

ABAB designs (or extensions and variations) can be replicated with different subjects, and the data can be integrated by using meta-analytic techniques (Rosenthal, 1978; White, Rusch, Kazdin & Hartmann, 1989). If the single-subject experiments can be assumed statistically independent, the probability value of the randomization test for the replicated ABAB design is obtained by perfonning a randomization test for each single-sub­ject design separately, following the guidelines described above, and then combining the resulting probability values (Edgington, 1972a, 1972b; Edgington & Haller, 1983, 1984).

In their randomization test for replicated ABAB designs, however, Marascuilo and Busk (1988) proposed to use the LMH test and the associ­ated "randomization procedure", to calculate a test statistic across subjects, and to calculate a single randomization distribution of this test statistic. Because the latter turned out to be laborious for more than two subjects, they proposed to use a normal approximation. As we argued above, the LMH test is inappropriate for the ABAB design, and consequently the Marascuilo and Busk (1988) test is inappropriate for replicated ABAB designs. The meta-analytic procedure we propose for replicated ABAB designs differs from the Marascuilo and Busk (1988) test in four respects: (a) the test statistic is not calculated across subjects but for each subject separately, (b) there is not a single randomization distribution but as many as there are subjects in the design, (c) there is no need for a reference to a normal approximation, and (d) the test is not based on the LMH test but on the alternative randomization test for the ABAB design described above.

RANDOMIZATION TESTS FOR MULTIPLE-BASELINE DESIGNS

In a multiple-baseline across subjects design, there is usually only one intervention (without withdrawal), and the data for all subjects are collect­ed simultaneously with sequential application of the intervention across subjects (Harris & Jenson, 1985). Single-subject multiple-baseline designs are possible when the intervention is sequentially aimed at different behaviors or settings for a single subject (Barlow & Hersen, 1984, pp. 210-243). The graphical analysis of the results obtained in a multiple­baseline design consists of both between- and within-baseline compar­isons (Barlow & Hersen, 1984; Harris & Jenson, 1985; Hayes, 1981). If, for example, an intervention is applied to one of the baselines and pro­duces a change in it, while little or no change is observed in the other baselines, better experimental control over historical sources of confound­ing is obtained than in a singular AB design (Cook & Campbell, 1979). Notice that in the multiple-baseline literature the term "baseline" is used to refer to the entire AB sequence.

164 ONGHENA

Wampold and Worsham (1986) precisely used this between-baseline fea­ture to develop a randomization test for the multiple-baseline design. They fixed the intervention points for the different baselines and they randomly selected the order in which the persons, behaviors, or settings were subject­ed to the treatment. A test statistic for the obtained data is then compared to a randomization distribution consisting of test statistics computed for all possible orders in which the persons, behaviors, or settings could have been subjected to treatment. Hence, in all baselines the intervention points of other baselines are located to obtain the randomization distribution.

As Marascuilo and Busk (1988) observed, however, this randomization test has low power because of the restricted randomization possibilities. Therefore, they proposed to improve the power by determining the interven­tion point at random instead of fixing it in advance. The random determina­tion of intervention points also randomly selects the order in which the per­sons, behaviors, or settings are subjected to the treatment. The control over historical sources of confounding is obtained by randomization within a baseline, and the randomization distribution is obtained not only by locating the intervention point of a baseline in the other baselines, but by locating all possible intervention points in a baseline. A multivariate test statistic such as the difference of the means for the A and the B phase summed over all baselines may be used, and the p-value of this statistic can be derived by comparing it to the single randomization distribution of this test statistic.

Marascuilo and Busk (1988) considered the randomization test with random intervention point inappropriate for multiple-baseline designs across behaviors or settings because of the probable intercorrelations between the baselines. As Edgington (1992) has argued, however, these intercorrelations cause interpretation difficulties but leave the validity of the test unaffected. This is because of the multivariate approach in constructing a randomization test for multiple-baseline designs: rejection of the random­ization null hypothesis is rejection of the overall null hypothesis that there is no effect on any of the baselines. Significant results can, however, not be interpreted for a specific baseline. The interpretation to a specific baseline can only be unequivocal if the baselines are independent, but then the mul­tiple-baseline design becomes a replicated AB design, and then the meta­analytic techniques of the previous paragraph are more appropriate.

Multiple-baseline designs can be extended to include other simultane­ous series (e.g., ABA, ABAB, ... ). The randomization test can be devel­oped after random determination of the intervention and withdrawal points, and by choosing an appropriate multivariate test statistic. It should be noted that the non-concurrent multiple-baseline design (Barlow & Hersen, 1984, p. 244; Hayes, 1981; Watson & Workman, 1981) is, in fact, a mere replicat­ed AB design (Mansell, 1982) and consequently can be analyzed with the meta-analytic techniques of the previous paragraph.

165 RANDOMIZATION TESTS FOR SINGLE-CASE DESIGNS

DISCUSSION

Although randomization tests appear to be valid and versatile in the area of single-case experimentation, Kazdin (1980, 1982, 1984) noted that several features of single-case research would preclude their use. The major obstacles in using randomization tests in single-case experi­mentation would be: (a) the necessity to have rapidly alternating condi­tions, (b) the possibility of multiple-treatment interference, (c) computa­tional complexity, and (d) the necessity of randomization (Kazdin, 1980, 1982, 1984).

The first obstacle, however, is based on a false assumption. It is not true that randomization tests can only deal with single-case designs where the conditions are alternated rapidly. In fact, rapid alternation of conditions was a requirement in none of the randomization tests described in this paper. Presumably, this false assumption stems from the fact that random­ization tests were introduced in the single-case literature by Edgington (1967) as a way to analyze alternating treatments designs.

Multiple-treatment interference is the confounding effect that the out­come of one treatment is due, in part, to previous treatments which may have been provided (Campbell & Stanley, 1966; Cook & Campbell, 1979). As such, it is a potential obstacle of a design and not of a statistical test that analyzes the data gathered following the design (Edgington, 1980a). Of course, if it were assumed that randomization tests are associated with alternating treatments designs, the possibility of multiple-treatment inter­ference would be especially acute with randomization tests.

Randomization tests for alternating treatments designs require a rapidly amounting number of data divisions with an increase in number of mea­surement times, so that for even a fairly small number of measurement times time-consuming computer programs or even approximations are required (Edgington, 1969, 1987; Kazdin, 1982, 1984). Fortunately, the randomization tests for single-case experimental designs involving random determination of points of change have a more restricted random assign­ment procedure and consequently can be performed with relatively simple computer programs. A computer program for the tests presented in this article is available from the author, together with a computer program for the random sampling of the k-tuple out of the set of all possible k-tuples of points of change. The wider availability of powerful computers and effi­cient algorithms ultimately removes this obstacle.

This leaves the fourth obstacle as the only real one. As Kazdin (1980) put it, "Randomization tests do not merely provide statistical tools to help the investigator analyze data obtained in single-case research but dictate changes in the basic designs that are used" (p. 258). As men­tioned above, the fourth step for the performance of a valid randomiza­

166 ONGHENA

tion test is always the randomization of some aspect of the design, and this has to be done before the data are collected. Randomization tests are thus incompatible with the systematic designs and the response-guided experimentation which are common in single-case research (Edgington, 1983, 1984).

For example, in a changing-criterion design (Barlow & Hersen, 1984, p. 205) the baseline condition (A) is followed by a treatment condition (B) until a preset criterion is met. Then, a condition (B 1) is introduced with a more stringent criterion, with treatment applied until this new level is met, and so forth. This changing-criterion element can also be com­bined with an ABAB element (e.g., in an ABABB1B2B3 design). If the number of measurement times for each phase can be specified a priori, for example, by characterizing each phase by the specification of the criterion but without the requirement that it should be met before it is respecified in the next phase, then it is possible to perform a randomization test for this version of the changing-criterion design because then it is equivalent to the parametric-variation design. In such a randomized changing-criteri­on design, generally, even stronger predictions are possible than in a para­metric variation-design because one has the preset criteria-predictions, and a test statistic has to be chosen accordingly (e.g., some measure of goodness-of-fit to the criterion), The underlying rationale of changing-cri­terion designs, however, is usually the shaping of behavior (Kazdin, 1980), and in shaping the number of measurement times cannot be speci­fied a priori.

Edgington (l980a) remarked that it is possible to perform a random­ization test in designs where the manipulation of conditions is partially dependent on the data (e.g., randomization after stability of the baseline or, in a changing-criterion design, after the criterion is met), but this may reduce the power substantially by restricting the randomization possibili­ties, and this leaves the basic obstacle that it is impossible to perform a randomization test where the manipulation of conditions is entirely depen­dent on the data.

It must be acknowledged however that, while it is true that randomiza­tion tests dictate a change in the basic designs that are used, namely the introduction of a random element, this change improves the internal and statistical conclusion validity of the results (Cook & Campbell, 1979). Randomization controls for unknown as well as known sources of con­founding, and Campbell and Stanley (1966) considered it important enough to serve as a criterion to distinguish experimental and quasi-exper­imental designs (Edgington, 1984, 1992). Hence, the effort to overcome the fourth obstacle is remunerated by an increased internal and statistical conclusion validity.

167 RANDOMIZATION TESTS FOR SINGLE-CASE DESIGNS

REFERENCES

Barlow, D. H., & Hersen, M. (Eds.). (1984). Single-case experimental designs: Strategies for studying behavior change (2nd ed.). Oxford: Pergamon Press.

Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.

Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal ofAbnormal and Social Psychology, 65, 145 -153.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale: Erlbaurn.

Cohen,1. (1990). Things I have learned (so far). American Psychologist, 45,1304-1312. Cook, T. D., & Campbell, D. T. (Eds.). (1979). Quasi-experimentation: Design and analysis

issues for field settings. Chicago: Rand McNally. Edgington, E. S. (1967). Statistical inference from N = I experiments. Journal of Psychology, 65,

195-199. Edgington, E. S. (1969). Approximate randomization tests. Journal ofPsychology, 72, 143-149. Edgington, E. S. (1972a). An additive method for combining probability values from independent

experiments. Journal of Psychology, 80, 351-363. Edgington, E. S. (1972b). A normal curve method for combining probability values from indepen­

dent experiments. Journal of Psychology, 82, 85-89. Edgington, E. S. (l975a). Randomization tests for one-subject operant experiments. Journal of

Psychology, 90, 57-68. Edgington, E. S. (1975b). Randomization tests for predicted trends. Canadian Psychological

Review, 16, 49-53. Edgington, E. S. (1980a). Overcoming obstacles to single-subject experimentation. Journal of

Educational Statistics, 5,261-267. Edgington, E. S. (1980b). Random assignment and statistical tests for one-subject experiments.

Behavioral Assessment, 2, 19-28. Edgington, E. S. (l980c). Validity of randomization tests for one-subject experiments. Journal of

Educational Statistics, 5, 235-251. Edgington, E. S. (1982). Nonparametric tests for single-subject multiple schedule experiments.

Behavioral Assessment, 4, 83-91. Edgington, E. S. (1983). Response-guided experimentation. Contemporary Psychology, 28,

64-65. Edgington, E. S. (1984). Statistics and single case analysis. In M. Herscn, R. M. Eisler, & P. M.

Miller (Eds.), Progress in behavior modification: Vol. 16 (pp. 83-120). New York: Raven Press.

Edgington, E. S. (1987). Randomization tests (2nd ed.). New York: Marcel Dekker. Edgington, E. S. (1992). Nonparametric tests for single-case experiments. In T. R. Kratochwill &

J. Levin (Eds.), Single case design and analysis (pp. 133-157). Hillsdale, NJ: Erlbaurn. Edgington, E. S., & Haller, O. (1983). A computer program for combining probabilities.

Educational and Psychological Measurement, 43, 835-837. Edgington, E. S., & Haller, O. (1984). Combining probabilities from discrete probability distribu­

tions. Educational and Psychological Measurement, 44, 265-274. Gabriel, K. R., & Hall, W. 1. (1983). Re-randomization inference on regression and shift effects:

computationally feasible methods. Journal of the American Statistical Association, 78, 827-836.

Gabriel, K. R., & Hsu, C. F. (1983). Power studies of re-randomization tests, with application to weather modification experiments. Journal of the American Statistical Association, 78, 766-775.

Goldfried, M. R. (1959). One-tailed tests and "unexpected" results. Psychological Review, 66, 79-80.

Goldman, M. (1960). Some further remarks on one-tailed tests and "unexpected" results. Psychological Reports, 6,171-173.

168 ONGHENA

Harris, F. N., & Jenson, W. R. (1985). Comparisons of multiple-baseline across persons designs and AB designs with replication: Issues and confusions. Behavioral Assessment, 7, 121-127.

Hayes, S. C. (1981). Single case experimental design and empirical clinical practice. Journal of Consulting and Clinical Psychology, 49, 193-211.

Kazdin, A. E. (1980). Obstacles in using randomization tests in single-case experimentation. Journal ofEducational Statistics,S, 253-260.

Kazdin, A. E. (1982). Single-case research designs: Methods for clinical and applied sellings. New York: Oxford University Press.

Kazdin, A. E. (1984). Statistical analyses for single-ease experimental designs. In D. H. Barlow & M. Hersen (Eds.), Single-case experimental designs: Strategies for studying behavior change (2nd ed.) (pp. 285-324). Oxford: Pergamon Press.

Kempthorne, O. (1952). The design and analysis ofexperiments. New York: Wiley. Kempthorne, 0., & Doerfler, T. E. (1969). The behavior of some significance tests under experi­

mental randomization. Biometrika, 56,231-247. Kimmel, H. D. (1957). Three criteria for the use of one-tailed tests. Psychological Bulletin, 54,

351-353. Kirk, R. E. (1982). Experimental design: Procedures for the behavioral sciences (2nd ed.). Pacific

Grove, CA: Brooks/Cole. Levin, J. R., Marascuilo, 1.. A., & Hubert, 1.. J. (1978). N = Nonparametric randomization tests. In

T. R. Kratochwill (Ed.), Single-subject research: Strategies for evaluating change (pp. 167-196). New York: Academic Press.

Mansell, J. (1982). Repeated direct replication of AB designs. Journal of Behavior Therapy and Experimental Psychiatry, 13, 26l.

Marascuilo,1.. A., & Busk, P. 1.. (1988). Combining statistics for multiple-baseline AB and repli­cated ABAB designs across subjects. Behavioral Assessment, 10, 1-28.

May, R. B., Masson, M. E. J., & Hunter. M. A. (1990). Applicaztion of statistics in behavioral research. New York: Harper & Row.

Onghena, P. (1991). Het onderscheidingsvermogen van randomiseringstoetsen bij N=I-experi­menten: De relatieve efficil!ntie van AB-proefopzetten en proefopzellen met alternerende behandelingen [The power of randomization tests for single-case experiments: The relative efficiency of AD designs and alternating treatments designs]. Unpublished manuscript, Katholieke Universiteit Leuven, Centrum voor Mathematische Psychologie en Psychologische Methodologie, Leuven, Belgium.

Onghena, P. (1992, May). The power ofrandomization tests. Paper presented at the Meeting of the University Center of Statistics, the NFWO contact group Data Analysis, and the NFWO con­tact group Probability Theory and its Applications, Leuven, Belgium.

Onghena, P., & Delbeke, 1.. (1992, July). Power analysis of randomization tests for single-case designs. International Journal OfPsychology, 27, 379.

Rosenthal, R. (1978). Combining results of independent studies. Psychological Bulletin, 85, 185-193.

Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of the studies? Psychological Bulletin, 105, 309-316.

Wampold, B. E., & Furlong, M. 1. (1981). Randomization tests in single-subject designs: Illustrative examples. Journal ofBehavioral Assessment, 3, 329-341.

Wampold, B. E., & Worsham, N. 1.. (1986). Randomization tests for multiple-baseline designs. Behavioral Assessment, 8, 135-143.

Watson, P. J., & Workman, E. A. (1981). Non-concurrent multiple-baseline across individuals design: An extension of the traditional multiple-baseline design. Journal ofBehavior Therapy and Experimental Psychiatry, 12, 257-259.

White, D. M., Rusch, F. R., Kazdin, A. E., & HarImann, D. P. (1989). Applications of meta analy­sis in individual-subject research. Behavioral Assessment, 11, 281-296.

169 RANOOMIZATION TESTS FOR SINGLE-CASE DESIGNS

APPENDIX A

THE CARDINALITY THEOREM FOR SINGLE-CASE DESIGNS INVOLVING RANDOM DETERMINATION OF POINTS

OF CHANGE

The Unrestricted Case

The total number of possible data divisions for a single-case design involving random determination of k points of change with N measurement times is

(1)

Proof The determination of k points of change with N measurement times is equivalent to filling k positions with slashes and N positions with dashes on a line with N+k positions. There are (N;t] ways of selecting k of the N+k positions to be filled by slashes, so there are (N;t] ways of deter­mining k points of change with N measurement times. Finally, each k-tuple uniquely defines a data division, so there are h t ] possible data divisions.

Remark. Because there are no restrictions on the position of the k points of change, this formula assumes that it is possible to have two or more points of change in a row or to begin or end with a point of change, implying no change of condition or compressing the design. For example, suppose we want to collect N = 24 measurements according to an ABAB design. Ac­cording to formula (1) the total number of possible data divisions is (N;t] = (~7) =2925, because there are k = 3 points of change in a ABAB design. This includes, however, (using the slash-dash notation of the proof):

----II--------------------/ and other AA-designs, and

-----------------11/------­

and other AB-designs. In order to avoid these and other aberrations it is necessary to restrict the possible positions of the k points of change.

The Restricted Case

The total number of possible data divisions for a single-case design involving random determination of k points of change with N measure­

170 ONGHENA

ment times and at least n measurement times in each phase is

(2)

Proof If there are k points of change, there are m = k + 1 phases. Let n;' represent the number of measurement times in phase i and let n represent the minimum number of measurement times in each phase. Hence, n/ and n are both integers, with n/ ~ n for i = 1, , m, and nl'+ nz' +...+n/ +...+nm' =N. Define the ordered m-tuple (nl', nz', , n/,... , nm') to be a data division of N measurement times into m phases. Then the mapping

A • (" , ') -4 ("nl -n, nz -n, ... , nj,-n, ... , nm-n')tJ. nl , nz ,... , nj ,... , nm

is a bijection from the set A of data divisions of N measurement times into m phases with at least n measurement times in each phase onto a set B of unrestricted data divisions of N-mn into m phases. According to Equation (1), the cardinality of set B, #B, is equal to (N - nm t k ) or (N - n(k \ 1) + k).

Because of the bijection #A =#B. and consequently #A = (N - n(k \ 1) + k),

as claimed.

Remark. With n ~ 1 the aberrations are excluded. To have an ABAB design, several measurements are needed in each phase, hence n ~ 2 and usually n ~ 4. In single-case research it is, however, not uncommon to require a longer baseline or a longer final phase. Therefore, it may be of interest to generalize Equation (2) to allow for phases with unequal restrictions.

The Generalized Case

The total number of possible data divisions for a single-case design involving random determination of k points of change with N measurement times and at least nj measurement times in phase i, with nj not necessarily equal to nj, for i,j = I, ... , m, and i;t. j. is

~ k+1 J (- (j~ ni) + k)

(3) K

Proof If there are k points of change, there are m =k+ 1 phases. Let n/ rep­resent the number of measurement times and nj the minimum number of measurement times in phase i. Hence, n/ and nj are both integers, with n/ ~ nj for i =1, , m, and n/ +nz'+...+n;'+...+nm' =N. Define the ordered m-tuple (n;' , nz', ,nj',....nm') to be a data division of N measurement times into m phases. Then the mapping

171 RANDOMIZATION TESTS FOR SINGLE-CASE DESIGNS

is a bijection from the set A of data divisions of N measurement times into m phases with at least nj measurement times in phase i onto a set B of unrestricted data divisions of N-L%lnj into m phases. According to Equation (1), the cardinality of set B, #B, is equal to Formula (3). Because of the bijection #A =#B, and consequently #A is equal to Formula (3), as claimed.

Remark. If nj =nj for all i,} =1, ... , m (with i::l:- }), then the generalized case specializes to the restricted case. If in addition nj = nj = 0 for all i, } = 1, ... , m (with i::l:- }), then the generalized case specializes to the unrestricted case.

REcEIVED: 19 AUGUST 1991 FiNAL ACCEPTANCE: 5 JULY 1992