unit 11: analysis of covariance (ancova)

Unit 11: Analysis of Covariance(ANCOVA)

STA 643: Advanced Experimental Design

Derek S. Young

1

Learning Objectives

I Become familiar with the basic orthogonal designs theory ofANCOVA

I Understand the benefits and how we include covariates intoANOVA from a designed experiment

I Become familiar with the single-factor ANCOVA model

I Know the assumptions of ANCOVA

I Know how to construct an ANCOVA model when there ismore than one covariate

I Know how to construct and analyze an ANCOVA model witha blocked design

I Know how to construct and analyze a multi-factor ANCOVAmodel

2

Outline of Topics

1 Linear Model Theory

2 Single-Factor ANCOVA

3 Generalizations of ANCOVA

3

Outline of Topics




4

Overview

I Analysis of covariance (or ANCOVA) is a technique that combinesfeatures of ANOVA and regression.

I ANCOVA can be used for either observational or designed experiments.

I The idea is to augment the ANOVA model containing the factor effects

(we will only consider fixed effects) with one or more additional

quantitative variables that are related to the response variable.

I These quantitative variables are called covariates (or they alsocalled concomitant variables).

I This augmentation is intended to reduce large error term variances thatare sometimes present in ANOVA models.

I Note that if the covariates are qualitative (e.g., gender, politicalaffiliation, geographic region), then the model remains an ANOVA modelwhere the original factors are of primary interest while the covariates aresimply included for the purpose of error variance reduction.

5

Coordinatized-Version of Linear ModelI In an ANCOVA model, we have observations that are taken in

different categories as in ANOVA, but we also have someother predictors which are measured on an interval scale as inregression.

I In the general form of the ANCOVA model, we observe

Y ∼ NN (δ + Xγ, σ2I), δ ∈ Q, γ ∈ Rs, σ2 > 0, (1)

where X is a known N × s matrix (which we call a covariatematrix in ANCOVA) and Q is a known q-dimensionalsubspace.

I We assume that δ, γ, and σ2 are unknown parameters.I Typically, Q is a subspace from ANOVA.

I We call the ANOVA model with subspace Q the associatedANOVA model.

6

Coordinate-Free Version of Linear ModelI Let

X∗ = PQ⊥X, Y∗ = PQ⊥Y.

I In order to guarantee that the covariates are not confoundedwith the ANOVA component of the model, we need to assumethat X∗ has full rank; i.e., that rank(X∗) = s.

I Note that it is not enough that X have full column rank inorder to have a possible covariance analysis.

I Let U be the s-dimensional subspace spanned by the columnsof X∗ and let V be the subspace of possible value for µ; i.e.,

V = {µ = δ + Xγ : δ ∈ Q,γ ∈ Rs}.I Then

Y ∼ NN (µ, σ2I), µ ∈ V, σ2 > 0,

which is just the ordinary linear model.

7

Adjusted EstimatorsI Note that U ⊥ Q and

µ = δ + PQXγ + X∗γ, δ + PQXγ ∈ Q, X∗γ ∈ U.I Therefore,

µ ∈ V = Q⊕ U.I Hence

µ = PV Y = PQY + PUY = δu + X∗γ

δu = PQY

γ = (X∗TX∗)−1X∗TY = (X∗TX∗)−1X∗TY∗

I Note that δu is just the estimator of δ based on the associated ANOVA modeland that γ is just the least squares estimator of γ found by regressing Y (orY∗) on X∗.

I The “u” subscript means the result has been unadjusted for the covariate, whilethe “a” subscript means the result has been adjusted for the covariate.

I Further results using orthogonal direct sums allow us to write

‖µ‖2 = ‖δu‖2 + ‖X∗δ‖2,

and p = dim(V ) = dim(Q) + dim(U) = q + s.

8

Optimal Estimators

I Note thatγ = (X∗TX∗)−1X∗Tµ, γ = (X∗TX∗)−1X∗Tµ.

I Therefore, γ is the optimal (i.e., best unbiased) estimator of γ; however,

E[δu] = PQ(δ + Xγ) = δ + (X− X∗)γ.

I Therefore, δu is the optimal estimator of δ + (X− X∗)γ for the ANCOVAmodel.

I Note that the optimal estimator of

δ = (PQ − (X− X∗)(X∗TX∗)−1X∗T)µ

isδa = (PQ − (X− X∗)(X∗TX∗)−1X∗T)µ = δu − (X− X∗)γ.

I Thus, we have δa, which has been adjusted for the covariate.

9

Adjusted Sums of SquaresI We can now construct the sums of squares and degrees of freedom for the error

when adjusting for covariates.I First, SSEa and dfe;a for the ANCOVA model are given by

SSEa = ‖Y− µ‖2 = ‖Y‖2 −(‖δu‖2 + ‖X∗γ‖2

)= SSEu − ‖X∗γ‖2

dfEa = N − dim(V ) = N − (q + s) = dfe;u − s,

where SSEu and dfe;u are the sum of squares and df for the error for theassociated ANOVA model.

I We can compute MSEa = SSEa/dfe;a in the obvious way.

I In order to produce the results presented above, it is necessary to find X∗.I Let δu = PQY be the estimator of δ under the associated ANOVA model.I Then, the residual vector for the full ANOVA model is given by

Y− δu = PQ⊥Y = Y∗.

I To find X∗ = PQ⊥X, we apply the same operation to the x values thatwe apply to the y values when we find the residuals Y∗ for the associatedANOVA model.

10

Inference SetupI Next, consider testing the null hypothesis that δ ∈ T , a known t-dimensional

subspace of Q.I Let

X∗∗ = PT⊥X, Y∗∗ = PT⊥Y.

I Note that if X∗ has full rank then X∗∗ also does.I As written earlier, Y∗∗ is the vector of residuals for the reduced form of the

associated ANOVA model and X∗∗ is the same function of the columns of X asY∗∗ is of Y.

I Let W be the subspace of possible values of µ under the null hypothesis.I By the same argument from earlier, we see that

µ = PWY = δu + X∗∗γ

δu = PTY

γ = (X∗∗TX∗∗)−1X∗∗TY = (X∗∗TX∗∗)−1X∗∗TY∗∗

‖µ‖2 = ‖µu‖2 + ‖X∗∗γ‖2

k = dim(W ) = t+ s.

11

Test StatisticI Therefore, for testing δ ∈ T for the ANCOVA model, the sum of squares, df,

and mean square for the hypothesis are given, respectively, by

SSHa = ‖µ‖2 − ‖µ‖2

=(‖δu‖2 + ‖X∗γ‖2

)−(‖δu‖2 + ‖X∗∗γ‖2

)= SSHu + ‖X∗γ‖2 − ‖X∗∗γ‖2

= SSHu + ‖X∗γ − X∗∗γ‖2

dfh;a = (q + s)− (t+ s) = q − t = dfh;u

MSHa = SSHa/dfh;a.

I SSHu and dfh;u are the sums of squares and df, respectively, for the hypothesis

in the associated ANOVA model.I Note that the df for the hypothesis in the associated ANCOVA model is

the same as the df for the hypothesis in the associated ANOVA modeland that if X∗∗ = X∗, then the sum of squares and mean square for thehypothesis in the ANCOVA are the same as the sum of squares and themean square for the hypothesis in the associated ANOVA model.

I As usual, the optimal size α test for this hypothesis rejects ifF ∗ = MSHa

MSEa> F1−α;dfh;a,dfe;a .

12

Outline of Topics




13

Example: Goats DataA veterinarian carried out an experiment on a goat farm to determinewhether a standard worm drenching program (to prevent the goats fromobtaining worms) was adequate. Forty goats were used in eachexperiment, where twenty were chosen completely at random anddrenched according to the standard program, while the remaining twentywere drenched more frequently. The goats were individually tagged, andweighed at the start and end of the year-long study. The objective in theexperiment is to compare the liveweight gains between the twotreatments. This comparison could be made using an ANOVA; however, acommonly observed biological phenomenon could allow us to increase theprecision of the analysis. Namely, the lighter animals gain more weightthan the heavier animals, so we have a “regression to the norm” setting.Since this can be assumed to occur within both treatment groups, it isappropriate to adjust the analysis to use that covariate information.

14

Example: Goats DataTo understand why incorporating the covariate information might be effective,look at the plot of the data below. Here, we plotted weight gain of the goatsseparately for each treatment (either “standard” or “intensive”). It is evidentthat the error terms – as shown by the scatter around the treatment meansgiven the respective horizontal lines – are fairly large. Thus, this indicates alarge error term variance.

5 10 15 20

24

68

10

Standard Treatment

Goat #

Wei

ght G

ain

5 10 15 20

24

68

10

Intensive Treatment

Goat #

Wei

ght G

ain

15

Example: Goats DataSuppose that we now utilize the goats’ initial weight. The scatterplot below shows thegoats’ weight gain versus their initial weight, with different colors corresponding to thedifferent treatments. Note that the two treatment regression lines happen to be linear,although this need not be the case. The scatter around the treatment regression linesis much less than the scatter around the treatment means shown on the previous slideas a result of the goats’ weight gain being highly linearly related to the initial weight.Thus, the figure on the previous slide shows the (relatively) larger error variance undera single-factor ANOVA while the figure below shows the smaller error variance under asingle factor ANCOVA.

18 20 22 24 26 28 30

24

68

10

Goat Weights Data

Initial Weight

Wei

ght G

ain

TreatmentStandardIntensive

16

Local Control with Covariates

I The choice of covariates is important.I Covariates commonly used with human subjects include prestudy

attitudes, age, socioeconomic status, weight, and aptitude.I Covariates commonly used with business units include previous

quarterly sales, employee salaries, and number of employees atindividual stores.

I If the chosen covariates have no relation to the responsevariable, then nothing stands to be gained by using anANCOVA and one may as well use an ANOVA.

I We know that local control practices reduce experimentalerror variance and increase the precision for estimates oftreatment means and tests of hypotheses.

I Covariates are often used to select and group units to controlexperimental error variation.

17

Covariates Unaffected by Treatments

I For a clear interpretation of the results, a covariate should beobserved before the study; or if observed during the study,then it should not be influenced by the treatments in any way.

I Whenever a covariate is affected by the treatments, anANCOVA will fail to show some (or most) of the effects thatthe treatments had on the response variable.

I Therefore, an uncritical analysis may be badly misleading.

I Typically, when a covariate is unaffected by the treatments,the distribution of subjects along the axis of the covariate willbe roughly similar for all treatments and subject only tochance variation.

18

Example: Training Methods StudySuppose that a testing program is interested in studying the effect of aparticular training method on students’ scores. Two training methodswere used and 12 students were assigned to the training method atrandom; i.e., six students were assigned to each training method. At theend of the program, a score was obtained to measure their amount oflearning. The researcher decided to use the amount of time devoted tostudying as a covariate, but found that the training method had virtuallyno effect. The second training method involved a computer-assistedlearning program which, generally, necessitated the student needing moretime to learn the program. In other words, both the learning score andthe amount of study time were influenced by the treatment in this case.As a result the high correlation between the amount of study time andlearning score, the marginal treatment effect of training method was smalland the test for treatment effects have no significant difference betweenthe two method. A figure of these data appears on the next slide.

19

Example: Training Methods Study

I To the right is a scatterplot of thetraining methods data.

I Method 1 is a paper-and-pencilmethod while Method 2 is thecomputer-assisted learningprogram.

I Method 2 needed more timedevoted to learning the program,so observations for the twotreatments tend to beconcentrated over differentintervals on the covariate’s axis.

80 100 120 140 160 180 20050

6070

8090

Training Method Data

Study Time (Minutes)

Test

Sco

re

Training MethodMethod 1Method 2

20

Development of Single-Factor ANCOVA ModelI There are various ways to define a single-factor ANCOVA model and we’ll

present one such model where the design could, potentially, be unbalanced.I Let the number of subjects for the ith factor level be denoted by ni and the

total number of cases be N =∑i ni.

I Recall that the single-factor ANOVA model with fixed effects is given by

Yij = µ+ τi + εij

I Let the response and covariate level for the jth case of the ith factor level begiven by Yij and xij , respectively.

I The covariance model starts with the above ANOVA model and adds anotherterm reflecting the relationship between the response and the covariate.

I A first approximation is the linear relationship

Yij = µ+ τi + βxij + εij

I In the above, β is a regression coefficient for the relation between Yij and xij ,however, the constant µ is no longer an overall mean.

I We can make the constant an overall mean and simplify calculations bycentering the covariate about the overall mean x··.

21

Single-Factor ANCOVA Model

I The single-factor ANCOVA model with fixed effects is:

Yij = µ+ τi + β(xij − x··) + εij , (2)

where

I µ is the overall mean (a constant);I i = 1, . . . , r and j = 1, . . . , ni;I τi are the (fixed) treatment effects subject to the constraint

∑ri=1 τi = 0;

I β is a regression coefficient for the relationship between the response andcovariate;

I xij are constants; andI εij are the (random) errors and are iid normal with mean 0 and varianceσ2.

I Since εij is the only random variable on the right-hand side of theANCOVA model, it follows that

E(Yij) = µ+ τi + β(xij − x··) ≡ µijVar(Yij) = σ2

22

Comparisons of Treatment Effects

I In ANOVA, all observations have the same mean response; i.e., E(Yij) = µi forall j.

I In ANCOVA, the expected response for the ith treatment is a regression line;i.e., E(Yij) = µij = µ+ τi + β(xij − x··).

I While we no longer characterize the mean response with the ith treatment sinceit varies with x, we can still measure the effect of any treatment compared withany other by a single number.

I The difference between two mean responses is the same for all values of xbecause the slopes of the regression lines are equal.

I Hence, we can measure the difference at any convenient value of x, say, x = x··:

µ+ τk − (µ+ τl) = τk − τl for k 6= l

I Thus, τk − τl measures how much higher or lower the mean response iswith treatment k relative to treatment l for any value of x.

I It follows that when all treatments have the same mean response for x, then thetreatment regression lines must be identical and, hence, τi = 0 for all i.

23

Constancy of Slopes

I The assumption in ANCOVA is that all treatment regressionlines have the same slope.

I Without this assumption, the difference between the effects oftwo treatments cannot be summarized by a single numberbased on the main effects; e.g., τk − τl.

I When the treatments interact with the covariate – resulting innonparallel slopes – ANCOVA is not appropriate and, instead,separate treatment regression lines need to be estimated andthen compared.

I Similar difficulties occur with nonlinear relationships, wherethe inferences regarding the responses must include acomplete description involving the effects of the treatmentsand the covariate.

24

Appropriateness of ANCOVA

I The key assumptions in ANCOVA are:1 Normality of error terms2 Equality of error variances for different treatments3 Uncorrelatedness of error terms4 Linearity of regression relation with covariates (i.e., appropriate

model specification)5 Equality of slopes of the different treatment regression lines

I The first four assumptions are checked using the standardresidual diagnostics and remedial measures.

I The last assumption is particularly important for ANCOVAand will be developed in detail.

25

SS Partitioning for ANCOVA

I We cannot directly apply the single-factor ANOVA, but rather need toadjust the treatment SS after fitting the covariate.

I Consider the following models, which we will label f , r1, and r2,

respectively:

I full model:Yij = µ+ τi + β(xij − x··) + εij

I reduced model without covariate:

Yij = µ+ τi + εij

I reduced model without treatment effects:

Yij = µ+ β(xij − x··) + εij

I The reduced model without the covariate (r1) is required to assess theinfluence of the covariate, while the reduced model without treatmenteffects (r2) is required to assess the significance of the treatment effectsin the presence of covariates.

26

SS Partitioning for ANCOVAI Least squares estimates can be derived for each of the three models: f , r1, and

r2.

I Using those least squares estimates, we have the following SSE quantities:I full model:

SSEf =∑i

∑j

[yij − µ− τi − β(xij − x··)

]2,

with N − r − 1 df.I reduced model without covariate:

SSEr1 =∑i

∑j

[yij − µ− τi]2 ,

with N − r df.I reduced model without treatment effects:

SSEr2 =∑i

∑j

[yij − µ− β(xij − x··)

]2,

with N − 2 df.

27

SS Partitioning for ANCOVA

I The SS reduction due to the addition of the covariate x to themodel is obtained as the following difference:

SS(Covariate) = SSEr1 − SSEf ,

with 1 df.

I The adjusted SS after fitting the covariate is:

SSTr(Adjusted) = SSEr2− SSEf ,

with r − 1 df.

I Mean squares for all of the previous sources of variability are found(as usual) by dividing the SS quantity by the respective df.

I Note that construction of the correct ANCOVA table (withadjustments) amounts to using the Type III SS quantities.

28

ANCOVA Table and Testing

Source df SS MSRegression 1 SS(Covariate) MS(Covariate)Treatment r − 1 SSTr(Adjusted) MSTr(Adjusted)Error N − r − 1 SSEf MSEf

Total N − 1 SSTot

I The test for reduction in variance due to the covariate

H0 : β = 0

HA : β 6= 0

has the test statistic F ∗ =MS(Covariate)

MSEf∼ F1,N−r−1.

I The test for adjusted treatment effects is

H0 : τ1 = · · · = τr = 0

HA : not all τj equal 0

has the test statistic F ∗ =MSTr(Adjusted)

MSEf∼ Fr−1,N−r−1.

29

Testing for Common Slopes

I The linear model with different regression coefficients for each treatmentgroup (call it model r3) is

Yij = µ+ τi + βi(xij − x··) + εij , (3)

which is exactly the same as the single-factor ANCOVA model, except βhas been replaced by βi (the regression coefficient for the ith treatment).

I Let SSEr3 be the SS for experimental error, which has N − 2r df.

I The SS to test homogeneity of regression coefficients for the treatmentgroups is

SS(Homogeneity) = SSEf − SSEr3

I The test for homogeneity (or equality) of regression coefficients

H0 : β1 = · · · = βr

HA : at least one βj is different

has the test statistic F ∗ = MS(Homogeneity)MSEr3

∼ Fr−1,N−2r.

30

ANCOVA Table with Interaction

I The required SS for testing homogeneity of regressioncoefficients can be found by constructing the ANCOVA tablewith a treatment by covariate interaction term and,subsequently, computing the Type III SS.

I Below is the ANCOVA table:

Source df SS MSTreatment r − 1 SSTr(Adjusted) MSTr(Adjusted)Regression 1 SS(Regression) MS(Regression)Treatment x Regression r − 1 SS(Homogeneity) MS(Homogeneity)Error N − 2r SSEr3 MSEr3Total N − 1 SSTot

31

Treatment Means Adjusted to Covariate

I The estimated regression equation for the ith treatment groupis

yi = yi· − β(x− x··),where x is an arbitrary value of the covariate that falls withinthe domain of the observed data; i.e., we are notextrapolating.

I The estimates of treatment means are adjusted to a commonvalue for the covariate if inclusion of the covariate in themodel significantly reduces the experimental error variance.

I The treatments means are found at the value of xi· and canbe adjusted to any value of the covariate, but usually weadjust to the overall mean x··:

yi;adj = yi· − β(xi· − x··)

32

Standard Errors for the Adjusted Treatment Means

I Denote the SS for the experimental error from an ANOVA based on thecovariate x by

Exx =r∑i=1

ni∑j=1

(xij − xi·)2

I The standard error estimator for an adjusted treatment mean is

syi;adj =

√MSE

[1

ni+

(xi· − x··)2

Exx

](4)

I The standard error estimator for the difference between two adjustedtreatment means is

s(yi1;adj−yi2;adj) =

√MSE

[1

ni1+

1

ni2+

(xi1· − xi2·)2

Exx

](5)

33

Regression Formulation of ANCOVA

I The design matrix for ANCOVA has a special structure that fits nicelyinto a model composed of indicator variables.

I For k = 1, 2, . . . , r − 1, define the following (ternary) indicator variables:

Iij,k =

1 if case i is from treatment j = k

0 if case i is from treatment j = r

−1 otherwise

I The ANCOVA model can now be expressed as follows:

Yij = µ+ τ1Iij,1 + · · ·+ τr−1Iij,r−1 + β(xij − x··) + εij

I Note that the treatment effects τ1, . . . , τr−1 are the regressioncoefficients for the indicator variables.

I This regression format becomes helpful in understanding the test forparallel slopes.

34

Another Test for Parallel Slopes

I Testing for parallel slopes in an ANCOVA means that all treatment regressionlines have the same slope β.

I The regression model on the previous slide can be generalized to allow fordifferent slopes for the treatments by introducing cross-product interactionterms.

I Letting γ1, . . . , γr−1 be the regression coefficients for the interaction terms, wehave our generalized model as

Yij = µ+ τ1Iij,1 + · · ·+ τr−1Iij,r−1 + β(xij − x··)+ γ1Iij,1(xij − x··) + · · ·+ γr−1Iij,r−1(xij − x··) + εij

I The test for parallel slopes is then:

H0 : γ1 = · · · = γr−1 = 0

HA : at least one γj is not 0, for j = 1, . . . , r − 1.

I We can then apply the general linear F -test where the model above is the “full”model and the model on the previous slide without the interaction terms is the“reduced” model.

35

Example: Cracker Promotion StudyA company studied the effects of three different types of promotions on sales of itscrackers:

I Treatment 1: Sampling of product by customers in store and regular shelf space

I Treatment 2: Additional shelf space in regular location

I Treatment 3: Special display shelves at ends of aisle in addition to regular shelfspace

N = 15 stores were selected for the study and a CRD was used. Each store wasrandomly assigned one of the promotion types, with ni = n = 5 stores assigned toeach type of promotion. Other relevant conditions under the control of the company(e.g., price and advertising) were kept the same for all stores in the study. Data on thenumber of cases of the product sold during the promotional period (y) and data onthe sales of the product in the preceding period (x, the covariate) are presented below.

Store (j)Treatment 1 2 3 4 5

(i) yi1 xi1 yi2 xi2 yi3 xi3 yi4 xi4 yi5 xi51 28 21 39 26 36 22 45 28 33 192 43 34 38 26 38 29 27 18 34 253 24 23 32 29 31 30 21 26 28 29

36

Example: Cracker Promotion Study

I To the right is a scatterplot of thecracker promotion data.

I It looks like there is an effect dueto the different treatments.

I Moreover, the change in theresponse as the covariate changesappears to be similar for eachtreatment (i.e., parallel slopes).

20 25 3025

3035

4045

Cracker Promotion Data

Sales in Preceding Period

Sal

es in

Pro

mot

iona

l Per

iod

Treatment 1Treatment 2Treatment 3

37


Anova Table (Type III tests)

Response: y

Sum Sq Df F value Pr(>F)

(Intercept) 165.88 1 47.308 2.663e-05 ***

x 269.03 1 76.723 2.731e-06 ***

treat 417.15 2 59.483 1.264e-06 ***

Residuals 38.57 11

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Above is the ANCOVA table for the sales during the treatment period using sales fromthe preceding period as a covariate (you can ignore the (Intercept) row). This usesthe adjusted treatment SS; i.e., the Type III SS. For testing H0 : β = 0 – whichdetermines the significance of the reduction in error variance due to sales from thepreceding period – we have F ∗ = 76.723 ∼ F1,11 with a p-value of 2.731e-06, whichis highly significant. Therefore, the addition of the covariate has significantly reducedexperimental error variability. For testing H0 : τ1 = τ2 = τ3 – which determinesequality of the adjusted treatment means – we have F ∗ = 59.483 ∼ F2,11 with ap-value of 1.264e-06, which is, again, highly significant. Therefore, the treatmentmeans adjusted for sales from the previous period are significantly different.

38



Response: y


(Intercept) 48.742 1 13.9172 0.004693 **

treat 1.263 2 0.1803 0.837923

x 243.141 1 69.4230 1.597e-05 ***

treat:x 7.050 2 1.0065 0.403181

Residuals 31.521 9

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Above is the ANCOVA table which includes the treatment by covariate interaction

term; i.e., the interaction between promotional treatment and sales from the previous

period. You can, again, ignore the (Intercept) row. The row for the interaction gives

us the results for the test of H0 : β1 = β2 = β3. This test has F ∗ = 1.007 ∼ F2,9,

which has a p-value of 0.403. Therefore, this test is not statistically significant and we

can claim that the assumption of equal slopes is appropriate.

39


I The estimated regressionequations for the threepromotional treatments are:

y1j = 38.2 + 0.899(x1j − 23.2)

y2j = 36.0 + 0.899(x2j − 26.4)

y3j = 27.2 + 0.899(x3j − 25.4)

I To the right, the three estimatedregression equations are overlaidon the scatterplot of the data. 20 25 30

2530

3540

45

Cracker Promotion Data

Sales in Preceding Period

Sal

es in

Pro

mot

iona

l Per

iod

Treatment 1Treatment 2Treatment 3

40

Example: Cracker Promotion StudyThe treatment means adjusted to the mean sales data from the previous period(x·· = 25.0) are

y1;adj = 38.2− 0.889(23.2− 25.0) = 39.82

y2;adj = 36.0− 0.889(26.4− 25.0) = 34.74

y3;adj = 27.2− 0.889(25.4− 25.0) = 26.84

The standard errors for the above adjusted treatment means are

sy1;adj =

√3.51

[1

5+

(23.2− 25.0)2

333.2

]= 0.86

sy2;adj =

√3.51

[1

5+

(26.4− 25.0)2

333.2

]= 0.85

sy3;adj =

√3.51

[1

5+

(25.4− 25.0)2

333.2

]= 0.84

41

Example: Cracker Promotion StudyThe pairwise difference between treatment means adjusted to the mean salesdata from the previous period are

y1;adj − y2;adj = 39.82− 34.74 = 5.08

y1;adj − y3;adj = 39.82− 36.84 = 2.98

y2;adj − y3;adj = 34.74− 26.84 = 7.90

The standard errors for the above differences are

s(y1;adj−y2;adj) =

√3.51

[1

5+

1

5+

(23.2− 26.4)2

333.2

]= 1.23


√3.51

[1

5+

1

5+

(23.2− 25.4)2

333.2

]= 1.21


√3.51

[1

5+

1

5+

(26.4− 25.4)2

333.2

]= 1.19

42

Outline of Topics




43

Overview

I Thus far, we have discussed only the single-factor ANCOVAsetting.

I However, we can have a measured covariate (or covariates) inmost of the design settings that we have discussed.

I In this lecture, we will introduce a few extensions toANCOVA, specifically when we have more than one covariate,more than one factor, and when a blocking factor is present.

44

Multiple CovariatesI The single-factor ANCOVA model is usually sufficient to reduce error variability

substantially.I However, the model can be extended in a straightforward manner to include two

(or more) covariates.I The single-factor ANCOVA model with p covariates is:

Yij = µ+ τi +

p∑k=1

βk(xijk − x··p) + εij , (6)

where all of the quantities in the model are the same as for the model with one

covariate, except thatI observation j from treatment i has p measured covariates xij1, . . . , xijp;I β1, . . . , βp are the slopes estimated for the measured covariatesxij1, . . . , xijp, respectively; and

I x··1, . . . , x··p are the respective means of the p covariates.

I Assumptions and estimation for the above model are nearly identical to the

single covariate model, except that the fitted regression surfaces using the

estimated β1, . . . , βk coefficients at different treatment levels must be parallel.I In other words, we had the parallel slopes assumption in the single

covariate setting, now we have the parallel surfaces assumption.

45

Nonlinearity

I The linear relationship between the response (y) and a covariate (x) isnot essential to ANCOVA.

I We can capture nonlinearities by defining a polynomial relationship usingthe previous model as follows:

Yij = µ+ τi +

q∑k=1

βk(xij − x··)k + εij , (7)

which has a polynomial relationship (in terms of the covariate) of order q.

I For example, q = 2 would yield a quadratic relation.

I Linearity of the relation leads to simpler analysis and is often a goodapproximation, but if it is not, then the above model can be used.

I But again, ANCOVA does require that the treatment response functionsbe parallel; in other words, there must not be an interaction effectbetween the treatment and covariates.

46

ANCOVA with a Blocking Effect

I As we noted earlier, ANCOVA can be applied to almost any experimentaldesign with a straightforward extension of the established principles.

I However, when using a blocking design, you must have more than one EUper treatment within each block in order to test for equality of slopesamong treatment groups.

I The ANCOVA model with a fixed effect and blocking factor is:

Yij = µ+ τi + ρj + β(xij − x··) + εij , (8)

where

I µ is the overall mean (a constant);I i = 1, . . . , r and j = 1, . . . , n (we assume balancedness for simplicity);I τi are the (fixed) treatment effects subject to the constraint

∑ri=1 τi = 0;

I ρj is the block effects subject to the constraint∑nj=1 ρj = 0;

I β is a regression coefficient for the relationship between y and x;I xij are the covariates; andI εij are the (random) errors and are iid normal with mean 0 and varianceσ2.

47

SS Partitioning

I Consider the following models, which we will label f , r1, r2, and r3,

respectively:I full model:

Yij = µ+ τi + ρj + β(xij − x··) + εij ,

where the SSEf has (n− 1)(r − 1)− 1 df.I regular RCBD model without covariate:

Yij = µ+ τi + ρj + εij

where the SSEr1 has (n− 1)(r − 1) df.I reduced model without treatment effects:

Yij = µ+ ρj + β(xij − x··) + εij ,

where the SSEr2 has (N − n− 1) df.I reduced model without block effects:

Yij = µ+ τi + β(xij − x··) + εij ,

where the SSEr3 has (N − r − 1) df.

I The three reduced models are used to assess the influence of thecovariate, treatment effects, and block effects, respectively.

48

SS Partitioning

I We can now further define the necessary SS quantities:I SS for adding the covariate x:

SS(Covariate) = SSEr1− SSEf

with 1 df.I Adjusted treatment SS after fitting the covariate and block

effect:SSTr(Adjusted) = SSEr2 − SSEf

with (r − 1) df.I Adjusted block SS after fitting the covariate and treatment

effect:SSBlk(Adjusted) = SSEr3

− SSEf

with (n− 1) df.

49

Inference

I For testing

H0 : β = 0

HA : β 6= 0

we use the test statistic F ∗ = MS(Covariate)MSE

∼ F1,N−r−n.

I For testing

H0 : τ1 = · · · = τr = 0

HA : at least one τi is different

we use the test statistic F ∗ = MSTr(Adjusted)MSE

∼ Fr−1,N−r−n.

I For testing

H0 : ρ1 = · · · = ρn = 0

HA : at least one ρj is different

we use the test statistic F ∗ = MSBlk(Adjusted)MSE

∼ Fn−1,N−r−n.

50

ANCOVA Table and Standard ErrorsSource df SS MSRegression 1 SS(Covariate) MS(Covariate)Block n− 1 SSBlk(Adjusted) MSBlk(Adjusted)Treatment r − 1 SSTr(Adjusted) MSTr(Adjusted)Error N − r − n SSEf MSEf

Total N − 1 SSTot

I Estimated standard errors for adjusted treatment means are found analogously to the single-factorANCOVA without a blocking factor.

I Denote the SS for the experimental error from an ANOVA based on the covariate x by

Exx =r∑

i=1

n∑j=1

(xij − xi·)2

I The standard error estimator for an adjusted treatment mean is

syi;adj=

√√√√MSEf

[1

n+

(xi· − x··)2

Exx

]

I The standard error estimator for the difference between two adjusted treatment means is

s(yi1;adj−yi2;adj) =

√√√√MSEf

[2

n+

(xi1· − xi2·)2

Exx

]

51

Example: Study of Soil NutrientsManagement methods on forest and range watersheds affect the nutrients in anyvegetations and soil-type complex. The availability of certain soil nutrients in thesewatershed soils is evaluated by a pot culture technique in a greenhouse with barleyplants. A researcher wants to determine the availability of nitrogen and phosphorus ina watershed dominated by a certain type of vegetation. He collected sample soilsunder the vegetation and composited the samples for a pot culture evaluation ofnitrogen and phosphorus availability. r = 4 treatments were used for the study: (1)check (which means no fertilizer added); (2) full (a complete fertilizer); (3) nitrogen(N0) omitted from full; and (4) phosphorus (P0) omitted from full. The nutrienttreatments were added as solutions to the soil, mixed, and placed in plastic pots in agreenhouse. The treatment pots were arranged on a greenhouse bench in an RCBD tocontrol experimental error variation caused by gradients in light and temperature inthe greenhouse. The barley plants were grown in the pots for seven weeks, after whichthey were harvested, dried, and weighed. A leaf blight infected the plants part waythrough the experiment, which was assumed to affect the growth of the plants. Thus,the percentage of blight was recorded. The total dry weight is the response (y) andthe leaf blight is a measured covariate (x).

52

Example: Study of Soil Nutrients

Below is a table of the response and covariate. The last row givesthe means of these variables for each treatment.

Treatment (i)Block Check Full N0 P0

(j) y1j x1j y2j x2j y3j x3j y4j x4j

1 23.1 13 30.1 7 26.4 10 26.2 82 20.9 12 31.8 5 27.2 9 25.3 93 28.3 7 32.4 6 28.6 6 29.7 74 25.0 9 30.6 7 28.5 6 26.0 75 25.1 8 27.5 9 30.8 5 24.9 9

Mean 24.48 9.8 20.48 6.8 28.3 7.2 26.42 8.0

53


I To the right is a scatterplot of thesoil nutrients study.

I Different colors are used for thetreatments and different plottingcharacters are used for the blocks.

I It does look like there is a treatmenteffect and that the assumption ofparallel slopes will be appropriate (thiswill be tested in a moment); however,accounting for a blocking effect is notas easy to discern from this plot. 6 8 10 12

2224

2628

3032

Barley Data

Percent of Blighted Leaf Area

Dry

Wei

ght (

g)

CheckFullP0M0

Block 1Block 2Block 3Block 4Block 5

54


I To the right is a boxplot of the dryweights by block.

I This does reveal that a blockingeffect (at least marginally) could helpexplain a significant amount ofexperimental error.

I It is somewhat of a stretch to assumecommon variance here – mainlybecause of blocks 2 and 3 – but wewill proceed to analyze the raw dataas is.

I Of course, you could explore avariance-stabilizing transformationas a remedial measure.

1 2 3 4 522

2426

2830

32Block

Dry

Wei

ght (

g)

55



Response: y


(Intercept) 276.725 1 242.0235 7.759e-09 ***

x 26.590 1 23.2554 0.0005338 ***

block 7.786 4 1.7024 0.2191876

treat 24.860 3 7.2474 0.0059198 **

Residuals 12.577 11

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Above is the ANOVA table with the adjusted SS. The significance of the covariate istested with the statistic F ∗ = 23.255 ∼ F1,11, which has a p-value of 0.001. Thus,the relationship between the percent blight damaged leaf area and dry matterproduction of the barley plants is highly significant. For the null hypothesis of nodifferences among treatment means, we have F ∗ = 7.247 ∼ F3,11. The correspondingp-value is 0.006, which means that there is at least one treatment mean significantlydifferent from the others. If one tests the block effect, the test statistic isF ∗ = 1.702 ∼ F4,11 with a p-value of 0.219. This is not a significant effect; however,we retain the blocking effect since (a) it still helps explain some variability (albeit nota significant amount) and (b) we want to illustrate its inclusion in an experiment.

56



Response: y


(Intercept) 320.45 1 222.9526 3.992e-07 ***

block 6.56 4 1.1413 0.403107

treat 2.69 3 0.6234 0.619596

x 18.76 1 13.0487 0.006859 **

treat:x 1.08 3 0.2502 0.859093

Residuals 11.50 8

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Above is the ANOVA table with the adjusted SS for testing homogeneity of theslopes. We are interested in the row for the treatment by covariate interaction.The test statistic is F ∗ = 0.2502 ∼ F3,8, which has a p-value of 0.859. Thisimplies we would fail to reject the null hypothesis and claim that there is nosignificant differences between the slopes for different treatments. In otherwords, we can assume the slopes are parallel.

57


I The estimated regression equations forthe four nutrient treatments are:

y1j = 24.48− 0.863(x1j − 9.8)

y2j = 30.48− 0.863(x2j − 6.8)

y3j = 28.30− 0.863(x3j − 7.2)

y4j = 26.42− 0.863(x4j − 8.0)

I The four estimated regression equationsare overlaid on the scatterplot to theright, where we have removed thedifferent plotting characters for block tohelp with the visualization. 6 8 10 12

2224

2628

3032

Barley Data

Percent of Blighted Leaf Area

Dry

Wei

ght (

g)

CheckFullP0M0

58

Example: Study of Soil NutrientsWe can calculate standard errors for the adjusted treatment means and theirdifferences in the same way as for the single-factor ANCOVA. We summarizethose standard errors below without repeating the formulas.

TreatmentAdjusted Standard

Mean Error

Check 26.08 0.58Full 29.49 0.52N0 27.65 0.50P0 26.46 0.48

Treatment(yi1;adj − yi2;adj)

StandardDifference Error

Check-Full -3.41 0.86Check-N0 -1.58 0.82Check-P0 -0.39 0.75Full-N0 1.83 0.68Full-P0 3.02 0.71N0-P0 1.19 0.69

59

Multifactor ANCOVAI ANCOVA can also be employed when multiple factors are present.

I For simplicity, we will consider the case where the treatment sample size is thesame for all treatments and where the number of factors is two.

I Recall that the fixed-effects ANOVA model for a two-factor balanced study is

Yijk = µ·· + αi + βj + (αβ)ij + εijk,

where i = 1, . . . , a, j = 1, . . . , b, k = 1, . . . , n, and all of the traditionalassumptions and constraints apply to each term in the above model.

I The ANCOVA model for a two-factor study with a single covariate (assumingthat the relationship between y and x is linear) is

Yijk = µ··· + αi + βj + (αβ)ij + γ(xijk − x···) + εijk, (9)

where xijk is the covariate value of observation k within level i of factor A andlevel j of factor B, and γ is now used to represent the regression coefficient.

I Estimation and SS partitioning all follow the same logic as with previousANCOVA models.

60

Example: Salable Flowers StudyA horticulturist conducted an experiment to study the effects of flower variety(factor A levels: LP and WB) and moisture level (factor B levels: lower andhigh) on yield of salable flowers (y). Because the plots were not the same size,the horticulturist wished to use plot size (x) as a covariate. Six replicationswere made for each treatment. The data are given in the table below.

Factor B (j)Factor A (i) B1 : low B2 : high

yi1k xi1k yi2k xi2k

A1 : LP

98 15 71 1060 4 80 1277 7 86 1480 9 82 1395 14 46 264 5 55 3

A2 : WB

55 4 76 1160 5 68 1075 8 43 265 7 47 387 13 62 778 11 70 9

61

Example: Salable Flowers Study

I To the right is a scatterplot of thesalable flowers data.

I Different colors and plottingsymbols are used to distinguishthe four different treatments.

I It looks like there could be atreatment effect and that theassumption of parallel slopes willbe appropriate, which we will test.

2 4 6 8 10 12 1450

6070

8090

100

Salable Flowers Data

Size of Plot

Num

ber o

f Flo

wer

s

TreatmentLP,lowLP,highWB,lowWB,high

62



Response: y


(Intercept) 8218.2 1 1306.8625 < 2.2e-16 ***

x 3994.5 1 635.2118 4.586e-16 ***

variety 96.6 1 15.3617 0.0009211 ***

moisture 323.8 1 51.4988 8.093e-07 ***

variety:moisture 16.0 1 2.5511 0.1267191

Residuals 119.5 19

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

After adjusting for the covariate (using Type III SS), we fit the full two-factor modelwith the covariate. The ANCOVA table is given above (ignore the (Intercept) row).First we test the interaction term:

H0 : (αβ)11 = (αβ)12 = (αβ)21 = (αβ)22 = 0

HA : not all (αβ)ij equal 0

The test statistic is F ∗ = 2.551 ∼ F1,19, which has a p-value of 0.127. Thus, theinteraction term is not statistically significant and we drop it from the model.

63



Response: y


(Intercept) 8286.9 1 1222.944 < 2.2e-16 ***

x 3978.5 1 587.128 2.692e-16 ***

variety 97.6 1 14.396 0.001138 **

moisture 324.4 1 47.879 1.016e-06 ***

Residuals 135.5 20

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

After dropping the interaction term, we refit the model and then test the significanceof the covariate. The ANCOVA table is given above. The test statistic isF ∗ = 587.128 ∼ F1,20, which has a p-value of 2.692e-16. Thus, the relationshipbetween the salable flowers and the plot size is highly significant. We then test thesignificance of each factor effect. The F -statistics for testing effects due to factor Aand B are, respectively, F ∗ = 14.396 and F ∗ = 47.879. These each follow a F1,20

distribution, resulting in p-values of 0.001 and 1.016e-06. Therefore, both variety andmoisture levels are significant effects on salable flowers in the presence of the covariateof plot size.

64



Response: y


(Intercept) 7736.8 1 1099.0567 < 2.2e-16 ***

variety 38.7 1 5.4927 0.03078 *

moisture 34.7 1 4.9239 0.03958 *

x 3689.6 1 524.1213 9.235e-15 ***

variety:x 4.6 1 0.6469 0.43172

moisture:x 4.2 1 0.5968 0.44984

Residuals 126.7 18

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Finally, we need to test the reasonableness of homogeneity of the slopes. We canaccomplish this by crossing the covariate with each factor. The ANCOVA table is givenabove. As we can see, the tests for crossing factor A and factor B with the covariatehave p-value of 0.432 and 0.450, respectively. Therefore, the slopes are not consideredsignificantly different and we can proceed to assume that the slopes are parallel.

65


I To the right is a scatterplot of thesalable flowers data with the fittedregression lines for each treatment:

y11k = 79 + 3.263(x11k − 9)

y12k = 70 + 3.263(x12k − 9)

y21k = 70 + 3.263(x21k − 8)

y22k = 61 + 3.263(x22k − 7)

I Clearly, plot size has an effect on theflower sales in this study.

I Further quantifications can beperformed, such as standard errors andmultiple comparisons, similarly to theother ANCOVA models discussed.

2 4 6 8 10 12 1450

6070

8090

100

Salable Flowers Data

Size of Plot

Num

ber o

f Flo

wer

s

TreatmentLP,lowLP,highWB,lowWB,high

66

Final Comments About ANCOVA

I ANCOVA combines features of ANOVA and regression thatpartitions the total variation into components ascribable to (1)treatment effects, (2) covariate effects, and (3) randomexperimental error due to the experimental design (e.g., blocking).

I Using covariates makes better use of exact values for quantitativefactors instead of, say, discretizing those value and defining them asclasses to use as blocking factors.

I ANCOVA for comparative observational studies is at a disadvantagesince EUs cannot be randomized to treatment groups and there is apossibility for an influence on the response by additional unobservedcovariates that are associated with the treatment groups, thusresulting in bias.

I Since there is a regression component to ANCOVA, the samecautions regarding extrapolation apply to ANCOVA as when weperform a regression analysis.

67

This is the end of Unit 11.

68

unit 11: analysis of covariance (ancova)

Documents