statistical analysis of the nonequivalent groups design

Statistical Analysis of the Statistical Analysis of the Nonequivalent Groups DesignNonequivalent Groups Design

Analysis RequirementsAnalysis Requirements

Pre-postPre-post Two-groupTwo-group Treatment-control (dummy-code)Treatment-control (dummy-code)

N O X ON O O

Analysis of CovarianceAnalysis of Covariance

yyii = = outcome score for the ioutcome score for the ithth unit unit

00 == coefficient for the coefficient for the interceptintercept

11 == pretest coefficientpretest coefficient

22 == mean difference for treatmentmean difference for treatment

XXii == covariatecovariate

ZZii == dummy variable for treatment(0 = control, 1= treatment)dummy variable for treatment(0 = control, 1= treatment)

eeii == residual for the iresidual for the ithth unit unit

yi = 0 + 1Xi + 2Zi + ei

where:

The Bivariate DistributionThe Bivariate Distribution

807060504030

90

80

70

60

50

40

30

pretest

post

test

Program group has a5-point pretest

Advantage.

Programgroupscores

15-pointshigher

onPosttest.

Regression ResultsRegression Results

Result is Result is biasedbiased!! CICI.95(.95(22=10)=10) == 22±2SE(±2SE(22))

== 11.2818±2(.5682)11.2818±2(.5682)== 11.2818±1.136411.2818±1.1364

CI = 10.1454 to 12.4182 CI = 10.1454 to 12.4182

Predictor Coef StErr t p Constant 18.714 1.969 9.50 0.000pretest 0.62600 0.03864 16.20 0.000Group 11.2818 0.5682 19.85 0.000

yi = 18.7 + .626Xi + 11.3Zi

The Bivariate DistributionThe Bivariate Distribution

807060504030

90

80

70

60

50

40

30

pretest

post

test

Regressionline slopesare biased.

Why?

Regression and ErrorRegression and ErrorY

X

No measurement error

Regression and ErrorRegression and ErrorY

X

Y

X

No measurement error

Measurement erroron the posttest only

Measurement erroron the pretest only

Regression and ErrorRegression and ErrorY

X

Y

X

Y

X

No measurement error

Measurement erroron the posttest only

How Regression Fits LinesHow Regression Fits Lines

How Regression Fits LinesHow Regression Fits Lines

Method of least squares

How Regression Fits LinesHow Regression Fits Lines

Method of least squares

Minimize the sum of the squares of the residuals from the

regression line.

How Regression Fits LinesHow Regression Fits Lines

Y

X

Method of least squares

Minimize the sum of the squares of the residuals from the

regression line.

Least squares minimizes on y not x.

How Error Affects SlopeHow Error Affects SlopeY

X

No measurement error,No effect

How Error Affects SlopeHow Error Affects SlopeY

X

Y

X

No measurement error,no effect.

Measurement erroron the posttest only,

adds variability aroundregression line, but

doesn’t affect the slope

Measurement erroron the pretest only:

Affects slopeFlattens regression lines

How Error Affects SlopeHow Error Affects SlopeY

X

Y

X

Y

X

No measurement error,no effect.

Measurement erroron the posttest only,

adds variability aroundregression line, but

doesn’t affect the slope.

How Error Affects SlopeHow Error Affects SlopeY

X

Y

X

Y

X

Y

X

Measurement erroron the pretest only:

Affects slopeFlattens regression lines

How Error Affects SlopeHow Error Affects SlopeY

X

Y

X

Y

X

Y

X

Notice that the true result inall three cases should

be a null (no effect) one.

How Error Affects SlopeHow Error Affects Slope

Notice that the true result inall three cases should

be a null (no effect) one.

Y

X

Null case

How Error Affects SlopeHow Error Affects Slope

But with measurement erroron the pretest, we get a

pseudo-effect.

Y

X

Pseudo-effect

Where Does This Leave Us?Where Does This Leave Us?

Traditional ANCOVA looks like it should Traditional ANCOVA looks like it should work on NEGD, but it’s work on NEGD, but it’s biasedbiased..

The bias results from the effect of The bias results from the effect of pretest measurement errorpretest measurement error under the under the least squares criterion.least squares criterion.

Slopes are flattened or “Slopes are flattened or “attenuatedattenuated”.”.

What’s the Answer?What’s the Answer?

If it’s a pretest problem, let’s fix the If it’s a pretest problem, let’s fix the pretestpretest..

If we could If we could remove the errorremove the error from the from the pretest, it would fix the problem.pretest, it would fix the problem.

Can we Can we adjust pretest scoresadjust pretest scores for error?for error? What do we know about What do we know about errorerror??

What’s the Answer?What’s the Answer?

We know that if we had We know that if we had nono error, error, reliability = 1; reliability = 1; allall error, reliability=0. error, reliability=0.

Reliability estimates the proportion of Reliability estimates the proportion of truetrue score. score.

UnreliabilityUnreliability=1-Reliability.=1-Reliability. This is the proportion of This is the proportion of errorerror!! Use this to Use this to adjustadjust pretest. pretest.

What Would a Pretest Adjustment Look What Would a Pretest Adjustment Look Like?Like?

Original pretest distribution

What Would a Pretest Adjustment Look What Would a Pretest Adjustment Look Like?Like?

Original pretest distribution

Adjusted dretest distribution

Y

X

How Would It Affect Regression?How Would It Affect Regression?

The regression

The pretestdistribution

Y

X

How Would It Affect Regression?How Would It Affect Regression?

The regression

The pretestdistribution

Y

X

How Far Do We Squeeze the Pretest?How Far Do We Squeeze the Pretest?

• Squeeze inward an Squeeze inward an amount amount proportionateproportionate to to the error.the error.

• If If reliability=.8reliability=.8, we want , we want to squeeze in about to squeeze in about 20%20% (i.e., 1-.8).(i.e., 1-.8).

• Or, we want pretest to Or, we want pretest to retain 80%retain 80% of it’s original of it’s original width.width.

Adjusting the Pretest for UnreliabilityAdjusting the Pretest for Unreliability

Xadj = X + r(X - X)_ _

Adjusting the Pretest for UnreliabilityAdjusting the Pretest for Unreliability

Xadj = X + r(X - X)_ _

where:

Adjusting the Pretest for UnreliabilityAdjusting the Pretest for Unreliability

Xadj = X + r(X - X)_ _

Xadj = adjusted pretest value

where:

Adjusting the Pretest for UnreliabilityAdjusting the Pretest for Unreliability

Xadj = X + r(X - X)_ _

Xadj = adjusted pretest value

X = original pretest value_

where:

Adjusting the Pretest for UnreliabilityAdjusting the Pretest for Unreliability

Xadj = X + r(X - X)_ _

r = reliability

Xadj = adjusted pretest value

X = original pretest value_

where:

Reliability-CorrectedReliability-CorrectedAnalysis of CovarianceAnalysis of Covariance

yyii = = outcome score for the ioutcome score for the ithth unit unit

00 == coefficient for the coefficient for the interceptintercept

11 == pretest coefficientpretest coefficient

22 == mean difference for treatmentmean difference for treatment

XXadjadj == covariate adjusted for unreliabilitycovariate adjusted for unreliability

ZZii == dummy variable for treatment(0 = control, 1= dummy variable for treatment(0 = control, 1=

treatment)treatment)eeii == residual for the iresidual for the ithth unit unit

yi = 0 + 1Xadj + 2Zi + ei

where:

Regression ResultsRegression Results

Result is Result is unbiasedunbiased!! CICI.95(.95(22=10)=10) == 22±2SE(±2SE(22))

== 9.3048±2(.6166)9.3048±2(.6166)== 9.3048±1.23329.3048±1.2332

CI = 8.0716 to 10.5380CI = 8.0716 to 10.5380

yi = -3.14 + 1.06Xadj + 9.30Zi

Predictor Coef StErr t pConstant -3.141 3.300 -0.95 0.342adjpre 1.06316 0.06557 16.21 0.000Group 9.3048 0.6166 15.09 0.000

Graph of MeansGraph of Means

3035404550556065707580

Pretest Posttest

Comparison

Program

pretest posttest pretest posttestMEAN MEAN STD DEV STD DEV

Comp 49.991 50.008 6.985 7.549Prog 54.513 64.121 7.037 7.381ALL 52.252 57.064 7.360 10.272

Adjusted PretestAdjusted Pretest

Note that the adjusted means are the Note that the adjusted means are the same as the unadjusted means.same as the unadjusted means.

The only thing that changes is the The only thing that changes is the standard deviation (variability).standard deviation (variability).

pretest adjpre posttest pretest adjpre posttestMEAN MEAN MEAN STD DEV STD DEV STD DEV

Comp 49.991 49.991 50.008 6.985 3.904 7.549Prog 54.513 54.513 64.121 7.037 4.344 7.381ALL 52.252 52.252 57.064 7.360 4.706 10.272

Original Regression ResultsOriginal Regression Results

807060504030

90

80

70

60

50

40

30

pretest

post

test

Ori

gin

al

Pseudo-effect=11.28

Corrected Regression ResultsCorrected Regression Results

807060504030

90

80

70

60

50

40

30

pretest

post

test

Ori

gin

al

Corrected

Pseudo-effect=11.28

Effect=9.31