statistical analysis of the nonequivalent groups design
Post on 21-Dec-2015
225 views
TRANSCRIPT
Statistical Analysis of the Statistical Analysis of the Nonequivalent Groups DesignNonequivalent Groups Design
Analysis RequirementsAnalysis Requirements
Pre-postPre-post Two-groupTwo-group Treatment-control (dummy-code)Treatment-control (dummy-code)
N O X ON O O
Analysis of CovarianceAnalysis of Covariance
yyii = = outcome score for the ioutcome score for the ithth unit unit
00 == coefficient for the coefficient for the interceptintercept
11 == pretest coefficientpretest coefficient
22 == mean difference for treatmentmean difference for treatment
XXii == covariatecovariate
ZZii == dummy variable for treatment(0 = control, 1= treatment)dummy variable for treatment(0 = control, 1= treatment)
eeii == residual for the iresidual for the ithth unit unit
yi = 0 + 1Xi + 2Zi + ei
where:
The Bivariate DistributionThe Bivariate Distribution
807060504030
90
80
70
60
50
40
30
pretest
post
test
Program group has a5-point pretest
Advantage.
Programgroupscores
15-pointshigher
onPosttest.
Regression ResultsRegression Results
Result is Result is biasedbiased!! CICI.95(.95(22=10)=10) == 22±2SE(±2SE(22))
== 11.2818±2(.5682)11.2818±2(.5682)== 11.2818±1.136411.2818±1.1364
CI = 10.1454 to 12.4182 CI = 10.1454 to 12.4182
Predictor Coef StErr t p Constant 18.714 1.969 9.50 0.000pretest 0.62600 0.03864 16.20 0.000Group 11.2818 0.5682 19.85 0.000
yi = 18.7 + .626Xi + 11.3Zi
The Bivariate DistributionThe Bivariate Distribution
807060504030
90
80
70
60
50
40
30
pretest
post
test
Regressionline slopesare biased.
Why?
Regression and ErrorRegression and ErrorY
X
No measurement error
Regression and ErrorRegression and ErrorY
X
Y
X
No measurement error
Measurement erroron the posttest only
Measurement erroron the pretest only
Regression and ErrorRegression and ErrorY
X
Y
X
Y
X
No measurement error
Measurement erroron the posttest only
How Regression Fits LinesHow Regression Fits Lines
How Regression Fits LinesHow Regression Fits Lines
Method of least squares
How Regression Fits LinesHow Regression Fits Lines
Method of least squares
Minimize the sum of the squares of the residuals from the
regression line.
How Regression Fits LinesHow Regression Fits Lines
Y
X
Method of least squares
Minimize the sum of the squares of the residuals from the
regression line.
Least squares minimizes on y not x.
How Error Affects SlopeHow Error Affects SlopeY
X
No measurement error,No effect
How Error Affects SlopeHow Error Affects SlopeY
X
Y
X
No measurement error,no effect.
Measurement erroron the posttest only,
adds variability aroundregression line, but
doesn’t affect the slope
Measurement erroron the pretest only:
Affects slopeFlattens regression lines
How Error Affects SlopeHow Error Affects SlopeY
X
Y
X
Y
X
No measurement error,no effect.
Measurement erroron the posttest only,
adds variability aroundregression line, but
doesn’t affect the slope.
How Error Affects SlopeHow Error Affects SlopeY
X
Y
X
Y
X
Y
X
Measurement erroron the pretest only:
Affects slopeFlattens regression lines
How Error Affects SlopeHow Error Affects SlopeY
X
Y
X
Y
X
Y
X
Notice that the true result inall three cases should
be a null (no effect) one.
How Error Affects SlopeHow Error Affects Slope
Notice that the true result inall three cases should
be a null (no effect) one.
Y
X
Null case
How Error Affects SlopeHow Error Affects Slope
But with measurement erroron the pretest, we get a
pseudo-effect.
Y
X
Pseudo-effect
Where Does This Leave Us?Where Does This Leave Us?
Traditional ANCOVA looks like it should Traditional ANCOVA looks like it should work on NEGD, but it’s work on NEGD, but it’s biasedbiased..
The bias results from the effect of The bias results from the effect of pretest measurement errorpretest measurement error under the under the least squares criterion.least squares criterion.
Slopes are flattened or “Slopes are flattened or “attenuatedattenuated”.”.
What’s the Answer?What’s the Answer?
If it’s a pretest problem, let’s fix the If it’s a pretest problem, let’s fix the pretestpretest..
If we could If we could remove the errorremove the error from the from the pretest, it would fix the problem.pretest, it would fix the problem.
Can we Can we adjust pretest scoresadjust pretest scores for error?for error? What do we know about What do we know about errorerror??
What’s the Answer?What’s the Answer?
We know that if we had We know that if we had nono error, error, reliability = 1; reliability = 1; allall error, reliability=0. error, reliability=0.
Reliability estimates the proportion of Reliability estimates the proportion of truetrue score. score.
UnreliabilityUnreliability=1-Reliability.=1-Reliability. This is the proportion of This is the proportion of errorerror!! Use this to Use this to adjustadjust pretest. pretest.
What Would a Pretest Adjustment Look What Would a Pretest Adjustment Look Like?Like?
Original pretest distribution
What Would a Pretest Adjustment Look What Would a Pretest Adjustment Look Like?Like?
Original pretest distribution
Adjusted dretest distribution
Y
X
How Would It Affect Regression?How Would It Affect Regression?
The regression
The pretestdistribution
Y
X
How Would It Affect Regression?How Would It Affect Regression?
The regression
The pretestdistribution
Y
X
How Far Do We Squeeze the Pretest?How Far Do We Squeeze the Pretest?
• Squeeze inward an Squeeze inward an amount amount proportionateproportionate to to the error.the error.
• If If reliability=.8reliability=.8, we want , we want to squeeze in about to squeeze in about 20%20% (i.e., 1-.8).(i.e., 1-.8).
• Or, we want pretest to Or, we want pretest to retain 80%retain 80% of it’s original of it’s original width.width.
Adjusting the Pretest for UnreliabilityAdjusting the Pretest for Unreliability
Xadj = X + r(X - X)_ _
Adjusting the Pretest for UnreliabilityAdjusting the Pretest for Unreliability
Xadj = X + r(X - X)_ _
where:
Adjusting the Pretest for UnreliabilityAdjusting the Pretest for Unreliability
Xadj = X + r(X - X)_ _
Xadj = adjusted pretest value
where:
Adjusting the Pretest for UnreliabilityAdjusting the Pretest for Unreliability
Xadj = X + r(X - X)_ _
Xadj = adjusted pretest value
X = original pretest value_
where:
Adjusting the Pretest for UnreliabilityAdjusting the Pretest for Unreliability
Xadj = X + r(X - X)_ _
r = reliability
Xadj = adjusted pretest value
X = original pretest value_
where:
Reliability-CorrectedReliability-CorrectedAnalysis of CovarianceAnalysis of Covariance
yyii = = outcome score for the ioutcome score for the ithth unit unit
00 == coefficient for the coefficient for the interceptintercept
11 == pretest coefficientpretest coefficient
22 == mean difference for treatmentmean difference for treatment
XXadjadj == covariate adjusted for unreliabilitycovariate adjusted for unreliability
ZZii == dummy variable for treatment(0 = control, 1= dummy variable for treatment(0 = control, 1=
treatment)treatment)eeii == residual for the iresidual for the ithth unit unit
yi = 0 + 1Xadj + 2Zi + ei
where:
Regression ResultsRegression Results
Result is Result is unbiasedunbiased!! CICI.95(.95(22=10)=10) == 22±2SE(±2SE(22))
== 9.3048±2(.6166)9.3048±2(.6166)== 9.3048±1.23329.3048±1.2332
CI = 8.0716 to 10.5380CI = 8.0716 to 10.5380
yi = -3.14 + 1.06Xadj + 9.30Zi
Predictor Coef StErr t pConstant -3.141 3.300 -0.95 0.342adjpre 1.06316 0.06557 16.21 0.000Group 9.3048 0.6166 15.09 0.000
Graph of MeansGraph of Means
3035404550556065707580
Pretest Posttest
Comparison
Program
pretest posttest pretest posttestMEAN MEAN STD DEV STD DEV
Comp 49.991 50.008 6.985 7.549Prog 54.513 64.121 7.037 7.381ALL 52.252 57.064 7.360 10.272
Adjusted PretestAdjusted Pretest
Note that the adjusted means are the Note that the adjusted means are the same as the unadjusted means.same as the unadjusted means.
The only thing that changes is the The only thing that changes is the standard deviation (variability).standard deviation (variability).
pretest adjpre posttest pretest adjpre posttestMEAN MEAN MEAN STD DEV STD DEV STD DEV
Comp 49.991 49.991 50.008 6.985 3.904 7.549Prog 54.513 54.513 64.121 7.037 4.344 7.381ALL 52.252 52.252 57.064 7.360 4.706 10.272
Original Regression ResultsOriginal Regression Results
807060504030
90
80
70
60
50
40
30
pretest
post
test
Ori
gin
al
Pseudo-effect=11.28
Corrected Regression ResultsCorrected Regression Results
807060504030
90
80
70
60
50
40
30
pretest
post
test
Ori
gin
al
Corrected
Pseudo-effect=11.28
Effect=9.31