simulation of propensity scoring methods...simulation of propensity scoring methods dee h. wu, ph.d...
TRANSCRIPT
Simulation of Propensity Scoring Methods
Dee H. Wu, Ph.D
University of Oklahoma Health Sciences Center, Oklahoma City, OK
WHAT IS PROPENSITY SCORING?• In certain clinical trials or observational studies,
proper random assignment of treatment and control groups is not always possible, so that selection bias may become an issue.
• Recent efforts to address issues of nonrandom assignment, including a class of methods known as ‘Propensity Scoring,’ are alternatives to reduce bias in the estimation of treatment effects when assignment is not random. ( Rosenbaum and Rubin in 1983).
But what about ANCOVA asalternative?
Treat 0 1
y
-3
-2
-1
0
1
2
3
4
5
6
x1
-4 -3 -2 -1 0 1 2 3 4
What aboutThe common Problemof HeterogeneityOf Regression?
THEORY SECTION Problem Statement in Pictures
Y=outcome
covariates
Y=outcome
covariates
Some covariatesAre highly correlatedWith treatment assignment
Y=outcome
covariates
Some covariatesAre highly correlatedWith treatment assignment
Y=outcome
covariates
A vector that is highly correlated with treatment assignment
Y=outcome
covariates
A vector that is highly correlated with treatment assignment
Y=outcome
covariates
For Simplicity in Drawing (we actuallyKeep multiple covariates in the problem)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 50 100 150 200 250 300 350 400 450
covariatesJust showing two compositeVectors a set which highly impactsThe treatment selection and one that doesn’t
covariates
Y=outcomeis disregardedFor moment
Now generate a propensity Score (a scalar function)
IIIIIIIVV
Increase in Probability Of treatment Y=outcome
Group(match)
I
II
III
III
VIV Now
We canDo our analysisOn groups
Y=outcome
X=covariates
Treatment may not be random on any of these
But in a Quasi-experiment, some of these variables will produce a greater likelihoodOf being in the treatment group ( the green ones)
1. a hypothesis for a causal relationship; 2. a control group and a treatment group; 3. to eliminate confounding variables that
might mess up the experiment and prevent displaying the causal relationship; and
4. to have larger groups with a carefully sorted constituency; preferably randomized, in order to keep accidental
differences from fouling things up. Quasi, when we don’t have all of the situations abovehttp://writing.colostate.edu/guides/research/experiment/pop3e.cfm
Y=outcome
X=covariates
Treatment may not be random on any of these
But in a Quasi-experiment, some of these variables will produce a greater likelihood of being in the treatment group ( the green ones:
Components of Collapsed Exposure variables which have less influence overThe ability to be in group that gets treatment
The Green ones are the components that influenceThe likelihood of being in the treatment.
Y=outcome
X=covariates
Treatment may not be random on any of these
This is no good, because the ANCOVA like methods only help push distributionsAlong the covariate axis and have strict requirements of LINEARITY and HOMOGENEITY OF REGRESSION !
Components of Collapsed Exposure variables which have less influence overThe ability to be in group that gets treatment
The Green ones are the components that influenceThe likelihood of being in the treatment.
Treat 0 1
y
-3
-2
-1
0
1
2
3
4
5
6
x1
-4 -3 -2 -1 0 1 2 3 4
Slide to means of theRegression access
Y=outcome
X=covariates
Treatment may not be random on any of these
Propensity methods provide a new composite variable too identify groups of variables
Groups of variables that have similar covariate patterns (I don’t know if thisIs a derived variable (linear combination) or the original covariates
In any case the GREEN ones are used to generate the propensity score , and stratification with these variables will help to reduce covariate imbalance in the analysis(i.e. because these variables predict the likelihood of treatment, thoseThat have a similar likelihood will get matched under striation.
Components of Collapsed covariates which have less influence overThe ability to be in group that gets treatment
The Green ones are the components that influenceThe likelihood of being in the treatment.
Along the Blue the likelihood of being treatment groupIs more random
Y=outcomeIs IGNORED
X=covariates (green and blue
Make a new axis for analysis which is the treatment group and ignore the output axis. Get
probabilities P of T(A or B) = Σ ai Xi + ε from the logistic regression analysis
Treatment A
Treatment B0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 50 100 150 200 250 300 350 400 450
observations
ProbabilityOf being inTreatment A
Y=outcomeIs IGNORED
X=covariates (green and blue
Make a new axis for analysis which is the treatment group and ignore the output axis
Get probabilities P of T(A or B) = Σ ai Xi + ε from the logistic regression analysis
Treatment A
Treatment B0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 50 100 150 200 250 300 350 400 450
observations
ProbabilityOf being inTreatment A
Stratifity the set groups(maybe Quintiles?)
I
IIIIIIV
V
NOTE EACH STRIATION WILL HAVE SOME MEMBERSThat got treatment A and some people that got B
Y=outcome
X=covariates (green and blue
ProbabilityOf being inTreatment A
G=Stratifity the set groups(maybe Quintiles?)
WITHIN EACH STRIATION DO analysis within each straitation (IS THISDONE AS A GROUP GLM (what is the Model statement for this?)
model Y = T Q (Q*T) test for interaction and then remove Variables are T (for treatment A or B)X are covariatesY is outcomeG is striation group
Some Examples (Mixtures based on one single covariate)
A 2
3
1
4
proc corr data=in;var y x1 x2 x3 x4 propensity;run;
• Pearson Correlation Coefficients, N = 500
Prob > |r| under H0: Rho=0y x1 x2 x3 x4 propensity
y 1.00000 0.61654 0.27374 0.47571 0.59396 0.66762<.0001 <.0001 <.0001 <.0001 <.0001
x1 0.61654 1.00000 0.03578 0.35222 0.02003 0.92131<.0001 0.4247 <.0001 0.6550 <.0001
x2 0.27374 0.03578 1.00000 0.00116 0.02229 0.22981<.0001 0.4247 0.9794 0.6190 <.0001
x3 0.47571 0.35222 0.00116 1.00000 -0.00268 0.38666<.0001 <.0001 0.9794 0.9523 <.0001
x4 0.59396 0.02003 0.02229 -0.00268 1.00000 0.05991<.0001 0.6550 0.6190 0.9523 0.1810
propensity 0.66762 0.92131 0.22981 0.38666 0.05991 1.00000<.0001 <.0001 <.0001 <.0001 0.1810
3. Calculate a propensity score based a linear combination (weights sum to 1) of the covariates X1 to X4, along with a small random normal error term The variable 'propensity score was generated by a linear combination of the covariates but could have been generated by any function. { I just made up a linear relationship from covariates}
propensity = 0.79*x1+0.15*x2+0.05*x3+0.01 *x4+0.3*Normal(0,1);
4. Assign to Treatment group based on the propensity score. Note we use run a standard random normal probability with mean 0 and std=1 to assign treatment group (i.e. x1 has a large influence on setting the treatment group).
Note, we used the inverse Probit transformation to control the treatment/control ratio (proportions) and for generating the proper threshold for classifying the treatment covariate.
5. Create observation vector Y for each of the test cases
Y= a1*x1 + a2*x2 + a3*x3 + a4*x4 + b*T+ c1*T*x1 + c2*T*x2 + c3*T*x3 + c4*T*x4 + 0.4*Norm(0,1)
where, a1=0.28,a2=0.13,a3=0.23,a4=0.352; b=0.4; c1=c2=c3=0.2,c4=0.02
Treat 0 1
y
-3
-2
-1
0
1
2
3
4
5
6
x1
-4 -3 -2 -1 0 1 2 3 4
Treat 0 1
y
-3
-2
-1
0
1
2
3
4
5
6
x2
-2 -1 0 1 2
The treatment set and control group are plotted against x1. In many applications, several covariates may be associated with treatment assignment, not just a single one like x1. x1 can represent a ‘modeled’ linear combination of covariates. X2 shows more randomness.
Low dependence on X2Red are treated groups(shown on x1)
The Model (we evaluate upto first order interactions)
Include Interaction Terms
Now for the SAS
Generate RandomDistributions(this is a bimodal one)
Actually , I had used another program to generate these distributions import tab deliminated or ed text files.
Tab delaminatedAscii code for tab is ’09’
Treat 0 1
y
-3
-2
-1
0
1
2
3
4
5
6
x2
-2 -1 0 1 2
Distribution of Y versus confounder x1 for dataset 4 9013:25 Friday, September 21,
2007
The MEANS Procedure
Analysis Variable : y
NTreat Obs Mean Std Dev
0 311 -0.2349022 0.4766791
1 189 0.6163011 0.5493193
0
0.1
0.2
0.3
0.40.5
0.6
0.7
0.80.9
1
0 50 100 150 200 250 300 350 400 450
EVENT='category' | keywordspecifies the event category for the
binary response model. PROC LOGISTIC models the probability of the
event category.. OUT= SAS-data-set
names the output data set. If you omit the OUT= option, the output data set is created and given a default name using the DATA n convention. PREDPROBS = requests individual, cumulative, or cross validated predicted probabilities. P=name -names the variable containing the predicted probabilities. For the events/trials syntax or single-trial syntax with binary response, it is the predicted event probability
This is Niave Approach
Ranks var phat, making new variablepsquintile
Distribution of Y versus confounder x1 for dataset 1 10:31 Sunday, September 23, 2007
The FREQ ProcedureTable of psquintile by Treat
psquintile(Rank for Variable phat)Treat
Col Pct ‚ 0‚ 1‚ Total
ƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆ0 ‚ 100 ‚ 0 ‚ 100‚ 20.00 ‚ 0.00 ‚ 20.00‚ 100.00 ‚ 0.00 ‚‚ 40.00 ‚ 0.00 ‚
ƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆ1 ‚ 92 ‚ 8 ‚ 100
‚ 18.40 ‚ 1.60 ‚ 20.00‚ 92.00 ‚ 8.00 ‚‚ 36.80 ‚ 3.20 ‚
ƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆ2 ‚ 47 ‚ 53 ‚ 100
‚ 9.40 ‚ 10.60 ‚ 20.00‚ 47.00 ‚ 53.00 ‚‚ 18.80 ‚ 21.20 ‚
ƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆ3 ‚ 10 ‚ 90 ‚ 100
‚ 2.00 ‚ 18.00 ‚ 20.00‚ 10.00 ‚ 90.00 ‚‚ 4.00 ‚ 36.00 ‚
ƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆ4 ‚ 1 ‚ 99 ‚ 100
‚ 0.20 ‚ 19.80 ‚ 20.00‚ 1.00 ‚ 99.00 ‚‚ 0.40 ‚ 39.60 ‚
ƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒˆTotal 250 250 500
Source DF Type III SS Mean Square F Value Pr > F
psquintile 4 49.49029688 12.37257422 117.18 <.0001Treat 1 0.94311653 0.94311653 8.93 0.0029psquintile*Treat 3 0.22326479 0.07442160
For X1
Source DF Type III SS Mean Square F Value Pr > F
psquintile 4 49.49029688 12.37257422 117.18 <.0001
Treat 1 0.94311653 0.943116538.93 0.0029
psquintile*Treat 3 0.22326479 0.07442160
For X3
Y=outcome
IV
quintile
Quintile Method (no interaction)
Naïve Method
Quintile Method
Surrogate phat for covariates
IPTW
Results spew
FORMAT variable-1 <. . . variable-n> format
Results and Discussion:
-0.120.27-
30.0%67.5%0.280.671.642assymetric4
-0.022-0.076-5.5%-
19.0%0.3780.3241.133lumpbroad3
0.0710.01117.8%2.7%0.4710.4111.452lump2
-0.0250.001-6.3%-0.3%0.3750.3991.621lump1
Delta Full
Delta Quint
Error Full
Error Quint
Full Model
Quintile MethodNiaveDescription
Distribution
A 2
3
1
4
{
Y= a1*x1 + a2*x2 + a3*x3 + a4*x4 + b*T+ c1*T*x1 + c2*T*x2 + c3*T*x3 + c4*T*x4 + 0.4*Norm(0,1)
where, a1=0.28,a2=0.13,a3=0.23,a4=0.352; b=0.4; c1=c2=c3=0.2,c4=0.02
Yellow Highlighted has better performance
• Note that in cases where the propensity scores behave “well” and cover a wide range of values, the propensity quintile method outperformed the Full regression model:
• model y = treat x1 x2 x3 x4 treat*x1 treat*x2 treat*x3 treat*x4
• However, when the propensity scores are more asymmetrically distributed or mixed, the propensity method was less useful. As we taught ourselves, the efficacy of the propensity scoring technique depended on its ability to create a well-behaved function, and on the ‘matching’ technique. Matching can be improved with better classification schemes; we performed a simple quintile stratification.
Discussion
And now for something….Completely Different
iiii XWY εβτα +++= '
iiii ACXMY +++= 'τα
(see next page)
iiii XWY εβτα +++= '
•Zi is an observed variable that affects selection into the treatment
•Wi = 1 for individuals in the treated group. Wi = 0 for those in the control group.
Futures• General Boosting Methods (in R) – i.e. more
fancy matching methods• True Multivariate Distributions - (to simulate
covariates better use a multivariate model). I think there are some IML routines out there, didn’t have time to play with them.
Conclusion:• Our approach to demonstrating the Propensity
Scoring technique by pictorial description over mathematical formulation was better received by faculty and students, helped our understanding, and built collaboration.
• This demonstration provided a tool that students and faculty can use to better understand the propensity technique.
• This work can be extended to the multiple varieties of discriminant/classification schemes available for matching. Regardless, users of the technique should beware of its pitfalls and challenges once they have understood the base method.