Propensity Score Methods Using Propensity Score Methods Using SASSAS®®

R. Scott Leslie, MPHR. Scott Leslie, MPHMedImpact Healthcare Systems, Inc.MedImpact Healthcare Systems, Inc.

San Diego, CASan Diego, CAscott.leslie@medimpact.comscott.leslie@medimpact.com

Observational ResearchObservational Research

Key strength: estimate effect of exposures or Key strength: estimate effect of exposures or treatment in treatment in ““real worldreal world”” conditionsconditionsAdvantagesAdvantages–– Data readily available, inexpensiveData readily available, inexpensive–– Generate quick resultsGenerate quick results–– Results more generalizable than controlled trialsResults more generalizable than controlled trials–– Offer solution to limitations of RCTOffer solution to limitations of RCT

EthicsEthicsFeasibilityFeasibility-- Costs/resourcesCosts/resourcesTimeTime-- results lagresults lagHawthorne effectHawthorne effectExternal validityExternal validity-- patient mixpatient mixSmall samplesSmall samples

Observational ResearchObservational Research

Key limitation: comparison groups not randomizedKey limitation: comparison groups not randomizedConsequence: biased estimate of treatmentConsequence: biased estimate of treatmentDisadvantagesDisadvantages–– Lack of randomizationLack of randomization–– Differential selectionDifferential selection-- leads to differences in observed leads to differences in observed

and unobserved characteristicsand unobserved characteristics–– Heterogeneity of populationsHeterogeneity of populations–– Varying statistical analysesVarying statistical analyses

Bias in Observational StudiesBias in Observational StudiesIs outcome due to treatment? Or other factors?Is outcome due to treatment? Or other factors?Limited by biasLimited by bias–– Selection biasSelection bias–– ConfoundingConfounding–– Reverse causalityReverse causality

Selection bias– General definition by Rothman- a distortion resulting from the

manner in which subjects are selected into the study population

– Specified by Faries- differential probability of an individual assigned to a treatment condition and the characteristics of that individual are confounded with treatment outcomes

– Overt (observed) and hidden (unobserved)Rothman KJ, Greenland S. Modern Epidemiology, 3rd Edition. Lippincott Williams & Wilkins. 2008.Faries et al, Analysis of Observational Health Care Data Using SAS, SAS Institute. 2010.

Drug A Drug B

Guidance on Observational ResearchGuidance on Observational Research

Good practices for observational studies– The International Society of Pharmacoepidemiology (ISPE)– International Society for Pharmacoeconomics and Outcomes Research (ISPOR)

Methods for CER reviews– The Effective Health Care Program working document. Effective Healthcare Methods Guide for CER

Reviews– Guide on evaluating quality CER - The GRACE Initiative (Good Research for Comparative Effectiveness)

Guidelines on reporting observational CER studies– The STROBE (Strengthening the Reporting of Observational Studies) Guidelines

Guidelines when working with patient registries – AHRQ, Registries for Evaluating Patient Outcomes: A User's Guide

Guidelines on Systematic Reviews - Cochrane Handbook for Systematic Reviews of Interventions

Dreyer, Epidemiology, 2011Sturmer, Epidemiology,2011

Methods to Estimate EffectsMethods to Estimate Effects

Design stageDesign stage–– Match subjectsMatch subjects–– Exclusion and inclusion criteria Exclusion and inclusion criteria

Analysis stageAnalysis stage-- Use statistical techniques Use statistical techniques –– Regression, ANCOVA, propensity scoringRegression, ANCOVA, propensity scoring

Goal = Goal = balance groups balance groups onon characteristicscharacteristicsmimic randomization or simulate random treatment mimic randomization or simulate random treatment

assignment , assignment , ““quasi randomizationquasi randomization””more confident stating outcome is due to treatment more confident stating outcome is due to treatment

vs. explained by other factorsvs. explained by other factorsD’Agostino Sr., Medical Care, 1995

Propensity Score Methods as a Propensity Score Methods as a Potential FixPotential Fix

Traditional techniques (e.g., regression adjustment) may be limited if using too few covariates in adjustment process

Propensity score techniques avoids limitation– Summarizes covariate information into a single score

Editorial by DEditorial by D’’Agostino (Jr. and Sr.) in JAMAAgostino (Jr. and Sr.) in JAMA–– Use 2 methods to adjust for group differencesUse 2 methods to adjust for group differences

Propensity scoringPropensity scoring-- balance groupsbalance groupsAnalysis of covarianceAnalysis of covariance-- add precisionadd precision

DD’’Agostino RB Jr & Sr, JAMA, 2007Agostino RB Jr & Sr, JAMA, 2007

What is the Propensity Score?

The propensity score is the conditional probability of The propensity score is the conditional probability of being treated based on individual covariatesbeing treated based on individual covariates–– Rosenbaum and Rubin demonstrated p scores can account for Rosenbaum and Rubin demonstrated p scores can account for

imbalances in treatment groups and reduce bias by resembling imbalances in treatment groups and reduce bias by resembling randomization of subjects into treatment groupsrandomization of subjects into treatment groups

Propensity score techniques used to compare groups Propensity score techniques used to compare groups while adjusting for group differenceswhile adjusting for group differences–– Regression adjustmentRegression adjustment–– MatchingMatching–– Stratification (subclassification)Stratification (subclassification)

Rosenbaum P.R. and Rubin D.B. 1983. Rosenbaum P.R. and Rubin D.B. 1983. ““The Central Role of the Propensity Score in The Central Role of the Propensity Score in Observational Studies for Causal EffectsObservational Studies for Causal Effects””, , BiometrikaBiometrika, 70, 41, 70, 41--55.55.

Creating Propensity Scores Using Creating Propensity Scores Using PROC LOGISTICPROC LOGISTIC

Logistic regression: Used to predict probability of event occurLogistic regression: Used to predict probability of event occurring ring as a function of independent variables (continuous and/or as a function of independent variables (continuous and/or dichotomous)dichotomous)

Logistic model:Logistic model:

Propensity scores created using PROC LOGISTIC or PROC Propensity scores created using PROC LOGISTIC or PROC GENMODGENMOD–– The propensity score is the conditional probability of each The propensity score is the conditional probability of each

patient receiving a particular treatment based on prepatient receiving a particular treatment based on pre--treatment treatment variablesvariables

–– Creates data set with predicted probabilities as a variableCreates data set with predicted probabilities as a variable–– Or use logit of p score log (1/1Or use logit of p score log (1/1--p)p)

)(11)(

iXieYP βα Σ+−+

=

Creating Propensity Scores: CodeCreating Propensity Scores: Code

proc logistic data = indsn;proc logistic data = indsn;class naive0;class naive0;model tx (event=model tx (event=’’Drug ADrug A’’) = age female b_hmo ) = age female b_hmo

pre_drug_cnt_subset naive0 pre_refill_pct pre_drug_cnt_subset naive0 pre_refill_pct copay_idxdrug pre_sulf pre_htn pre_asthma copay_idxdrug pre_sulf pre_htn pre_asthma pre_pain pre_lipo pre_depresspre_pain pre_lipo pre_depress

/link=logit rsquare;/link=logit rsquare;output out = psdataset pred = ps output out = psdataset pred = ps

xbeta=logit_ps;xbeta=logit_ps;run;run;

PS= predicted event probability of receiving treatment based on specified

factors

Estimating P Scores

Propensity score is the conditional probability of Propensity score is the conditional probability of each patient receiving a particular treatment each patient receiving a particular treatment based on prebased on pre--treatment variablestreatment variables–– More covariates better than less (Austin, 2007)More covariates better than less (Austin, 2007)–– Include characteristics that are unbalanced b/w Include characteristics that are unbalanced b/w

treatment groupstreatment groups–– Success: Did it balance treatment groups?Success: Did it balance treatment groups?–– Michael Doherty SAS paper/macroMichael Doherty SAS paper/macro

Rosenbaum P, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika 1983:70:41-55

Distribution of Propensity ScoresDistribution of Propensity Scores

Distribution of Propensity ScoresDistribution of Propensity Scores

Distribution of P Scores: CodeDistribution of P Scores: Code

proc univariate data=psds plot;title 'Histograms of Propensity Scores by Treatment Group';var ps;class tx;histogram ps / ctext=purple cfill=blue

kernel (k=normal color=green w=3 l=1)normal (color = red w=3 l= 2)ncols=1 nrows=2;

inset n='N' (comma6.0) mean='Mean' (6.2) median='Median' (6.2) mode='Mode'(6.2)normal kernel(type) / position=NW;

run;

Propensity Score Methods

Once the propensity score is calculated what to do you with them?

3 common methods as stated by Rosenbaum and Rubin, 1984–– Regression adjustmentRegression adjustment–– Stratification (subclassification)Stratification (subclassification)–– MatchingMatching

Rosenbaum P.R. and Rubin D.B. 1983. Rosenbaum P.R. and Rubin D.B. 1983. ““The Central Role of the Propensity Score in The Central Role of the Propensity Score in Observational Studies for Causal EffectsObservational Studies for Causal Effects””, , BiometrikaBiometrika, 70, 41, 70, 41--55.55.

Regression Adjustment MethodRegression Adjustment Method

Use p score as a covariate outcome model– Or use propensity score weights as the inverse of

propensity score

Use PROC GLM or PROC LOGISTIC to model outcome

– Add independent variables believed to confound outcome

Second step of 2 stage processSecond step of 2 stage process1.1. Use propensity scores to balance groupsUse propensity scores to balance groups2.2. Use ANCOVA modeling to create precisionsUse ANCOVA modeling to create precisions

Regression Adjustment: CodeRegression Adjustment: Code

Model continuous outcome adjusting for p scores/*create p score*//*create p score*/proc logistic data = indsn;proc logistic data = indsn;class naive0;class naive0;model tx (event=model tx (event=’’Drug ADrug A’’) = /*pre_tx_vars*/ ivar1 ivar2) = /*pre_tx_vars*/ ivar1 ivar2

/link=logit rsquare;/link=logit rsquare;output out = ps_dataset pred = ps xbeta=logit_ps;output out = ps_dataset pred = ps xbeta=logit_ps;run;run;

/*outcome model adjusting for p score*//*outcome model adjusting for p score*/procproc glmglm data = data = ps_dataset ps_dataset ;;class tx;class tx;model pdc = tx ps /solution;model pdc = tx ps /solution;lsmeans tx / om adjust = tukey pdiff cl;lsmeans tx / om adjust = tukey pdiff cl;quitquit;;

Stratifying by P Score: ObjectiveStratifying by P Score: Objective

Stratification, subclassification or binning Stratification, subclassification or binning involves grouping subjects into strata based on involves grouping subjects into strata based on subjectsubject’’s observed characteristicss observed characteristicsUse calculated p scores to place subjects into Use calculated p scores to place subjects into stratastrataObjective = subjects in the same stratum are Objective = subjects in the same stratum are similar in the characteristics used in the similar in the characteristics used in the propensity score development processpropensity score development process

Stratifying Propensity Scores

Bin 1

Bin 2Bin 3

Bin 4

Bin 5

Cochran, Biometrics, 1968 Cochran, Biometrics, 1968 -- 5 strata can remove 90% of the bias5 strata can remove 90% of the bias

Stratifying by P Score: CodeStratifying by P Score: Code

/*create 5 quintiles of p scores */

proc rank data = psdataset groups=5 out = rank_ds;

ranks rank;var ps;

data quintile;set rank_ds;quintile = rank + 1;run;

Stratifying by P Score: Estimate EffectStratifying by P Score: Estimate Effect

Result of code is 5 bins of homogenous subjects– Check differences between treatment groups– Sensitivity analysis if distributions don’t

overlapOutcomes can be compared within the 5 subclassesCalculate weighted mean of the subclasses to report an overall treatment effect

Stratifying by P Score: CodeStratifying by P Score: Code

Model continuous outcome adjusting for p scores

/*outcome model adjusting for quintile of p score*//*outcome model adjusting for quintile of p score*/procproc glmglm data = quintile;data = quintile;class tx;class tx;model pdc = tx quintile /solution;model pdc = tx quintile /solution;lsmeans tx / om adjust = tukey pdiff cl;lsmeans tx / om adjust = tukey pdiff cl;quitquit;;

P Score MatchingP Score Matching

Matching groups by p scores can balance Matching groups by p scores can balance groups on covariatesgroups on covariatesSubjects are matched by single score vs. Subjects are matched by single score vs. by one or more variables (traditional direct by one or more variables (traditional direct matching)matching)ChallengesChallenges–– incomplete matching (canincomplete matching (can’’t find a match)t find a match)–– inexact matching (how close is a match)inexact matching (how close is a match)

Matching by CharacteristicMatching by Characteristic

Match on single or multiple characteristicsMatch on single or multiple characteristics–– e.g., age, gender, disease severity, health plan, etc.e.g., age, gender, disease severity, health plan, etc.

1:1 or 1:many1:1 or 1:many

controlsuntreated

non intervention

casestreated

intervention

P Score MatchingP Score MatchingI need a match! Does

anyone have a propensity score near

0.824?

My propensity score is 0.859. Is

that close enough?

P Score Matching MethodsP Score Matching MethodsTechniquesTechniques–– StratifiedStratified–– Nearest neighborNearest neighbor–– Radius matchingRadius matching–– CaliberCaliber–– Kernal matchingKernal matching–– Mahalanobis metricMahalanobis metric

ReplacementReplacement-- back in pool for further possible matchingback in pool for further possible matchingW/o replacement or greedy algorithmW/o replacement or greedy algorithm-- find match and find match and keep itkeep itWhich is appropriate? Literature offers some guideWhich is appropriate? Literature offers some guide–– With replacement when matching pool is smallWith replacement when matching pool is small–– 2 to 1 match if control group is large2 to 1 match if control group is large–– Ease of calculationEase of calculation

GoalGoal-- Increase balance between groupsIncrease balance between groups

Baser, Value in Health, 2006; Austin, Biometrical Journal, 2009

PS Matching Using Greedy AlgorithmPS Matching Using Greedy Algorithm

Example of caseExample of case--control match using a control match using a greedy matching algorithmgreedy matching algorithmNearest available pair methodNearest available pair methodReducing the non matches and inexact Reducing the non matches and inexact matchesmatchesP scores used to balance treated and P scores used to balance treated and untreated groupsuntreated groups

Parsons, Lori. 2000. “Using SAS® Software to Perform a Case Control Match on Propensity Score in anObservational Study”. Proceedings of the Twenty-Fifth Annual SAS Users Group International Conference,Indianapolis, IN, 214-26.

PS Matching Using Greedy AlgorithmPS Matching Using Greedy AlgorithmTable 1: Original Population

Early Intervention N (%) Conservative N (%) p-valueTotal Patients 2,402 17,735Age (Mean±sd) 61.3 ±12.2 68.2±13.0 <0.0001Male Gender 1,744 (72.6) 10,914 (61.5) <0.0001White Race 2,079 (91.8) 15,002 (88.4) <0.0001Hx Angina 444 (18.5) 4,441 (25.0) <0.0001Hx MI 574 (23.9) 5,382 (30.3) <0.0001

Table 2: Greedy 5 to 1 Digit Matched PopulationEarly Intervention N (%) Conservative N (%) p-value

Total Patients 2,036 2,036Age (Mean±sd) 61.9 ±12.0 61.7±13.3 0.5405Male Gender 1,452 (71.3) 1,445 (71.0) 0.8087White Race 1,865 (91.6) 1,858 (91.3) 0.6952Hx Angina 390 (19.2) 381 (18.7) 0.7189Hx MI 488 (24.0) 491 (24.1) 0.9124

SummarySummary

Propensity score as the conditional probability of Propensity score as the conditional probability of treatment (or desired event) summarizes treatment (or desired event) summarizes observed values into a single scoreobserved values into a single scoreP scores uses:P scores uses:–– Match subjectsMatch subjects–– Stratify subjectsStratify subjects–– As a covariateAs a covariate

Purpose = Purpose = balancing groupsbalancing groups to remove bias to remove bias when assessing treatment effect on outcomeswhen assessing treatment effect on outcomes

AdvantagesAdvantages

Summarizes observed values into a single Summarizes observed values into a single score less sensitive to model score less sensitive to model misspecification misspecification –– Traditional techniques may be limited if Traditional techniques may be limited if

accounting for only a few covariatesaccounting for only a few covariatesP scores can diagnose comparability of P scores can diagnose comparability of groups before modeling stagegroups before modeling stage–– Distributions overlap?Distributions overlap?

If comparison groups are too different >>>difficult If comparison groups are too different >>>difficult to balance groupsto balance groups

P score is more robust approachP score is more robust approach–– Address selection bias and offers precisionAddress selection bias and offers precision

Assumptions & DisadvantagesAssumptions & Disadvantages

AssumptionsAssumptions–– All covariates that affect both treatment and outcome must be All covariates that affect both treatment and outcome must be

included in the model. How do you determine this?included in the model. How do you determine this?–– All patients have a non zero probability of receiving each All patients have a non zero probability of receiving each

treatmenttreatment

DisadvantagesDisadvantages–– Incorporates observed characteristics and thus doesnIncorporates observed characteristics and thus doesn’’t account t account

for unobserved factors, e.g., patient attitudes, socioeconomic for unobserved factors, e.g., patient attitudes, socioeconomic status, and education levelstatus, and education level

Modified if unobserved factors are correlated to observed factorModified if unobserved factors are correlated to observed factorss–– Large samples sizes may be needed to establish adequate Large samples sizes may be needed to establish adequate

variance in covariate distributionsvariance in covariate distributions

ConclusionConclusionSelection bias may create biased estimate of your Selection bias may create biased estimate of your outcome in observational studiesoutcome in observational studiesP score methods used to adjust for selection biasP score methods used to adjust for selection biasUse with traditional risk adjustment techniques to reduce Use with traditional risk adjustment techniques to reduce bias and better describe the effect of exposure on bias and better describe the effect of exposure on outcomesoutcomesMinimizes bias, not total adjustmentMinimizes bias, not total adjustmentObservables vs. unobservables: Instrumental variable Observables vs. unobservables: Instrumental variable method account for unobservablesmethod account for unobservablesUse multiple methods and consistent results add Use multiple methods and consistent results add robustness of researchrobustness of research

Questions and Comments

Thank you, BASUG and BASUG OfficersSpecial thanks to,– Bridget Neville– Karen Olsen

ReferencesReferencesAustin PC. Some methods of propensity-score matching had superior performance to others: results of an empirical investigation and Monte Carlo simulation. Biometrical Journal. 2009;51:171-184.Baser O. Too much ado about propensity score models? Comparison of types of propensity score matching. Value in Health. 2006; 9(6):677-385.Cochran WG. The effectiveness of adjustment b subclassification in removing bias in observational studies. Biometrics. 1968; 24:295-313.D’Agostino RB Sr, Kwan H. Measuring effectiveness: what to expect without a randomized control group. Medical Care. 1995;195:33 (4 suppl): AS95-AS105.D’Agostino RB, Jr, D’Agostino RB, Sr. Estimating treatment effects using observational data. JAMA. 2007;297 (3): 314-316. D’Agostino RB. Tutorial on Biostatistics: Propensity Score Methods for Bias Reduction in the comparison of a treatment to a non-randomized control group. Statistics in Medicine. 1998;17:2265-2281.Doherty M. Automating the process of choosing among highly correlated covariates for multivariable logistic regression. Proceedings of the 2008 Western Users of SAS Software Conference, Los Angeles, CA.Dreyer NA. Making observational studies count. Epidemiology. 2011; 22(3):295-297.Glynn RJ, Schneeweiss S, Sturmer T. Indications for propensity scores and review of their use in pharmacoepidemiology. Basic & Clinical Pharmacology & Toxicology. 2006, 98, 253–259.Faries, Douglas, Andrew C. Leon, Josep Maria Haro and Robert L. Obenchain. 2010. Analysis of Observational Health Care Data Using SAS®.Cary, NC: SAS Institute Inc.

References cont.References cont.Parsons, Lori. 2000. “Using SAS® Software to Perform a Case Control Match on Propensity Score in an Observational Study”. Proceedings of the Twenty-Fifth Annual SAS Users Group International Conference,Indianapolis, IN, 214-26.Rosenbaum PR, Rubin DB, The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70:41-55.Rosenbaum P.R. and Rubin D.B. 1984. “Reducing bias in observational studies using subclassification on the propensity score. J Am Stat Assoc. 1984;79:516-524.Rothman KJ, Greenland S. Modern Epidemiology, 3rd Edition. Lippincott Williams & Wilkins. 2008.Schlesselman JJ. Case-Control Studies: Design, Conduct, and Analysis. New York, Oxford University Press, 1982Shah BR , Laupacis A, Hux JE, Austin PC. Propensity score methods gave similar results to traditional regression modeling in observational studies: a systematic review. J. Clin. Epidemiol. 2005;58: 550–559. Stürmer T, Joshi M, Glynn RJ, Avorn J, Rothman K, Schneeweiss S. A review of the application of propensity score methods yielded increased use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. J. Clin. Epidemiol. 2006Sturmer T, Funk MJ, Poole C, Brookhart MA. Nonexperimental comparative effectiveness research using linked healthcare databases. Epidemiology. 2011; 22(3):298-301.

SAS ReferencesSAS ReferencesSAS Institute Inc. 2004. SAS Institute Inc. 2004. ““SAS Procedures: The LOGISTIC SAS Procedures: The LOGISTIC ProcedureProcedure””. . SAS SAS OnlineDocOnlineDoc®® 9.1.3.9.1.3. Cary, NC: SAS Institute Inc. Cary, NC: SAS Institute Inc. http://support.sas.com/documentation/onlinedoc/91pdf/sasdoc_913/http://support.sas.com/documentation/onlinedoc/91pdf/sasdoc_913/base_proc_8977_new.pdfbase_proc_8977_new.pdf

SAS Institute Inc. 2004. SAS Institute Inc. 2004. ““SAS Procedures: The GLM ProcedureSAS Procedures: The GLM Procedure””. . SAS SAS OnlineDocOnlineDoc®® 9.1.3.9.1.3. Cary, NC: SAS Institute Inc. Cary, NC: SAS Institute Inc. http://support.sas.com/documentation/onlinedoc/91pdf/sasdoc_913/http://support.sas.com/documentation/onlinedoc/91pdf/sasdoc_913/base_proc_8977_new.pdfbase_proc_8977_new.pdf