methods for cost estimation in cea: the glm approach henry glick university of pennsylvania
DESCRIPTION
Methods for Cost Estimation in CEA: the GLM Approach Henry Glick University of Pennsylvania AcademyHealth Issues in Cost-Effectiveness Analysis Washington, DC 06/10/2008. Outline. Policy-relevant parameter for cost-effectiveness Problems posed by nonnormality of cost data - PowerPoint PPT PresentationTRANSCRIPT
Methods for Cost Estimation in CEA: the GLM Approach
Henry Glick
University of Pennsylvania
AcademyHealth
Issues in Cost-Effectiveness Analysis
Washington, DC
06/10/2008
Outline
• Policy-relevant parameter for cost-effectiveness • Problems posed by nonnormality of cost data • Generalized linear models as a response to the
problems– Identifying links and families (gets a little technical)
• General comments
My objective is to provide practical advice for ways implement GLM models. Slides available at:
http://www.uphs.upenn.edu/dgimhsr/presentations.htm
Policy Relevant Parameter for CEA
• Policy relevant parameter: differences in the arithmetic, or sample, mean– In welfare economics, a project is cost-beneficial if the
winners from any policy gain enough to be able to compensate the losers and still be better off themselves
• Thus, we need a parameter that allows us to determine how much the losers lose, or cost, and how much the winners win, or benefit
– From a budgetary perspective, decision makers can use the arithmetic mean to determine how much they will spend on a program
Policy Relevant Parameter for CEA (2)
• In both cases, substitution of some other parameter for the sample mean can be justified only if it provides a better estimate of gains and losses or spending
Are Sample Means Always the Best Estimator?
• In simulation, when cost data are sufficiently nonnormal, the relative bias (truth - observed)2 for other parameters such as the median or adjusted geometric mean can sometimes be lower than the relative bias observed for the arithmetic mean
• HOWEVER,– Distribution required to be sufficiently nonnormality
that ln(cost) is also substantially nonnormally distributed
– In actual data, since we never know truth, it is difficult to determine whether other parameters will have lower relative bias than sample mean
The Problem
• Common feature of cost data is right-skewness (i.e., long, heavy, right tails)
05
.0e-
051
.0e-
041
.5e-
04
0 10000 20000 30000 40000 0 10000 20000 30000 40000
0 1
Den
sity
True other medical costGraphs by treat
The Problem (cont.)
• Distributions with long, heavy, right tails will have a mean that differs from the median, independent of “outliers”
• Cost data also can’t be negative, and can have large fractions of observations with 0 cost
• Nonnormality of cost data can pose problems for common parametric tests such as t-test, ANOVA, and OLS regression
Common (Relatively Bad) Responses To Violation Of Normality
• Adopt nonparametric tests of other characteristics of the distribution that are not as affected by the nonnormality of the distribution (“biostatistical” approach)
• Transform the data so they approximate a normal distribution (“classic econometric” approach)
Recommended Response: Adopt More Flexible Models
• Generalized Linear Models (GLM)– Have the advantages of the log models, but
• don’t require normality or homoscedasticity• and evaluate a direct of the difference in cost and
don’t raise problems related to retransformation from the scale of estimation to the raw scale
– To build them, one must identify a "link function" and a "family“ (based on the data)
Stata and SAS Code
• STATA code:
glm y x, link(linkname) family (familyname)• General SAS code (not appropriate for gamma family /
log link):
proc genmod;
model y=x/ link=linkname dist=familyname;
run;
• When running gamma/log models, the general SAS code drops observations with an outcome of 0
• If you want to maintain these observations and are predicting y as a function of x (M Buntin):
proc genmod;
a = _mean_;
b = _resp_;
d = b/a + log(a)
variance var = a2
deviance dev =d;
model y = x / link = log;
run;
SAS Code for a Gamma Family / Log Link
The Link Function
• Link function directly characterizes how the linear combination of the predictors is related to the prediction on the original scale– e.g., predictions from the identity link -- which is used
in OLS -- equal:
ˆi i iY = β X
The Log Link
• Log link is most commonly used in literature• When we adopt the log link, we are assuming:
ln(E(y/x))=Xβ• GLM with a log link differs from log OLS in part because
in log OLS, one is assuming:
E(ln(y)/x)=Xβ• ln(E(y/x) ≠ E(ln(y)/x)
i.e. log of the mean mean of the log costs
ln(E(y/x) ≠ E(ln(y)/x)
Variable Group 1 Group 2
Observations
1 15 35
2 45 45
3 75 55
Arithmetic mean 45 45
Log, arith mean cost 3.806662 3.806662 *
Natural log
1 2.70805 3.555348
2 3.806662 3.806662
3 4.317488 4.007333
Arith mean, log cost 3.610734 3.789781 †
* Difference = 0; † Difference = 0.179047
Comparison of Results of GLM Gamma/Log and log OLS Regression
Variable Coefficient SE z/T p value
GLM, gamma family, log link
Group 2 0.000000 0.405730 0.00 1.000
Constant 3.806662 0.286894 13.27 0.000
Log OLS
Group 2 0.179048 0.492494 0.36 0.74
Constant 3.610734 0.348246 10.32 0.000
The Power Link Function
• Stata’s power link provides a flexible link function
• It allows generation of a wide variety of named and unnamed links, e.g.,
– power 1 = Identity link; = BiXi
– power .5 = Square root link; = (BiXi)2
– power .25: = (BiXi)4
– power 0 = log link; = exp(BiXi)
– power -1 = reciprocal link; = 1/(BiXi)
– power -2 = inverse quadratic; = 1/(BiXi)0.5
ˆiu
ˆiu
ˆiu
ˆiu
ˆiu
ˆiu
Negative Power Links
• Retranslation of negative power links to the raw scale:
• When using a negative power link, negative coefficients yield larger estimates on the raw scale; positive coefficients yield smaller estimates
i 1
abs(power)i i
1y =
B X
Selecting a Link Function
• There is no single test that identifies the appropriate link• Instead can employ multiple tests of fit
– :Pregibon link test checks linearity of response on scale of estimation
– Modified Hosmer and Lemeshow test checks for systematic bias in fit on raw scale
– Pearson’s correlation test checks for systematic bias in fit on raw scale
– Ideally, all 3 tests will yield nonsignificant p-values• Others (e.g., Hardin and Hilbe) have proposed use of
(larger) log likelihood, (smaller) deviance, AIC and BIC statistics
The Family
• Specifies the distribution that reflects the mean-variance relationship– Gaussian: Constant variance– Poisson: Variance is proportional to mean– Gamma: Variance is proportional to square of mean– Inverse Gaussian or Wald: Variance is proportional to
cube of mean• Use of the poisson, gamma, and inverse Gausian
families relax the assumption of homoscedasticity
• A “constructive” test that recommends a family given a particular link function
• Implemented after GLM regression that uses the particular link
• The test predicts the square of the residuals (res2) as a function of the log of the predictions (lnyhat) by use of a GLM with a log link and gamma family to– Stata code
glm res2 lnyhat,link(log) family(gamma), robust• If weights or clustering are used in the original GLM,
same weights and clustering should be used for modified Park test
Modified Park Test
• Recommended family derived from the coefficient for lnyhat:– If coefficient ~=0, Gaussian– If coefficient ~=1, Poisson– If coefficient ~=2, Gamma– If coefficient ~=3, Inverse Gaussian or Wald
• Given the absence of families for negative coefficients:– If coefficient < -0.5, consider subtracting all
observations from maximum-valued observation and rerunning analysis
Recommended Family, Modified Park Test
Variance function: V(u) = u^2Link function: g(u) = ln(u)
[Gamma]
[Log]
Deviance = 214.8893Log likelihood = -20782.51
AIC 20.7895BIC -14933.71
cost Coef Std Err z P>|t| 95% CI
treat -.0361495 .015123 -2.39 0.017 -.065790 -.006510
dissev1 .1250814 .101609 1.23 0.218 -.074696 .324232
dissev2 -.6837741 .076062 -8.99 0.000 -.832853 -.534696
blcost .0069198 .023707 0.29 0.770 -.039545 .053385
blqscore -.1993532 .046650 -4.27 0.000 -.290786 -.107921
bledvis .0158549 .005082 3.12 0.002 .005894 .025816
_cons 9.756198 .090169 108.20 0.000 9.57947 9.93293
Example, GLM gamma/log
miscand1.dta
glm cost treat dis* bl*,link(log) family(gamma)
FITTED MODEL: Link = Log ; Family = Gamma
Results, Modified Park Test (for Family)
Coefficient: 1.1680
Family, Chi2, and p-value in descending order of likelihood
Family Chi2 P-value
Poisson: 0.0675 0.7951
Gamma: 1.6539 0.1984
Gaussian NLLS: 3.2599 0.0710
Inverse Gaussian or Wald: 8.0193 0.0046
Results of tests of GLM Identity link
Pearson Correlation Test: 0.9688
Pregibon Link Test: 0.8529
Modified Hosmer and Lemeshow: 0.8818
GLM Diagnostics, log/gamma
miscand1.dta
Variance function: V(u) = uLink function: g(u) = ln(u)
[Poisson]
[Log]
Deviance = 25171259Log likelihood = -1296805
AIC = 1296.812BIC = 2556110
cost Coef Std Err z P>|t| 95% CI
treat -.0342075 .00041 -83.64 0.000 -.035010 -.033406
dissev1 .1209046 .00274 44.12 0.000 .115533 .126276
dissev2 -.6857563 .00206 -333.4 0.000 -.689788 -.681725
blcost .0072812 .00064 11.41 0.000 .006031 .008532
blqscore -.1970875 .00125 -157.6 0.000 -.199538 -.194637
bledvis .0159274 .00014 116.7 0.000 .015660 .016195
_cons 9.756579 .00243 4014 0.000 9.75182 9.76134
Change Family to Poisson and Rerun Model
miscand1.dta
FITTED MODEL: Link = Log ; Family = Poisson
Results, Modified Park Test (for Family)
Coefficient: 1.1627
Family, Chi2, and p-value in descending order of likelihood
Family Chi2 P-value
Poisson: 0.0621 0.8032
Gamma: 1.6460 0.1995
Gaussian NLLS: 3.1734 0.0748
Inverse Gaussian or Wald: 7.9249 0.0049
Results of tests of GLM Identity link
Pearson Correlation Test: 0.9882
Pregibon Link Test: 0.8136
Modified Hosmer and Lemeshow: 0.8928
GLM Diagnostics, log/poisson
miscand1.dta
Passes Tests, But Can We Improve the Link?
• Iteratively evaluate power links (in 0.1 intervals) between -2 and 2– Use the modified Park test to select a family– Evaluate the fit statistics– Don’t show you the results here, but I then fine tune
the power link in 0.01 intervals within candidate regions of the power link
Results for Selected Power Links
Power Family Pearson Pregibon mH&M
-0.2 Poisson 0.9835 0.7089 0.8929
-0.1 Poisson 0.9865 0.761 0.8943
0.0 Poisson 0.9882 0.8136 0.8928
0.1 Poisson 0.9905 0.8669 0.9063
0.2 Poisson 0.9934 0.9209 0.9559
0.3 Poisson 0.9969 0.9737 0.9461
0.4 Poisson 0.9991 0.9730 0.9359
0.5 Poisson 0.9946 0.9201 0.9369
0.6 Poisson 0.9895 0.8678 0.9008
0.7 Poisson 0.9839 0.8164 0.8125
0.8 Poisson 0.9778 0.7661 0.7444
miscand1.dta
Individual Criteria Do Not Uniquely Identify Link, But…
Power Family Pearson Pregibon mH&M
-0.2 Poisson 0.9835 0.7089 0.8929
-0.1 Poisson 0.9865 0.761 0.8943
0.0 Poisson 0.9882 0.8136 0.8928
0.1 Poisson 0.9905 0.8669 0.9063
0.2 Poisson 0.9934 0.9209 0.9559
0.3 Poisson 0.9969 0.9737 0.9461
0.4 Poisson 0.9991 0.9730 0.9359
0.5 Poisson 0.9946 0.9201 0.9369
0.6 Poisson 0.9895 0.8678 0.9008
0.7 Poisson 0.9839 0.8164 0.8125
0.8 Poisson 0.9778 0.7661 0.7444
miscand1.dta
Can We Develop a Summary Measure?
Power Family Pearson Pregibon mH&M Summary
-0.2 Poisson 0.9835 0.7089 0.8929 .096426
-0.1 Poisson 0.9865 0.761 0.8943 .068476
0.0 Poisson 0.9882 0.8136 0.8928 .046376
0.1 Poisson 0.9905 0.8669 0.9063 .026586
0.2 Poisson 0.9934 0.9209 0.9559 .008356
0.3 Poisson 0.9969 0.9737 0.9461 .003607
0.4 Poisson 0.9991 0.9730 0.9359 .004839
0.5 Poisson 0.9946 0.9201 0.9369 .010395
0.6 Poisson 0.9895 0.8678 0.9008 .027428
0.7 Poisson 0.9839 0.8164 0.8125 .069124
0.8 Poisson 0.9778 0.7661 0.7444 .120533
miscand1.dta
LL, AIC, BIC, and Deviance
Power Family LL AIC BIC Deviance
-0.2 Poisson -1296851.7 1296.8587 2556204 2571353
-0.1 Poisson -1296825.5 1296.8325 2556152 2571300
0.0 Poisson -1296804.8 1296.8118 2556110 2571259
0.1 Poisson -1296789.7 1296.7967 2556080 2571229
0.2 Poisson -1296779.9 1296.7869 2556060 2571209
0.3 Poisson -1296775.5 1296.7825 2556052 2571200
0.4 Poisson -1296776.3 1296.7833 2556053 2571202
0.5 Poisson -1296782.3 1296.7893 2556065 2571214
0.6 Poisson -1296793.5 1296.8005 2556088 2571236
0.7 Poisson -1296809.8 1296.8168 2556120 2571269
0.8 Poisson -1296831.1 1296.8381 2556163 2571311
miscand1.dta
Why Not Simply Use AIC and BIC?
• In the current example:– AIC, BIC, log likelihood, and deviance all agreed– They yielded an answer that was similar answer to
that from the Pearson correlation test, Pregibon link test, and modified Hosmer and Lemeshow tests
• Power link 0.3• AIC, BIC, log likelihood, (and deviance?) already
commonly used for decisions about model fit• Why do we need the new tests?
AIC / BIC
• There are at least 3 reasons why in the long run log likelihood, AIC, BIC, and deviance are unlikely to be the recommended tests for identifying the appropriate link function
• First, when there are a large fraction of observations with zero cost:– The recommendations from log likelihood / AIC agree
with each other– The recommendations from BIC / deviance agree with
each other– But the log likelihood/AIC recommendations differ
from the BIC/deviation recommendations
AIC / BIC (2)
• Second, the 4 statistics aren’t stable across families, and the shifts in their magnitude across families do not provide information about which family/link is best
• For example, in a dataset where the modified Park test recommends a gamma family for power links < 0.4, but recommends a poisson family for power links > 0.5, the magnitude of the AIC statistic shifts from ~18 for the gamma family to ~454 for the poisson family– Although the smaller AIC values associated with
power links < 0.4 suggest that these links have the better fit, the Pearson, Pregibon, and H&M tests all suggest that the power links > 0.5 are actually superior
AIC / BIC (3)
• Third, while this instability across families is less of a problem when our statistical packages offer 4 continuous families only, it will eliminate comparability across links when statistical packages begin to offer more flexible GLM power families– i.e., when we don’t have to choose between Gaussian
(0) and poisson (1) families, but instead can use a family of 0.7
• In this case, given that each power will be associated with a slightly different family, it will be impossible to compare the resulting AIC/BIC statistics
• Basu and Rathouz (2005) have proposed use of extended estimating equations (EEE) which estimate the link function and family along with the coefficients and standard errors
• Tends to need a large number of observations (thousands not hundreds) to converge
• Currently can’t take the results and use them with a simple GLM command (makes bootstrapping of resulting models cumbersome)
Extended Estimating Equations
GLM Disadvantages
• Disadvantages– Can suffer substantial precision losses
• If heavy-tailed (log) error term, i.e., log-scale residuals have high kurtosis (>3)
• If family is misspecified
• The distribution of cost can pose problems for common parametric tests of cost
• Responses in the literature that suggest that we should evaluate something other than the difference in the sample mean (or a direct transformation of this difference) – e.g., nonparametric tests of other characteristics of the distribution or transformations of cost – generally create more problems than they solve
• Use of more flexible models that evaluate a direct transformation of the difference in cost generally pose fewer problems– Does require we identify functional forms for the
relationship between the predictors and the mean and for the variance structure
Review
EXTRA SLIDES
Fine Tuning (1)
Power Family Pearson Pregibon mH&M Summary
0.31 Poisson 0.9973 0.9790 0.9399 .00406
0.32 Poisson 0.9760 0.9844 0.9394 .00392
0.33 Poisson 0.9980 0.9897 0.9390 .00383
0.34 Poisson 0.9984 0.9951 0.9386 .00379
0.35 Poisson 0.9988 0.9996 0.9382 .00382
0.36 Poisson 0.9992 0.9943 0.9378 .00390
0.37 Poisson 0.9996 0.9889 0.9373 .00405
0.38 Poisson 1.0000 0.9836 0.9367 .00427
0.39 Poisson 0.9995 0.9783 0.9363 .00452
miscand1.dta
Fine Tuning (2)
Power Family LL AIC BIC Deviance
0.31 Poisson -1296775.324 1296.7823 2556051.2 2571199.801
0.32 Poisson -1296775.224 1296.7822 2556051.0 2571199.600
0.33 Poisson -1296775.176 1296.7822 2556050.9 2571199.505
0.34 Poisson -1296775.181 1296.7822 2556050.9 2571199.514
0.35 Poisson -1296775.238 1296.7822 2556051.0 2571199.628
0.36 Poisson -1296775.337 1296.7823 2556051.2 2571199.847
0.37 Poisson -1296775.509 1296.7825 2556051.6 2571200.170
0.38 Poisson -1296775.723 1296.7827 2556052.0 2571200.598
0.39 Poisson -1296776.989 1296.7830 2556052.5 2571201.130
miscand1.dta
AIC / BIC
• (When there are a large fraction of 0s) log likelihood / AIC recommendations differ from BIC / deviance recommendations
• 4 statistics aren’t stable across families, and shifts in their magnitude across families do not provide information about which family/link is best
Different Recommendations
Power Family LL AIC BIC Deviance
-0.5 Gamma -6408.4 18.767 -3144.6 1094.8
-0.4 Gamma -6399.1 18.740 -3127.7 1111.8
-0.3 Gamma -6390.3 18.714 -3103.9 1135.6
-0.2 Gamma -6382.3 18.691 -3073.6 1165.9
-0.1 Gamma -6374.1 18.667 -3035.8 1203.7
0.0 Gamma -6359.1 18.623 -2979.5 1259.9
heroin2.dta
4 Statistics Aren’t Stable Across Families
Power Family LL AIC BIC Deviance
-1.2 Gamma -4495.52 18.0061 -2985.78 84.2
-1.1 Gamma -4495.35 18.0054 -2986.12 83.9
-1.0 Gamma -4495.19 18.0048 -2986.44 83.6
. . . . . .
0.4 Gamma -4493.99 18.0000 -2988.85 81.2
0.5 Gamma -4493.97 17.9999 -2988.89 81.1
0.6 Poisson -113533 454.16 219123 222193
0.7 Poisson -113504 454.04 219066 222136
0.8 Poisson -113502 454.03 219061 222131
0.9 Poisson -113526 454.13 219109 222178
1.0 Poisson -113576 454.32 219210 222280
rchapter5.dta
4 Statistics Aren’t Stable Across Families
Power Family Pearson Pregibon mH&M
-1.2 Gamma .0102 .0004 .0275
-1.1 Gamma .0115 .0007 .0411
-1.0 Gamma .0144 .0010 .0673
. . . . .
0.4 Gamma .5259 .4715 .5783
0.5 Gamma .6181 .6064 .6607
0.6 Poisson .8783 .6736 .5934
0.7 Poisson .9286 .8231 .4050
0.8 Poisson .9853 .9812 .3191
0.9 Poisson .9514 .8582 .4205
1.0 Poisson .8818 .7021 .5134
rchapter5.dta
Limitations If Power Family Becomes Available
• Generally not a big problem when we are limited to the 4 named continuous families– Because, as in the example, within families we can
look for the power at which the statistics reach a maximum (ll) or maximum (AIC, BIC, deviance)
• When a more flexible GLM family is added to our statistical packages that allows a family of 0.7 or 1.3, rather than forcing us to round to a poisson family (1.0), the change in the scale of the AIC, etc., will make these statistics difficult, if not impossible, to use