methods for systematic reviews of economic evaluations for the
TRANSCRIPT
Methods for SystematicReviews of Economic Evaluationsfor the Guide to Community Preventive ServicesVilma G. Carande-Kulis, MS, PhD, Michael V. Maciosek, PhD, Peter A. Briss, MD, Steven M. Teutsch, MD, MPH,Stephanie Zaza, MD, MPH, Benedict I. Truman, MD, MPH, Mark L. Messonnier, MS, PhD,Marguerite Pappaioanou, DVM, PhD, Jeffrey R. Harris, MD, MPH, Jonathan Fielding MD, MPH, MBA,Task Force on Community Preventive Services
Objectives: This paper describes the methods used in the Guide to Community Preventive Services:Systematic Reviews and Evidence-Based Recommendations (the Guide) for conducting systematicreviews of economic evaluations across community health-promotion and disease-preven-tion interventions. The lack of standardized methods to improve the comparability ofresults from economic evaluations has hampered the use of data on costs and financialbenefits in evidence-based reviews of effectiveness. The methods and instruments devel-oped for the Guide provide an explicit and systematic approach for abstracting economicevaluation data and increase the usefulness of economic information for policy making inhealth care and public health.
Methods: The following steps were taken for systematic reviews of economic evaluations: (1) systematicsearches were conducted; (2) studies using economic analytic methods, such as costanalysis or cost-effectiveness, cost-benefit or cost-utility analysis, were selected according toexplicit inclusion criteria; (3) economic data were abstracted and adjusted using astandardized abstraction form; and (4) adjusted summary measures were listed in summarytables.
Results: These methods were used in a review of 10 interventions designed to improve vaccinationcoverage in children, adolescents and adults. Ten average costs and 14 cost-effectivenessratios were abstracted or calculated from data reported in 24 studies and expressed in 1997USD. The types of costs included in the analysis and intervention definitions variedextensively. Gaps in data were found for many interventions.
Medical Subject Headings (MeSH): cost-effectiveness, costs, economic evaluation, system-atic reviews (Am J Prev Med 2000;18(1S):75–91) © 2000 American Journal of PreventiveMedicine
Introduction
This paper describes the procedures and instru-ments used to collect, abstract, adjust, and sum-marize results from economic studies reporting
on cost, cost-effectiveness, cost-benefit or cost-utility of
selected interventions for the Guide to Community Pre-ventive Services: Systematic Reviews and Evidence-Based Rec-ommendations (the Guide). Methods were developed sothat studies using disparate analytical methods can beconsistently compared. Although these methods weredeveloped specifically for the Guide, they are sufficientlycomplete and general enough to be readily adapted toa range of systematic reviews of economic evaluations inhealth care and public health.
Evidence-based reviews of effectiveness (e.g., theGuide to Clinical Preventive Services1) usually have notreported data regarding costs and financial benefits ofpreventive interventions. Users of evidence-based effec-tiveness reviews often do not have enough informationto identify among effective interventions, those provid-ing the greatest amount of health or financial benefitper dollar of resource used. This might be because ofthe absence of economic evaluation studies or prob-
From the Division of Prevention Research and Analytic Methods,Epidemiology Program Office (Carande-Kulis, Briss, Zaza, Truman,Messonnier, Harris), Centers for Disease Control and Prevention(CDC), Atlanta, Georgia; Health Partners Research Foundation(Maciosek), Bloomington, Minnesota; Merck & Co., Inc. (Teutsch),West Point, Pennsylvania; Division of Global Health (Pappaioanou),CDC, Atlanta, Georgia; UCLA Schools of Public Health and Medi-cine, and County of Los Angeles Department of Health Services(Fielding), Los Angeles, California
The names and affiliations of the Task Force members are listed onpage v of this supplement and at http://www.thecommunityguide.org
Address correspondence and reprint requests to: Vilma G. Car-ande-Kulis MS, PhD, Economist, Community Preventive ServicesGuide Activity, Epidemiology Program Office, Centers for DiseaseControl and Prevention, 4770 Buford Highway MS-K73, Atlanta, GA30341: E-mail: [email protected].
75Am J Prev Med 2000;18(1S) 0749-3797/00/$–see front matter© 2000 American Journal of Preventive Medicine PII S0749-3797(99)00120-8
lems of comparability among available studies. Al-though little can be done on a short-term basis toincrease the quantity of available studies, systematicreviews of economic studies can address problems ofcomparability. In addition, they can improve the use-fulness of existing studies just as systematic reviewsbring together and interpret a body of evidence of theeffectiveness of interventions.
The lack of accepted, or even openly debated, meth-ods and instruments for systematic reviews of economicstudies is significant in light of both the need for suchmethods and the precedent set for such methods in theeffectiveness literature. Inconsistencies in the methodsemployed in published, peer-reviewed economic evalu-ation studies make results not comparable and lead toskepticism2 regarding the validity of the results of suchstudies. The lack of standardization in the design,analysis and reporting of economic evaluations is amatter of concern because changes in design andanalysis produce substantially different results.
Independent published reviews regarding cost-effec-tiveness of clinical and public health interventions havebeen conducted by several authors. Tengs et al.3 con-ducted a comprehensive review on the cost-effective-ness of 500 life-saving interventions in the UnitedStates, including motor vehicle safety devices, controlof environmental hazards, cancer screening, heart dis-ease screening and treatment, vaccination for infec-tious diseases and interventions to promote smokingcessation. Interventions ranged from those that arecost-saving to those that cost $10 billion per life-yearsaved, with a median cost of $42,000 per life-year saved(1993 USD). Ramsberg et al.4 conducted a reviewregarding the cost-effectiveness of 165 life-saving inter-ventions in Sweden, including screening and treatmentfor hypertension, road safety, smoking cessation andfire protection. Interventions ranged from those thatare cost-saving to ones costing $4.9 billion per life-yearsaved, with a median of $19,500 per life-year saved(1993 USD). Graham et al.5 reported on the cost-effectiveness of 40 interventions in the United States,including cancer, coronary heart disease and injuryprevention interventions. Cost-effectiveness ratiosranged from those costing less than $1,000 per quality-adjusted life-year (QALY) to ones costing $1 million(1995 USD) per QALY. Most of the cost-effectivenessratios reviewed were clustered in the range of $10,000to $100,000 per QALY.
In the process of providing the public health com-munity with lists of the cost-effectiveness of variousinterventions expressed in consistent units, the previ-ously mentioned reviews (and other reviews concentrat-ing on a more restricted set of health conditions6–8),have also provided the first steps toward the standard-ization of systematic reviews of published economicevaluations. In addition to published lists of cost-effec-tiveness ratios, these studies have reported general
inclusion and adjustment criteria. However, they usu-ally do not provide an explicit or full description of theprocedure used to adjust from the results of the pri-mary studies to the results they report, nor do theyprovide the instruments used in the abstraction andadjustment of data. In addition, they leave the compa-rability of the results from various studies in questionbecause they (1) make a limited number of adjustmentsto published ratios; and (2) report point estimates ofcost-effectiveness ratios without sensitivity analysis ofthe adjustments.
Therefore, to our knowledge, standardized instru-ments for systematically translating economic data intocomparable economic information have not beenwidely debated and are not available in the publicdomain.
The Guide is being developed by the Task Force onCommunity Preventive Services (the Task Force) incollaboration with many public and private partnersunder the auspices of the U.S. Department of Healthand Human Services.9 The Guide will include systematicreviews of a variety of issues, including effectiveness,applicability of effectiveness, harm, other positive andnegative effects, barriers to intervention implementa-tion, and economic evaluations of selected interven-tions for which evidence of effectiveness is strong orsufficient based on explicit criteria. Standardized meth-ods and instruments for searching, including and ab-stracting studies of effectiveness were developed for theGuide10 to reduce inconsistencies within and amongchapters. A similar process was undertaken to developmethods and instruments for the systematic review ofeconomic evaluations with the purpose of reducingerror and bias in the abstraction and adjustment ofresults and making them comparable acrossinterventions.
Methods
The methods for reviewing economic evaluationsinvolve (1) applying explicit criteria for decidingwhich evaluation studies were to be included in thereviews; (2) using a standard abstraction form torecord individual study characteristics, abstract data,and adjust reported results; and (3) interpreting andsummarizing economic information from relatedstudies of each effective intervention assessed.
Finding and Selecting Relevant Studies
For each chapter of the Guide, candidate economicstudies are identified by systematic searches of theliterature. A systematic search requires study inclusioncriteria. Systematic reviews of economic evaluations inthe Guide will, as much as possible, use consistentcriteria across chapters (Table 1). Each chapter alsorequires specific inclusion criteria relevant to the inter-
76 American Journal of Preventive Medicine, Volume 18, Number 1S
ventions reviewed in that chapter. For example, for theReviews of Evidence for Interventions to Improve Vac-cination Coverage in Children, Adolescents andAdults,12 a study was eligible if it met the standardcriteria and addressed selected interventions to im-prove coverage levels for universally recommendedvaccinations for children, adolescents, and adults.a
Accounting for MethodologicalDifferences in Primary Studies
A key decision in the design of the review process waswhether or not to devise a method for scoring thequality of the study as was done for the review ofeffectiveness literature in the Guide.10 A quality scoringwas not done for several reasons. First, although somedifferences in methods in economic studies couldreasonably be labeled as differences in quality, manydifferences in methods are attributable to differentobjectives or reasonable differences in opinion regard-ing the most appropriate methods. Second, even incases where differences in methods could be reasonablylabeled as quality related, the results of the study mightstill be informative in cases where the results indicatelarge cost-savings or very high cost per health benefit.Third, differences in methods affect the results of somestudies more than others. For example, the differencebetween the use of 5% and 10% discount rates in astudy with mostly short-term costs and benefits mightbe negligible, whereas the difference would be substan-tial in studies where most costs accrue many years in thefuture. Finally, the number of economic studies avail-able from which to select is extremely limited for most
community preventive services. Therefore, the TaskForce and chapter development teams decided to in-clude the studies that met the inclusion criteria and todevelop a method for adjusting the results of studiesrather than excluding some studies because of qualitylimitations. Each study is abstracted by two reviewers tofacilitate more thorough and balanced reviews. Differ-ences of opinion about the adjustments are resolved byconsensus.
Abstracting Economic EvaluationResults from Primary Studies
The second step in performing reviews is abstractingresults from included studies and adjusting them toimprove comparability across studies. The abstractionform provides an explicit instrument for performing thesetasks. The major sections of the form are summarized inTable 2; selected sections of the form are presented inAppendix A. The entire form including instructions canbe obtained from the corresponding author or from theGuide’s Internet home page at the following address:http://www.web.health.gov/communityguide.
Essential information including intervention defini-tion, methods and data sources of the primary study isrecorded in Sections I–IV of the abstraction form.Section I records information about the topic andintervention under consideration in the Guide and thetype of analytic method and summary measure re-ported in the study. Section II compares the interven-tion, as defined in the Guide, with the interventiondescribed in the study. Section III assists the reviewer inlocating the main elements of analysis in the article andSection IV allows descriptive study information such asperspective, comparator, analytic horizon, data sourcesand type of costs to be recorded. Sections V–VII dealwith adjustments to results of economic evaluation andare discussed below.
Adjusting Results of Primary Studies
Published results are adjusted to reflect what the resultsof the study would have been had the study followed achosen set of standards. Adjustments are made sequen-tially. The reference case of the Panel on Cost Effec-tiveness in Health and Medicine (PCEHM)13 is thestandard with which cost-effectiveness and cost-utilitystudies are compared. The PCEHM reference caseprovides an explicit and well-justified set of rules forconducting and reporting cost-effectiveness studies in amanner that allows the results of different studies to becompared with one another. Although perhaps not theoriginal intent of the PCEHM, the reference case alsoprovides a reasonable set of rules with which to stan-dardize the results of existing cost-effectiveness studies.For cost-analysis and cost-benefit studies, all standardspertaining to costs in the reference case also apply. The
a For example, measles, mumps, and rubella vaccinations are recom-mended for young children; hepatitis B vaccinations are recom-mended for adolescents; and annual influenza vaccinations arerecommended for adults aged $65.
Table 1. Standard inclusion criteria for economicevaluation studies for the Guide to Community PreventiveServices
Criteria Description
A Evaluated interventions that the Task Force hasalready classified as “strongly recommended”or “recommended”
B Used an analytical method (e.g., cost-analysis,cost-effectiveness, cost-benefit, or cost-utility)
C Provided sufficient detail regarding methodsand results to enable use and adjustment ofthe study’s data and results
D Was a primary study rather than, for example, aguideline or review
E Had a publication date appropriate to theparticular subject matter
F Was conducted in one or more EstablishedMarket Economies (EME) as defined by theWorld Bank11
G Was written in English
Am J Prev Med 2000;18(1S) 77
choice of the PCEHM reference case does not meanthat the Task Force necessarily believes the correspond-ing methods are better than alternative methods. Thischoice was made because the reference case provides aconvenient, thoroughly considered and generally ac-cepted standard for conducting and reporting cost-effectiveness studies. Of note is the fact that thePCEHM does not recommend these methods as thesole means of conducting cost-effectiveness studies.Rather, the PCEHM recommends the reporting of thereference case, when possible, to facilitate comparisonsamong studies.
Section V of the abstraction form enables the re-viewer to adjust the costs and health outcome measuresreported in the study. The first set of adjustments areintended to convert data expressed in various curren-cies and base years to USD in a consistent base year.First, costs expressed in a foreign currency are con-verted to USD for the base year reported in the studyusing purchasing power parity rates. Purchasing powerparity rates are estimates of the exchange rate at whichan equivalent amount of goods or services could bepurchased in two different countries, regardless of thecurrency being cited. Purchasing power parity rates areused for this purpose because they are less susceptibleto financial flows and governmental exchange ratemanipulation than are market exchange rates. Al-though purchasing power parity rates are less problem-atic than market exchange rates in converting the valueof resources used from one currency to another, con-versions can be inaccurate.14 Therefore, studies that arenot based on US prices and not reported in USD areconsidered only when similar USD-denominated stud-ies are not available.
Second, costs are adjusted to the 1997 base year usingeither the consumer price index (CPI) or the medicalconsumer price index (MCPI).15 Although the choice
of price index might not be particularly important forstudies that report results in a base year one or twoyears before 1997, it can be very important for studiespublished earlier. For example, for studies reported in1985 USD, the use of the CPI to convert to 1997 USDwould inflate the numerator of a cost-effectiveness ratioby 49%, whereas the use of the MCPI would inflate thenumerator by 106%. Therefore, we developed rules(Table 3) on when to apply the CPI or MCPI ratherthan relying on the use of one index in all cases.
Any rule for adjusting the value of the summarymeasureb is inherently inferior to re-estimation of anew value. For example, even if enough detail werepresented in a study to allow the adjustment of individ-ual prices by respective inflation rates, it is very unlikelythat the weighted average prices of the interventionwould increase at the estimated rate of inflation for thegeneral economy (as would be required to accuratelyupdate prices using the CPI) or that the weightedaverage of medical prices would increase at the esti-mated rate of medical inflation. Though imperfect, webelieve these rules are an improvement on standardiza-tion approaches using the same price index to adjust allcost-effectiveness ratios because they risk large errors inupdating studies.
After base year, costs are adjusted for discount rate.The rate used to discount costs and health outcomesaffects the value of the summary measure. The discountrate adjustment is based on the reference case of thePCEHM.13 The recommended discount rate is 3% forboth effectiveness and costs. The discount rate adjust-ment will be possible if the timing and amount offuture intervention costs, costs of illness and healthbenefits are known or can be approximated from data
b Summary measure refers to cost, net cost, cost-utility or cost-effectiveness ratio.
Table 2. Elements of the abstraction form used for reviewing economic evaluation studies selected for the Guide toCommunity Preventive Services
Section Purpose Completed by
I. Classificationinformation
Record tracking information, analytic method, and summarymeasure used in study
Guide staff and checkedby reviewers
II. Comparisonintervention
Record differences between intervention reviewed in the Guideand intervention analyzed in the study
Guide staff and checkedby reviewers
III. Identifyinginformation
Identify key result from study and key data elements reported instudy
Guide staff and checkedby reviewers
IV. Study information Record detailed study information, including location, audience,setting, perspective, data sources, etc.
Reviewers
V. Costs and healthoutcome measureadjustments
Adjust key result to 1997 USD, adjust for discount rate, add orremove costs, remove productivity losses, adjust health outcomemeasure, and assess uncertainty
Reviewers
VI. Nonadjustabledifferences
Summarize differences between study methods and standardmethods for which the study’s key result cannot be adjusted
Reviewers
VII. Applicability Summarize key parameters that might affect the interpretation ofthe results if intervention is applied in other settings
Reviewers
VIII. Summary table Condense findings of review and allow for revision of abstractedinformation after detailed review of study
Reviewers
78 American Journal of Preventive Medicine, Volume 18, Number 1S
provided in the study. If so, the full abstraction formprovides a table listing the effect of discounting oneunit (whether 1 USD or one QALY) over various timeperiods for several common discount rates (0%, 3%,5%, 7% and 10%).
The next set of adjustments ensures, to the extentpossible, that the costs and effects included in theanalysis are consistent from study to study. The addi-tional costs sections provide entry for those administra-tive, clinical preventive services, follow-up or patienttime and transportation costs missing from the analysis.The societal perspective requires that all costs andbenefits be considered, regardless of who bears thecosts or receives the benefits. In some instances, costestimates used in the study might not represent thesetting. For example, authors sometimes intentionallyuse high or low costs to obtain “conservative” estimatesof the summary measure. The PCEHM reference casedoes not include a valuation of time lost as a result ofchronic illness, long-term disability or death in thenumerator (“productivity”) to avoid double counting ofquality of life which is already accounted for in thedenominator. Many studies report results with and
without this valuation of time, therefore allowing pro-ductivity to be subtracted.
The conversion of the health outcome measure toQALYs addresses the need to improve comparability ofresults by using a common metric for health outcome.It is difficult, for example, to compare the cost-effec-tiveness of different studies in which results are ex-pressed as dollars per case of disease prevented, dollarsper life-year saved or dollars per QALY saved. Infollowing the PCEHM reference case whenever possi-ble, we use QALYs for this metric. QALYs capturemortality in terms of life-years lost and capture morbid-ity as a product of years in a reduced health state timesthe value of that reduction from full health. The healtheffect measure adjustment will not be done for thosecases where the study considered only intermediateoutcomes or other health outcome measures (e.g.,disability-adjusted life years [DALYs]). In the first case,an intermediate summary measure will be chosen de-pending on the chapter under consideration. In thesecond case, the summary measure will be expressed inUSD per DALY as originally reported in the article andreported in a separate table.
Table 3. Criteria for base-year adjustments for the Guide to Community Preventive Services
Rules Condition Index Justification
1 If both program costs and costs averteda areprimarily non-medical, or if both programcosts and costs averted are a mixture ofmedical and non-medical resources
CPI Reflects the fact that the CPI is already partiallyweighted by estimates of medical careinflation
2A If program costs are primarily non-medical,cost averted are primarily medical, and netcosts are negative (cost-savingintervention)
MCPI Although a mixture of costs exists, the negativenet costs indicate that the medical prices inthe costs averted are likely to play a largerrole in changes to the net costs over time
2B If program costs are primarily non-medical,cost averted are primarily medical, netcosts are positive, and the reviewer candetermine that the ratio of costs averted toprogram costs is ,0.25a
CPI General price inflation is likely to be a moreimportant factor in changes of the net costsover time
2C If program costs are primarily medical, netcosts are positive, and the reviewer candetermine that the ratio of costs averted toprogram costs is .0.75a
MCPIb Medical costs are playing a substantial role, andnet costs are likely to be decreasing over timerather than increasing because medicalinflation is historically greater than generalinflation
2D If program costs are primarily non-medical,costs averted are primarily medical, netcosts are positive, and (a) the ratio of costsaverted to costs incurred falls between 0.25and 0.75a or (b) the ratio cannot bedetermined with a reasonable amount ofcertainty
No adjustment Ratio is either relatively stable over time or thedirection in which the adjustment should bemade is too unpredictable
a “Program costs” (CP) include all positive costs due to the intervention and “costs averted” (CA) include all saved costs resulting from theintervention. When CA are considerably smaller than CP, the CA/CP ratio is relatively small (, 0.25), and the percent change in net costs (CP2 CA) over time due to inflation approaches the economy-wide inflation rate. When CA approaches the magnitude of CP, the CA/CP ratio isrelatively large (. 0.75) and the percent change in net costs over time due to inflation approaches negative infinity. The percent change in netcosts will be approximately equal to the medical inflation rate when the CA/CP ratio equals 0.75. Reviewers are cautioned that the actual percentchange in net cost may be significantly larger (in absolute value) than the medical inflation rate in cases where rule 2C applies and that inflationmay cause net costs to decrease to zero and become negative. When the CA/CP ratio equals 0.5, the percent change in net costs over time dueto inflation is approximately zero. Using either rule 2B or rule 2C (adjusting for inflation with either the CPI or MCPI) is not likely to improvethe estimate of net costs when CA/CP is between 0.25 and 0.75.b Ratio or net costs should be decreased using the MCPI rather than increased.
Am J Prev Med 2000;18(1S) 79
The results of studies addressing a service which pre-vent significant morbidity and express results in terms ofdollars per life-year saved must be adjusted to betterreflect the total health benefits of the service. In mostcases, these adjustments are made by scaling time spent inillness or disability to 70% of the value of time spenthealthy. With this approximation, each year of illness ordisability prevented by this service reduces 0.3 years ofquality of life for most chronic and acute diseases ordisabling conditions. The estimate of 0.3 is the differencebetween perfect health, valued at 1.0 on health utilityscales, and a range of approximately 0.55 to 0.85, (mid-point, 0.7) for persons living with most chronic and acutediseases or disabling conditions as reported in publishedutility scales16,17 (12 0.7 5 0.3).c
Although making adjustments with QALY weightsthat vary according to the specific condition would bepreferable, the simple rule of 0.3 QALYs lost annuallyfor most conditions was adopted because no publishedscales include all conditions and health states that arelikely to be needed in a cross-cutting review of preven-tive services. Regardless of the weights chosen, sensitiv-ity analysis on the weights used to adjust health benefitsis needed because no single means of defining weightsis clearly more correct that others. For many condi-tions, mortality tends to dominate the estimate of QALYand the adjustment will not be sensitive to the QALYweight for morbidity.
Table 4 contains an illustration of how the healthoutcome measure adjustment might be made. Thereviewer can use a higher weight if the 0.3 value isinsufficient to represent the averted impact of majormorbidities such as mental impairment, severe mentalillness or blindness. If an alternative weight is used, the
reviewer states the reason on the form. Adjustmentsshould be made whenever possible and in fact, approx-imations with wide ranges in sensitivity analysis arepreferable to determining that differences are nonad-justable. However, reviewers are asked to record differ-ences for which adjustments are not possible in SectionVI of the abstraction form, “Nonadjustable Differenc-es.” This section addresses differences in effect size,characteristics of the target population and frequencyand intensity of delivery of the intervention evaluatedin the study under review and the “typical” interventionclass evaluated in the Guide review.
Improving Comparability andConsistency of Base Adjustments
The adjustments noted previously are only as precise asthe underlying data and the reporting of the originalstudy allow. However, three additional steps are takento improve the quality of reviews. First, the adjustmentsare subjected to sensitivity analysis. Under the section“Predicted Degree of Error (Uncertainty of Adjust-ment),” reviewers are asked to assess the reliability ofindividual adjustments using alternate values for eachof the adjustments. This assessment is virtually identicalto a single-variable sensitivity analysis on the finaladjusted value of the summary measure. Based uponthis sensitivity analysis, reviewers are asked to determineto which of four predefined categories of uncertaintyeach adjustment belongs. The ranges of uncertaintyassociated with the categories (60%–10%, 11%–30%,31%–50% and .50%) are defined as a percentage ofthe final adjusted ratio, rather than as a percent of theratio resulting from each individual adjustment, so thatthe sensitivity of each adjustment is evaluated on acommon basis and the range of uncertainty has under-lying meaning. No theoretical or empirical basis is
c Based on the fact that when mortality is measured, years of life arevalued at 1, not at 0.9 (the rough average reported for those with nodiseases or disabilities).
Table 4. Example of a cost-effectiveness ratio conversion from dollars per year of life saveda to dollars per quality adjustedlife year (QALY) for an immunization program to immunize infants against Hepatitis B
Item Parameter Source Valuea
A Cost of program for infants, 1993 USD, millions Table 4, chronic, medical costs18 47.0B Medical cost without program, 1993 USD, millions Table 4, chronic, medical costs18 9.5C Medical cost without program, 1993 USD, millions Table 4, chronic, medical costs18 3.1D Medical cost averted, 1993 USD, millions (B 2 C) 6.4E Net cost, 1993 USD, millions (A 2 D) 40.6F Net cost, discounted, USD, millions (E) Discounted 50 years at 5% 3.7G Years of life saved, number Table 518 18,879H Years of life saved, discounted, number (G) Discounted 50 Years at 5% 1,699I Chronic infections prevented, number Table 518 4,702J Quality adjusted life years from morbidity, number (I) 3 10 years of chronic infection 3 0.3 QALYs 14,106K Quality adjusted life years from morbidity,
discounted, number(J) Discounted 50 years at 5% 1,270
L Quality adjusted life years from morbidity andmortality, number
(H 1 K) 2,969
M Cost-effectiveness ratio in 1993 dollars per QALYs (F/L) 1,246aRounded to better reflect precision of adjustments.
80 American Journal of Preventive Medicine, Volume 18, Number 1S
currently available for defining these ranges; therefore,the ranges were chosen to identify interpretable de-grees of uncertainty.
Second, in Section VI, “Nonadjustable Differences,”reviewers describe differences in methods and defini-tions of interventions where adjustments are not possi-ble because of lack of information. This section allowsthe reviewer to indicate whether the final summarymeasure from Section V might under- or overstate thevalue that would have been observed had the economicmethods in the study not differed from the referencecase or had the specific intervention evaluated notdiffered from a more typical case for the interventionsunder review.
Third, on the basis of their experience in reading thearticle and making adjustments, reviewers identify vari-ables to which the results are particularly sensitive andwhich are likely to change if the intervention is appliedin predictable, but specialized situations. Section VII,“Applicability,” is designed to record information thatcould be used to alert users of the Guide to predictableand substantial changes in the value of the summarymeasure. Variables in this section needing particularattention from reviewers include baseline prevalence(if applicable), population characteristics, effective-ness, size of the population and costs in the case of amore intensive application of an intervention.
Section VIII, “Summary Table,” condenses the find-ings of the study review and adjustments. It is com-pleted only after the full instrument has been com-pleted and allows for revision of abstraction andadjustments. This summary table lists the main compo-nents of the study and the sequential adjustments madeto the original ratio, costs, or cost-saving value. Thistable also facilitates the process of reconciling differ-ences and reaching consensus in the abstraction andadjustments by the two reviewers.
Summarizing Results: Example for theChapter on Vaccine-Preventable Diseases
An excerpt of the economic summary table for one ofthe interventions included in the Reviews of Evidencefor Interventions to Improve Vaccination Coverage inChildren, Adolescents and Adults12 is provided in Ap-pendix B. Additional examples are shown in thatreport. Cost-effectiveness was expressed in this particu-lar case as the cost of the program per additionalvaccination above baseline coverage or the cost perfully vaccinated child. Baseline vaccine coverage is thecoverage that would have occurred in the absence ofthe intervention. Average cost was defined as the cost ofthe program per person served by the program. Aver-age cost is provided to give program planners a startingpoint for estimating the costs of new initiatives. Thecost-effectiveness of interventions to increase vaccina-
tion rates is not summarized in terms of health out-comes (e.g., QALYs) because (1) effectiveness studiesreviewed in the Guide considered only intermediateoutcomes; and (2) the health benefits vary by vaccine.
The table in Appendix B summarizes informationregarding the study and study results including type ofanalytic methods used in the study or used by thereviewer to obtain the summary measure from datareported in the study. It also lists the type of summarymeasure, original currency, costs included in the anal-ysis, the results of the study before and after adjust-ments, characteristics of the study population andestimates of effectiveness used in the evaluation.
Results from the table are interpreted and summa-rized in the body of the Guide for each particularintervention under the “Economic Evidence” subhead-ing. The economic information is available to usersalthough they do not affect the recommendations. Oneof the summaries from the report on vaccine-prevent-able diseases follows:
The search identified 11 economic evaluations ofclient reminder/recall interventions. Nine studiesprovided 12 cost-effectiveness ratios for single-component reminder/recall interventions and3 cost-effectiveness ratios for multicomponent in-terventions that include reminder/recall. Ad-justed cost-effectiveness ratios for the single-com-ponent interventions based on those studiesranged from $3 to $46/additional vaccinationwith a median of $9/additional vaccination. Theadjusted cost-effectiveness ratios for multicompo-nent interventions were $4/additional vaccina-tion for a combination of client and providerreminders; $51/additional vaccination for a com-bination of reminders and a lottery-type incentive;and $43/additional vaccination for a combinationof mailed reminders and free vaccinations. Ad-justed average costs based on two available studiesvaried from $0.65 to $5.75/child. The lowerboundary is an underestimate because the costs ofthe in-kind contribution of volunteer time werenot included and the upper boundary might bean overestimate because it includes costs of clini-cal time to provide vaccinations.
Discussion
At present, the body of economic evidence available tocompare costs and returns of interventions to improvehealth is substantially limited in both quantity andcomparability. Methods to review, abstract and summa-rize economic evaluations need to be developed anddebated if economic information is going to be usefulto managers, policy makers, program planners andresearchers.
The development of the procedures and instrumentsfor economic evaluation in the Guide required balanc-
Am J Prev Med 2000;18(1S) 81
ing several competing needs. The abstraction formneeds to be flexible enough to allow for review ofstudies using various analytic methods (e.g., cost, cost-effectiveness, cost-benefit and cost-utility) and interven-tions. However, the abstraction process must be consis-tent to ensure comparability of data acrossinterventions, control uncertainty and limit the num-ber of subjective judgements which reviewers need tomake.
The Task Force recognizes that no process of adjust-ment or other means of reviewing existing economicevaluations is flawless. However, data that is made morecomparable is thought to be preferable to alternativesthat would (1) ignore economic information entirely;(2) attempt to use noncomparable data; or (3) adjust inways that are not systematic or explicit. These methodswill continue to evolve with time, experience and inputfrom interested readers. Further application and test-ing will determine whether the process performs well ina variety of studies and disease topic areas and whetherthe instrument could be streamlined.
The systematic review of the economics of interven-tions in health care and public health calls for standard-ized and explicit methodology. The efforts of thePCEHM were an important step toward achieving thisgoal. The PCEHM recommends that, in addition toother results, study authors report results on the basisof the reference case whenever possible. However,methods used to compare the large number of previ-ously published studies, which employ disparate meth-odologies, are still needed. Although the comparabilityof published studies may have improved since thepublication of the PCEHM recommendations, methodsfor comparing published studies will continue to beneeded as some study authors face data constraints,funding constraints, editorial constraints and studyobjectives that are partially at odds with the reporting ofreference case results.
The provisional methods presented here representsan initial effort to meet these current and future needsin the Guide to Community Preventive Services. This paperdescribes the methods used to review economic evalu-ations in the Guide with the hope that others (1) willprovide recommendations for improving the methodsfor the Guide; (2) will adapt these methods to the needsof other reviews of economic evaluations; (3) willengage in a process of developing systematic methodsfor evaluating economic evaluations literature thatequals efforts to develop methods for the systematicreview of effectiveness literature; and (4) will encour-
age economic evaluation studies of interventions forwhich economic information is not available.
The authors are particularly grateful to Marthe R. Gold MD,MPH, C. Kay Smith-Akin, MEd, Robert Deuson, PhD, ReginaPana-Cryan, PhD, and Scott Grosse PhD for their contribu-tions and advice in the development of the economic evalu-ation abstraction form.
References1. US Preventive Services Task Force. Guide to clinical preventive services.
2nd ed. Baltimore: Williams & Wilkins, 1996.2. Kassirer JP, Angell M. The Journal’s policy on cost-effectiveness analyses.
N Engl J Med 1994;331:669–70.3. Tengs TO, Adams ME, Pliskin JS, et al. Five hundred life-saving interven-
tions and their cost-effectiveness. Risk Anal 1995;15(3):369–90.4. Ramsberg JAL, Sjoberg L. The cost-effectiveness of life-saving interventions
in Sweden. Risk Anal 1997;17:467–78.5. Graham JD, Corso PS, Morris JM, Segui-Gomez M, Weinstein MC. Evalu-
ating the cost-effectiveness of clinical and public health measures. Ann RevPub Health 1998;18:125–53.
6. Schrappe M, Lauterbach K. Systematic review on the cost-effectiveness ofpublic health interventions for HIV prevention in industrialized countries.AIDS 1998;12(suppl):231–7.
7. van der Weijden T, Knottnerus JA, Ament AJ, Stoffers HE, Grol RP.Economic evaluation of cholesterol-related interventions in general prac-tice: an appraisal of the evidence. J Epidemiol Community Health 1998;52:586–94.
8. Brown ML, Fintor L. Cost-effectiveness of breast cancer screening: prelim-inary results of a systematic review of the literature. Breast Cancer Res Treat1993;25:113–8.
9. Pappaioanou M, Evans C. Development of the Guide to CommunityPreventive Services: a US Public Health Service initiative. J Public HealthMgmt Practice 1998;4:48–54.
10. Zaza S, Wright-De Aguero LK, Briss PA, et al. and the Task Force onCommunity Preventive Services. Data collection instrument and procedurefor systematic reviews in the Guide to Community Preventive Services. Am JPrev Med 2000;18(suppl 1):44–74.
11. Murray CJL, Lopez AD. Global health statistics. Boston: Harvard UniversityPress, 1996.
12. Briss PA, Rodewald L, Hinman A, et al. and the Task Force on CommunityPreventive Services. Reviews of evidence for interventions to improvevaccination coverage in children, adolescents and adults. Am J Prev Med2000;18(suppl 1):97–140.
13. Gold M, Siegel JE, Russell LB, Weinstein MC, eds. Cost-effectiveness inhealth and medicine. New York: Oxford University Press, 1996: AppendixA304–11.
14. Kanavos P, Mossialos E. International comparisons of health care expen-ditures: what we do know and what we do not know. J Health Services ResPolicy 1999;4:122–6.
15. Statistical Abstract of the United States. Washington, DC: US Bureau of theCensus, 1997:492–510.
16. Fryback DG, Dasbach EJ, Klein R, Klein BE, Dorn N, et al. The Beaver Damhealth outcomes study: initial catalog of health-state quality factors. MedDecis Making 1993;13:89–102.
17. Gold MR, Franks P, McCoy KI, Fryback DG. Toward consistency incost-utility analyses: using national measures to create condition-specificvalues. Med Care 1998;36:778–92.
18. Margolis HS, Coleman PJ, Brown Re, Mast EE, Sheingold SH, Arevalo JA.Prevention of hepatitis B virus transmission by immunization: an economicanalysis of current recommendations. JAMA 1995;274:778–92.
82 American Journal of Preventive Medicine, Volume 18, Number 1S
Appe
ndix
BSa
mpl
eEc
onom
icEv
alua
tion
Sum
mar
yTa
ble
88 American Journal of Preventive Medicine, Volume 18, Number 1S
Refe
renc
esto
Appe
ndix
B1.
Buc
hn
erD
M,
Lar
son
EB
,W
hit
eR
F.In
flue
nza
vacc
inat
ion
inco
mm
unit
yel
derl
y:a
con
trol
led
tria
lof
post
card
rem
inde
rs.
JA
mG
eria
trSo
c19
87;3
5:75
5–60
.2.
Ch
iuT
T,
Bar
ata
SL,
Un
sick
erD
M,
Bre
nn
anL
.C
omm
unit
ym
obili
zati
onfo
rpr
esch
ool
imm
uniz
atio
ns:
the
“Sh
ots
byT
wo”
proj
ect.
Am
JPu
blic
Hea
lth
1997
;87:
462–
3.3.
Fram
eP,
Zim
mer
J,W
erth
P,H
all
J,E
berl
yS.
Com
pute
rize
d-ba
sed
vs.
man
ual
hea
lth
mai
nte
nan
cetr
acki
ng.
Arc
hFa
mily
Med
1994
;3:5
81–
8.4.
Fran
kJW
,M
cMur
ray
L,
Hen
ders
onM
.In
flue
nza
vacc
inat
ion
inth
eel
derl
y:th
eec
onom
ics
ofse
ndi
ng
rem
inde
rle
tter
s.C
anM
edA
ssoc
J19
92;1
32:5
16–
8.5.
Hut
chis
onB
G,
Shan
non
HS.
Eff
ect
ofre
peat
edan
nua
lre
min
der
lett
ers
onin
flue
nza
imm
uniz
atio
nam
ong
elde
rly
pati
ents
.J
Fam
ilyPr
acti
ce19
91;3
3:18
7–9.
6.G
rabe
nst
ein
JD,
Har
tzem
aA
G,
Gue
ssH
A,
Joh
nst
onW
P,R
itte
nh
ouse
BE
.C
omm
unit
yph
arm
acis
tsas
imm
uniz
atio
nad
voca
tes.
Cos
t-eff
ecti
ven
ess
ofa
cue
toin
flue
nza
vacc
inat
ion
.M
edC
are
1992
;30:
503–
13.
7.L
ieu
TA
,B
lack
SB,
Ray
P,et
al.
Com
pute
r-ge
ner
ated
reca
llle
tter
sfo
run
deri
mm
uniz
edch
ildre
n:
how
cost
-eff
ecti
ve?
Pedi
atr
Infe
cD
isJ
1997
;16:
28–3
3.8.
Lie
uT
A,C
apra
AM
,Mak
olJ,
Bla
ckSB
,Sh
inefi
eld
HR
.Eff
ecti
ven
ess
and
cost
-eff
ecti
ven
ess
ofle
tter
s,au
tom
ated
tele
phon
em
essa
ges,
orbo
thfo
run
deri
mm
uniz
edch
ildre
nin
ah
ealt
hm
ain
ten
ance
orga
niz
atio
n.P
edia
tric
s19
98;1
01:E
3.9.
McL
eod
D,
Bow
ieR
D,
Klja
kovi
cM
.C
ost
ofch
ildh
ood
imm
uniz
atio
nin
gen
eral
prac
tice
.N
ZM
edJ
1998
;111
:73–
6.10
.M
oran
WP,
Nel
son
K,
Wof
ford
JL,
Vel
ezR
,C
ase
LD
.In
crea
sin
gin
flue
nza
imm
uniz
atio
nam
ong
hig
h-r
isk
pati
ents
:ed
ucat
ion
orfi
nan
cial
ince
nti
ve.
Am
JM
ed19
96;1
01:6
12–
62.
11.
Nex
oeJ,
Kra
gstr
upJ,
Ron
ne
T.
Impa
ctof
post
alin
vita
tion
san
dus
erfe
eon
infl
uen
zava
ccin
atio
nra
tes
amon
gth
eel
derl
y.A
ran
dom
ized
con
trol
led
tria
lin
gen
eral
prac
tice
.Sc
and
JPr
imar
yH
ealt
hC
are
1997
;15:
109
–12.
12.
Ros
ser
WW
,H
utch
ison
BG
,M
cDow
ell
I,N
ewel
lC
.U
seof
rem
inde
rsto
incr
ease
com
plia
nce
wit
hte
tan
usbo
oste
rva
ccin
atio
n.
Can
Med
Ass
ocJ
1992
;146
:911
–17.
13.
McD
owel
lI,
New
ell
C,
Ros
ser
W.
Com
pari
son
ofth
ree
met
hod
sof
reca
llin
gpa
tien
tsfo
rin
flue
nza
vacc
inat
ion
.C
anM
edA
ssoc
J19
86;1
35:9
91–7
.
Am J Prev Med 2000;18(1S) 91
Reprinted by permission of Elsevier Science from: Methods for Systematic Reviews of Economic Evaluations for the Guide to Community Preventive Services. Vilma Carande-Kulis, Michael V. Maciosek, Peter A. Briss et al., American Journal of Preventive Medicine, Vol 18 No 1S, pp 75-91, Copyright 2000 by American Journal of Preventive Medicine