Volume 2 • Number 2 • 1999VALUE IN HEALTH

© ISPOR 1098-3015/99/$14.00/73 73–77

73

Panel 1: Methodological Issues in Pharmacoeconomic Evaluations—Clinical Studies

Co-chairs: Margaret Healey, PhD,

1

Patricia Deverka, MD, MS

2

Panelists: Steven Fox, MD,

3

Kathleen Gondek, PhD,

4

Barbara Edelman Lewis, PhD,

5

Eva Lydick, PhD,

6

Gurkipal Singh, MD,

7

Robert Temple, MD,

8

Jeff Trotter, MBA

9

1

The Institute for Research & Education, HealthSystems Minnesota, Minneapolis, MN,

2

Merck Medco Managed Care, LLC, Montvale, NJ,

3

Center for Outcomes & Effectiveness Research, Agency for Health Care Policy and Research, Rockville, MD,

4

Boehringer Ingelheim Pharmaceuticals Inc., Ridgefield, CT,

5

Focus Managed Research Inc., Worcester, MA,

6

SmithKline Beecham, Collegeville, PA,

7

Stanford

University Medical Center, Palo Alto, CA,

8

Food & Drug Administration, Rockville, MD,

9

ClinTrials Ovation Inc., Highland Park, IL

T

he goal of this panel was to identify key con-tentious methodology issues in conducting

healthcare pharmacoeconomic evaluations in thecontext of clinical studies. Its specific objectiveswere to:

• identify and prioritize the key issues associatedwith including pharmacoeconomic and out-comes research projects in clinical studies;

• identify a plan of action to resolve these issues;• recommend next steps.

Background and Context

Consumers of pharmacoeconomic information in-clude patients, clinicians, and, in particular, health-care decision- and policy-makers in managed careand government. Evaluations of the costs and out-comes of various treatment options are in increas-ing demand to inform formulary and treatmentguideline decision-making. In order to assess thevalue, and not just the price, of new interventions,consumers need data that are both valid and gen-eralizable. Pharmacoeconomic data used to pro-vide such evidence of the value of therapeutic in-terventions can be generated from a variety ofstudy methodologies.

The strengths and limitations of various clinicalstudy designs for economic evaluation, particu-larly randomized controlled trials (RCTs), havebeen described elsewhere [1–4]. Briefly, the use ofRTCs to develop pharmacoeconomic data has sev-eral advantages: these data are based on a well-established and accepted methodology, are readilyinterpretable, have statistical rigor, and can be de-signed free of most types of bias. The process ofrandomization is particularly critical in decreasingthe probability of selection bias. Especially when

studying small effects, other methodologies (e.g.,observational studies) are less reliable.

The limitations of RCTs have also been recog-nized. These arise primarily from the costs associ-ated with carrying out trials with multiple com-parators, trials of long duration, and trials largeenough to detect infrequent outcomes. In addi-tion, the prospective nature of RCTs means thatthere is a substantial delay between posing a ques-tion and answering it. Furthermore, it is difficultto separate protocol-driven costs from actual costsof care. In many cases, the choice of clinical trialoutcome measures may not be relevant to stan-dard or usual care. Concern has also been ex-pressed that when therapies with proven efficacyare translated to “usual care” settings, the resultsmay fall short of those predicted by controlledstudies. The outcomes achieved when new inter-ventions are implemented in actual practice set-tings are often referred to as evidence of the effec-tiveness (as distinguished from the efficacy) of thattreatment.

These limitations on the usefulness and the gen-eralizability of the results from controlled clinicaltrials have led to the increased use of alternativedesigns, including observational studies, to ad-dress these problems. The advantages of using ob-servational studies to collect pharmacoeconomicdata include the ability to follow relatively largenumbers of patients for extended periods of time.Whereas the outcomes in RCTs sometimes repre-sent intermediate outcomes, prolonged follow-upin observational studies allows for capturing long-term outcomes, including mortality.

Observational designs are often used whenstudying the effectiveness of a treatment. Whereaschoice of comparator is often precluded, compara-tors are generally those used most frequently in

74

Healey and Deverka

practice settings, enhancing the generalizability ofstudy results.

The most serious limitations of observationalstudies are methodological. Arguably the greatestweakness of the observational study is the inabil-ity to control for selection bias. In addition, strate-gies to control for confounding variables are lim-ited.

Users of pharmacoeconomic data need infor-mation on both the efficacy and effectiveness ofnew treatments and how these compare for alter-native treatments. Although few would argue thatthe RCT is the “gold standard” when evaluatingthe efficacy of an intervention, it may be possibleto maximize the usefulness of other methods bycontrolling for as many confounding variables aspossible. Our understanding of the relative valueof interventions could be advanced by the adop-tion of study designs chosen to balance scientificexigencies with practical application.

Problem Statement

The study methods that are acceptable for demon-strating drug efficacy are largely agreed upon andare, in most cases, randomized controlled trials(RCTs). RCTs have been criticized, however, fornot adequately addressing value in applied set-tings. In addition, limiting pharmacoeconomicdata to those derived from RCTs would reducethe body of timely information. Alternative meth-ods, such as observational studies, utilized byhealth economists and epidemiologists for assess-ing questions of value are often compromised bymethodological inadequacies, particularly selec-tion bias. Developing a better understanding ofwhen different methods are appropriate will im-prove the usefulness and validity of pharmacoeco-nomic information.

Issues

This paper addresses four primary issues:

1. Under what circumstances should RTCs be theprimary approach for assessing questions ofvalue?

2. How can RCTs be modified to improve theirusefulness to inform economic decision-making?

3. Under what circumstances can observationalstudies be used to assess questions of value?

4. How can observational studies be modified toimprove their usefulness to inform economicdecision-making?

Under What Circumstances Should RCTs Be the Primary Approach for Assessing Questions of Value?

Pharmacoeconomic data can be derived from ran-domized trials, but randomized controlled trialsare not always the first choice for a variety of rea-sons, both practical and scientific. Randomizedcontrolled trails should be considered the primaryapproach, however, under the following circum-stances:

• The significance of the results justifies the ex-pense. Not every question merits the level ofassurance that a result is replicable and freefrom most types of bias that an RCT can pro-vide. The “cost-effectiveness” of the studyneeds to be evaluated. For example, the Fed-eral Trade Commission Policy Statement Re-garding Advertising Substantiation [5] specifi-cally mentions that the cost of developing theevidence for substantiation of a cost-effective-ness conclusion is one of the key issues in de-ciding the adequacy of the scientific basis forsupport. The FTC’s extensive experience inregulating the promotion of nonprescriptiondrug claims provides an important perspectivewhen considering whether to invest in a clini-cal trial to evaluate the economic impact of aparticular drug.

• There is a reasonable likelihood of observingpatient outcomes within a relatively short time,and/or intermediate endpoints (e.g., success incontrolling blood pressure) are acceptable.When RCTs are used to study long-term out-comes, they can be very costly. Care must beexercised, however, in extrapolating to long-term outcomes that have not been causallylinked to intermediate endpoints. When inter-mediate outcome variables are used, the datasupporting the level of certainty in the link be-tween the intermediate variable and final out-comes need to be well established and dis-closed.

• There is consensus on the appropriate compar-ator(s). If there is agreement (within a definedpractice setting) that one intervention is thestandard treatment approach, a single largestudy (RCT) with a limited number of treat-ment arms presents a practical approach.

• Effect sizes are predicted to be small, but stillimportant from a clinical or public health per-spective. Under these circumstances, observa-tional studies are of limited usefulness becausethe magnitude of the observed effect of thedrug may be similar to the amount of uncon-

Panel 1

75

trolled confounding. A randomized controlledtrial will provide the strongest evidence thatthe observed differences between treatmentsare not due to confounding and that the asso-ciation between treatment and outcome is oneof cause and effect.

• Randomization is acceptable to (physician) in-vestigators and to patients.

How Can RCTs Be Modified to Improve Their Usefulness to Inform Economic Decision-Making?

Most of the concern expressed about RCTs, par-ticularly from decision-makers, is not attributableto methodological flaws, but to questions aboutthe generalizability of the results to practice set-tings. To mitigate these concerns:

• Conduct RCTs in settings where decisions aremade, for example, in managed care organiza-tions.

• Adopt more naturalistic study designs.• Liberalize inclusion criteria and minimize ex-

clusion criteria:• Use active comparators where reasonable.

Placebos are generally perceived by deci-sion-makers as not pertinent. When ex-pected measures of effectiveness are less ro-bust (e.g., in studies of depression), aplacebo comparator may be needed forcredible assessments of effectiveness.

• Use relevant comparators. Select compara-tors based on community accepted stan-dard treatments whenever feasible. If theoptimal dose of the comparator agent isunknown, it may be necessary to test multi-ple doses.

• Mimic usual care as closely as possible. Theeffectiveness of new interventions is criticalto providers, as well as to decision-makers.Protocol required assessments and follow-ups should be structured to align as closelyas possible with acceptable community stan-dards. Whenever possible, the frequency ofthose assessments should not be dictated bythe protocol.

• Reduce the burden of adverse event report-ing, if possible, without compromising pa-tient safety. For a marketed drug, or even adrug in late phase III testing, it is not neces-sary to collect data on all adverse drugevents (ADEs). It would be possible, for ex-ample, to collect data only on ADEs thatlead to changes in drug dose, medication

switching, an intervention, and/or a prema-ture termination of a patient from a study.

• Use nested designs. For example, answer sec-ondary questions in subpopulations if thepower is adequate.

• Report variances around economic measures.Variance around economic measures has gen-erally been less adequately described than thataround clinical outcomes, limiting the useful-ness of this data for decision-makers.

Under What Circumstances Can Observational Studies Be Used to Assess Questions of Value?

Discussion here is limited to observational studiesthat use primary data sources. The use of adminis-trative databases in economic evaluations is ad-dressed by Panel 3. Primary data in this contextrefers to data that are derived from prospectivelydefined study questions that specify the particularendpoints of interest. The data may then comefrom a number of sources, including the medicalrecord or a relevant database.

Observational studies should be considered asa primary approach when the following criteriaare met:

• The effect size is expected to be large.• The cost-effectiveness of an RCT is question-

able. (Many of the following circumstances con-tribute to the unacceptable cost of an RCT.)

• The primary outcome is a rare event. Observa-tional studies may be a more appropriate ap-proach for rare outcomes requiring very largesample sizes. For example, if the true rate of anevent is 0.001 and you want to detect a two-fold increase in risk, it would be necessary tofollow 40,000 individuals [6]. Randomizedcontrolled trials of this size are prohibitivelyexpensive for all but the most important ques-tions.

• There are multiple appropriate comparators. Var-iation in clinical practice is a well-described phe-nomenon and there are often multiple drugsthat can be used to treat a particular condition.Because relevant comparators typically differacross practice settings, it is unlikely that allpossible drugs will be evaluated in a random-ized controlled trial.

• Observation of the primary outcome requiresextended follow-up. This is particularly truefor studies in chronic diseases. Long-term com-plications of diabetes and rheumatoid arthritis,for example, often manifest 10–20 years after

76

Healey and Deverka

onset of disease. It is neither easy nor practicalto conduct randomized clinical trials for suchlong durations. In addition, if there is a consid-erable loss-to-follow-up over time (as is likely),then results from an RCT will suffer from va-lidity as well as prohibitive cost. The only fea-sible solution under such circumstances is lon-gitudinal observational studies.

• The intervention to be studied has been avail-able for a long time and data are availablefrom multiple sources including government,third-party payers, and managed care organi-zations and vendors.

• People are likely to refuse randomization (forexample, comparing invasive procedures tononinvasive treatments).

• The primary goal of the study is to assess prac-tice patterns, as is typically the case in cost ofillness studies.

How Can Observational Studies Be Modified to Improve Their Usefulness to Inform Economic Decision-Making?

Although the use of good clinical research prac-tices obviously applies to the conduct of bothRCTs and observational studies, observationalstudies are more frequently criticized for method-ological inadequacies. To address this concern, theInternational Society for Pharmacoepidemiology(ISPE) has developed guidelines for good epidemi-ology practices for drug, device, and vaccine re-search in the United States [7]. While acknowledg-ing the limitations of nonexperimental data, theISPE authors conclude that properly conductedepidemiologic studies provide valuable informa-tion about the relationship between drugs and hu-man health. The good epidemiology practices(GEPs) provide guidance on those issues that areunder the control of the investigator, specificallydata quality, study design, and study conduct. TheGEPs do not dictate specific research methods, butrather propose minimum practices and proceduresto help ensure the quality of the data used in ob-servational studies and to provide adequate docu-mentation of the study design and methods.

We recommend a similar approach be adoptedfor the use of observational data in the conduct ofeconomic evaluations. Emphasis should be placedon the importance of specifying a priori:

• the research question(s) (which has been hy-pothesis driven);

• the study design;• the data collection methods;

• the data analysis plan;• dissemination plans.

Additional recommendations include replica-tion of studies in more than one environment, andadjustment for factors believed to be associatedwith assignment of treatment. Analytic rigor forobservational studies involves adjusting for differ-ences in baseline characteristics across treatmentgroups. The measures of association between in-tervention and outcome should take into accountthose factors that are likely to influence treatmentselection or outcome.

Recommendations and Next Steps

The following areas would benefit from furthermethodological development:

• Develop new methods to account for protocol-related costs (particularly in studies where thesecosts cannot be equated across study groups).

• Develop consensus on the definition of usualcare. With the recommendation that trialsshould adopt more naturalistic designs, it isanticipated that the use of “usual care” as acomparator will continue to increase.

• Explore methods to supplement intent-to-treatanalyses in usual care trials. Not to minimizethe importance of the intent-to-treat analysis,but there may be research questions importantto decision-makers that may require an alter-nate or additional analysis. For example, be-cause economic trials are typically open-label,there can be a tendency for differential rates ofcrossover between treatment arms. Often pa-tients randomized to usual care have higherrates of crossover to the new drug treatmentarm, resulting in patients receiving the sametreatment (the new drug) in both arms. In thissituation useful information can still be de-rived from subgroup analyses of completers.

• Address problems of pooling economic datafrom multiple study sites. This is particularlyproblematic in studies when significant differ-ences in practice styles exist across study sites(e.g., urban and rural sites, Veterans’ Adminis-tration and managed care settings). This prob-lem is exacerbated in international studieswhere data are collected across countries.

• Improve statistical methods for adjusting forselection bias, which is a major drawback ofobservational studies. There are several meth-ods to reduce this bias, from commonly usedtechniques such as adjustment by analysis of

Panel 1

77

covariance or other multivariate techniques, tomore sophisticated techniques such as instru-mental variables, two-stage regressions, andpropensity scores. More research is needed todetermine when a particular technique is ap-propriate for pharmacoeconomic analysis.

• Use better methods for estimating variancearound resource utilization and cost. Tradi-tional approaches that assume independence ofobservations and a normal Gaussian distribu-tion are unsuitable for calculating variancearound economic parameters. Techniques suchas boot strapping and jack-knifing can be usedin these scenarios. These resampling methodsare also useful in estimating variances aroundcost-effectiveness ratios.

• Conduct systematic comparisons of RCTs andobservational studies of the same interven-tions. It would be useful whenever an RCT isused to answer a pharmacoeconomic questionto also carry out observational studies of thesame issues to look at correlation, to explaindifferences, etc.

• Explore approaches from other disciplines (e.g.,psychology, sociology, marketing research) toenhance current methods, particularly in theareas of data collection, instrumentation, andanalytic techniques.

• Measure resource utilization in large simpletrials. Resources currently dedicated to mar-keting studies could be reallocated for theidentification and collection of relevant eco-nomic data.

• Create better methods of measuring directmedical costs that are not routinely captured(e.g., nursing time, telephone care).

• Create better methods of measuring relevantindirect costs (e.g., caregiver, lost productiv-ity). Pharmacoeconomic analyses are oftenconducted from a payer’s viewpoint, ignoringindirect costs. Yet time lost from work, lostproductivity, caregiver expenses, etc., are im-portant to the patient and to society. Thereneeds to be a greater appreciation of these costdrivers, and simple, easy methods to measureand document them.

• Encourage inclusion of standardized outcomemeasures in the evolving electronic medicalrecord.

Summary

Useful pharmacoeconomic data can be derivedfrom RCTs and observational studies. There is,however, no single pharmacoeconomic study thatwill establish the value of an intervention. Differ-ent audiences bring a variety of perspectives, aswell as definitions of value, to the generation,framing, and interpretation of research questions.Establishing value can result from the synthesis ofdata from a variety of sources, including RCTs,observational studies, and modeling, with appro-priate attention to the weaknesses and limitationsof each kind of data.

The most important factors for a researcher toconsider in designing a clinical trial to answerquestions of value are knowing the precise ques-tion(s) that the study is required to answer andknowing the informational needs of the target au-dience. Over the long term, it is recommendedthat effort be directed towards better consensusand a greater degree of standardization of defini-tions and methodologies for assessing health eco-nomic questions through clinical studies.

References

1 Drummond M, Davies L. Economic analysis along-side clinical trials: revisiting the methodological is-sues. Int J Tech Ass Health Care 1991;7:561–73.

2 Drummond M. The future of pharmacoeconomics:bridging science and practice. Clin Ther 1996;18:969–78.

3 Haycox A, Drummond M, Walley T. Pharmacoeco-nomics: integrating economic evaluation into clini-cal trials. Br J Clin Pharmacol 1997;43:559–62.

4 O’Brien B. Economic evaluation of pharmaceuti-cals: Frankenstein’s monster or vampire of trials?Med Care 1996;34:D599–DS108.

5 FTC Bureau of Economics and Consumer Protec-tion. In the Matter of Pharmaceutical Marketingand Information Exchange in the Managed CareEnvironments: Public Hearings. Dockett no. 95–N0228. Washington, DC: Federal Trade Commis-sion, 1996.

6 Faich GA, Guess HA, Kuritsky JN. Postmarketingsurveillance for drug safety. In: Cato AE (ed.), Clin-ical Drug Trials and Tribulations. New York: Mar-cel Dekker, Inc., 1988.

7 International Society for Pharmacoepidemiology.Good research practice guidelines. Pharmacoepide-miol Drug Safety 1996;5:333–8.