outcomes research: what is it and why does it matter?

Internal Medicine Journal 2003; 33: 110–118

REVIEW

Outcomes research: what is it and why does it matter?

M. JEFFORD, M. R. STOCKLER and M. H. N. TATTERSALL

1Ludwig Institute for Cancer Research, Heidelberg, Victoria, 2Department of Medicine, Department of Public Health and Community Medicine and the National Health and Medical Research Council Clinical Trials Centre, and 3Department of Cancer Medicine, University of Sydney, New South Wales, Australia

Abstract

Outcomes research is a broad umbrella term withouta consistent definition. However it tends to describeresearch that is concerned with the effectiveness ofpublic-health interventions and health services; thatis, the outcomes of these services. Attention is fre-quently focused on the affected individual – withmeasures such as quality of life and preferences – butoutcomes research may also refer to effectiveness ofhealth-care delivery, with measures such as cost-effectiveness, health status and disease burden. The

present review details the historical background ofoutcomes research to reveal the origins of its diver-sity. The value and relevance of outcomes research,commonly employed research techniques and exam-ples of recent publications in the area are also dis-cussed. (Intern Med J 2003; 33: 110–118)

Key words: cost-benefit analysis, decision supporttechniques, health-care quality, access and evaluation,outcome and process assessment (health care), qualityof life.

WHAT IS ‘OUTCOMES RESEARCH’?

There appears to be no consistent definition of whatconstitutes health outcomes research. It has beensuggested that ‘outcomes research’ is a difficult termto define because of the breadth of research includedunder this ‘umbrella expression’.1 The US NationalLibrary of Medicine does not include the term as amedical subject heading (MESH).2 Related termsinclude ‘health services research’ and ‘outcomeassessment (health care)’. Health services research isdefined as:

the integration of epidemiologic, sociological, economic, and other analytic sciences in the study of health services. Health services research is usually concerned with relationships between need, demand, supply, use, and outcome of health services. The aim of the research is evaluation, particularly in terms of structure, process, output and outcome.2

Outcome assessment (health care) is defined as:

research aimed at assessing the quality and effectiveness of health care as measured by the attainment of a specified end result or outcome. Measures include parameters such as improved health, lowered morbidity or mortality, and improvement of abnormal states (such as elevated blood pressure).

The term ‘outcomes research’ describes a variety offields of research that use a variety of methodologies,often with differing aims.1 The US Agency forHealthcare Research and Quality suggests that:

outcomes research seeks to understand the end results of particular health care practices and interventions. End results include effects that people experience and care about, such as change in the ability to function. In particular, for individuals with chronic conditions – where cure is not always possible – end results include quality of life as well as mortality. By linking the care that people get to the outcomes they experience, outcomes research has become the key to developing better ways to monitor and improve the quality of care.3

Correspondence to: Michael Jefford, Peter MacCallum Cancer Institute, Locked Bag 1, A’Beckett Street, Vic. 8006, Australia. Email: [email protected]

Received 24 October 2001; accepted 10 April 2002.

What is outcomes research? 111


An assessment of the historical factors that haveshaped the field is required in order to understand thediversity of outcomes research.

HISTORICAL BACKGROUND

In 1913, a Massachusetts surgeon, Ernst Codman,noted that hospitals reported the number of patientstreated, but did not indicate whether patientsappeared to benefit from treatment.4 He argued thathospitals should produce a report of their treatmentresults, standardized to allow comparison betweenhospitals. Furthermore, he suggested that all aspectsof hospital practice should be evaluated to ensure thatthey produced favourable outcomes. This includesnot only a focus on patient encounters, but also ananalysis of all ‘products’ of the hospital including, forexample, the efficiency of administrative proceduresand staff and student training.

The term ‘outcome’ was coined by Donabedian, whodeveloped a paradigm for quality assessment,comprising structure, process and outcome.5 Herecognized that, while some outcomes (such as death)might be easily recognized and measured, othersmight be less easily defined and measured. Amongthe latter he included ‘patient attitudes and satisfac-tions, social restoration and physical disability andrehabilitation’.6 Donabedian suggested that,‘outcomes, by and large, remain the ultimate valida-tors of the effectiveness and quality of medical care’.He has continued to guide the assessment of quality.7

In 1973, Wennberg and Gittelsohn reported widevariation in medical practice within Vermont, USA.8

There was considerable variation in resource input,use and expenditure among neighbouring commun-ities but without obvious adverse sequelae. Wennberget al. attributed this variation to differences in phys-icians’ diagnostic style and to physicians’ beliefs in theefficacy of particular treatments.9,10 In 1982,McPherson et al. confirmed variations in the use ofsix common surgical procedures in the USA, UnitedKingdom and Norway.11 There appeared to be a lackof consensus regarding effective medical proceduresand techniques. There also seemed to be no clear linkbetween these procedures and favourable results.Chassin et al. queried whether inappropriate use ofprocedures could explain geographical variation ofresults.12 They studied the use of coronary angi-ography, carotid endarterectomy and upper-gastrointestinal-tract endoscopy in a variety ofregions. Although they recognized a high incidence ofinappropriate use, Chassin et al. concluded that this

did not explain variations in the use of services. Otherstudies around this time, including further obser-vations by Wennberg et al.,13,14 reported variation inthe use of hospital services yet noted that healthoutcomes were not compromised. These studiesemphasized the need for research to determine thecauses and implications of differences in practice.

Around this time McNeil et al. reported a study thatwas designed to assess preferences for treatment,involving a trade between improved quality andquantity of life, following the diagnosis of laryngealcancer.15 The study surveyed the attitudes of healthyvolunteers and found that, to maintain their voices,approximately 20% of volunteers would chose radi-ation treatment over surgery. The study was importantin that it emphasized the use of outcomes that werepatient-focused.

In the 1988 Shattuck lecture, Ellwood called for thedevelopment of ‘outcomes management, a “tech-nology of patient experience”’.16 Reviewing thepreceding changes to the US health system, Ellwoodstated that, ‘the most destabilizing consequence of therestructuring of the health system has also been, in myview, the most desirable one: patients, payers, andexecutives of health care organizations now have bothhigher expectations and greater power’. The problem,he claimed, was that:

too often, payers, physicians, and health care executives do not share common insights into the life of the patient…the problem is our inability to measure and understand the effect of the choices of patients, payers and physicians on the patient’s aspirations for a better quality of life. The result is that we have uninformed patients, skeptical payers, frustrated physicians, and besieged health care executives.16

He called for a technology of ‘outcomes manage-ment’, designed to allow all parties to make rationalchoices based on improved insights into the effectsthat those choices have upon the lives of patients.Outcomes management, he discussed, involves:(i) increased reliance on standards and guidelines,(ii) the collection of clinical outcome data (as well aspatient-completed functional and well-being data),(iii) pooling and analysis of data through the use ofcentralized databases and (iv) subsequent dissemin-ation of results and modification of guidelines.

Based upon the recognition of wide variations in prac-tice, evidence of inappropriate use and the need forhealth-care evaluation, the US Health Care Financing

112 Jefford et al.


Administration and the Public Health Service beganto assume an active role in providing information toguide medical practice.17

Reflecting upon Ellwood’s vision, Relman suggestedthat the era of assessment and accountability repre-sented the third revolution in medical care, followingthe ‘era of expansion’ of health services from the1940s through the 1960s and the ‘era of cost contain-ment’ beginning in the 1980s.18

Paralleling these changes was an evolving shift from aprovider-centred model of medicine to patient-centred health care.19 Patient-based outcomes andgreater accountability therefore became more relevantand important.

Available governmental funding further droveoutcomes research in the USA. Initial projects fundedby the Agency for Health Care Policy and Researchwere determined on the basis of ‘the number ofindividuals affected, the extent of uncertainty orcontroversy with respect to the use of a procedure orits effectiveness, the level of related expenditure, andthe availability of data’.20 Initially, prospective, rand-omized studies were discouraged. Projects fundedsubsequently have encouraged the generation ofprimary data regarding the effectiveness of interven-tions, either through randomized trials or prospectivelongitudinal studies. Thus outcomes research hascome to describe a range of activities (Table 1).

Lee et al. have proposed a conceptual framework ofhow outcomes research interfaces with clinical trials.1

The framework developed by Lee et al. allows us torecognize both overlapping and exclusive elementsof these fields at the levels of: (i) research topics,(ii) outcomes, (iii) secondary analysis and (iv) appli-cations. Clinical trials are predominantly designed toassess the efficacy and effectiveness of specific inter-ventions. The goal of outcomes research may includethe assessment of quality of care, access and the effec-tiveness of general health-care strategies. The typicaloutcomes in clinical trials research are survival,response and adverse event rates. The outcomes intypical outcomes research studies are processes, costsand health-related quality of life. Methods ofsecondary analysis, such as meta-analysis anddecision analysis, may use the same data sets, but arefrequently considered to be elements of clinical andoutcomes research, respectively. Clinical trialsresearch is aimed primarily at informing clinical deci-sions, whereas outcomes research is aimed primarilyat informing health policy decisions.

However, outcomes research and clinical trialsresearch have not developed in isolation. Outcomesresearch has progressed beyond studies of effective-ness to encompass quality of care, decision analysisand the analysis of administrative databases.Modern clinical trials often include measures ofhealth status, quality of life, resource utilization andcost. Lee et al. recognize that the distinctionsbetween clinical trials research and outcomesresearch are useful conceptually, but may be over-simplifications in practice.

Thus, clinical trials and outcomes research aredifferent, but they complement, rather than competewith, each other. Pronovost and Kazandijan providesome innovative examples of how the concepts ofclinical trial design and outcomes research can beintegrated to answer questions about quality improve-ment using modifications of case-control studies andrandomized trials.21

THE VALUE OF OUTCOMES RESEARCH

Outcomes research complements clinical trialresearch. It aims to: (i) provide better information toinform patient decisions, (ii) guide health providersand (iii) inform health policy decisions. In Table 2 wehave suggested ways that outcomes research mightbenefit the individual patient, the health-care prac-titioner, the health-care organization and thegovernment. These benefits may also be derived fromclinical research and other means.

Table 1 The scope of outcomes research

Outcomes research may focus on:Quality-of-life measuresEffectivenessCostQuality of carePatient preferencesAppropriatenessAccessHealth status

In areas such as:Disease preventionScreeningDrug treatmentMedical proceduresMedical practicesDiagnostic testsGuidelinesHealth-care policy



EXAMPLES OF RECENT PUBLICATIONS IN OUTCOMES RESEARCH

Below we briefly review four recent publications toillustrate aspects of outcomes research.

Who gets adjuvant treatment for stage II and stage III rectal cancer?Method: the use of linked databases

Value: greater use of effective interventions

Schrag et al. conducted a retrospective cohort studyto examine the relationship between patient charac-teristics and the use of adjuvant pelvic radiation withor without chemotherapy.22 American patients withstage-II or stage-III rectal adenocarcinoma were iden-tified from the linked Surveillance, Epidemiology andEnd-Results, and Medicare databases. Analysis ofcoding identified episodes of surgery consistent withdefinitive tumour resection that occurred ≤6 monthsafter diagnosis. Medicare coding identified the appro-priate procedural and revenue codes consistent withthe use of radiotherapy and chemotherapy. Informa-tion was also collected on: (i) type of surgicalprocedure, (ii) comorbidities and (iii) demographicdata (i.e. age, gender, race and median income forgeographical area). The study found that only 57% ofpatients received adjuvant radiation (mostly

combined with chemotherapy). The patient charac-teristic most highly correlated with receipt ofradiation was age at diagnosis.

The data suggest that adjuvant radiation is under-used. This information would not be found withtraditional clinical trials research. This study alsoidentified important areas for further research, forexample, the exploration of physicians’ and patients’knowledge and attitudes about adjuvant radiationtherapy.

Cost-effectiveness of radiofrequency ablation (RFA) for supraventricular tachycardia (SVT).

Method: decision analysis

Value: cost savings, greater use of effective interventions

Cheng et al. used a decision-analysis model tocompare the health and economic outcomes of threetreatment strategies for patients with SVT: (i) initialRFA, (ii) long-term antiarrhythmic drug therapy or(iii) treatment of acute episodes of arrhythmia withantiarrhythmic drugs.23 Costs were estimated from amajor academic centre and from the literature, andtreatment effectiveness was estimated from reportsof clinical studies. Probabilities of clinical outcomeswere also based on published data. Utility data werebased upon patient-reported quality of life before

Table 2 Suggested benefits of outcomes research

ConsumerIncreased participation in decision-makingIncreased choice regarding hospital/practitioner/treatment optionsAssurance regarding effectiveness of interventionsAssessment and development of interventions to improve well-being, not just survival

Health-care providerGreater certainty regarding the benefit of an interventionStandards/guidelines to guide clinical practiceShared responsibility in decision-makingProtection from malpractice suits (if complying with above)

Health-care organization managementGreater use of effective interventionsDiscontinuation of ineffective interventions/practicesAn organizational culture emphasizing qualityCost savings as inappropriate use is eliminated (i.e. interventions, medications, hospitalizations)

GovernmentCost savings as inappropriate use is eliminated (i.e. interventions, medications, hospitalizations)Greater ability to plan health servicesOnly effective pharmaceuticals and services are subsidizedTarget research in areas of greatest potential impact based on examination of databases etc.

114 Jefford et al.


and after RFA. This study concluded that RFAsubstantially improves quality of life and reducesexpenditures when used to treat symptomaticpatients. RFA improved quality-adjusted life expect-ancy by 3.1 quality-adjusted life years (QALY) andreduced lifetime medical expenditure by $US27 900,compared with long-term drug therapy. Long-termdrug therapy was more effective and had lower costscompared with episodic drug treatment. Thefindings were highly robust in sensitivity analyses;that is, similar results were obtained when multiplefactors were varied across a wide range of plausiblevalues.

This study included a measure of patient preference(utility assessment) and economic analysis andreported measures of marginal effectiveness. Decisionanalysis provides a rational approach for decisionmaking in areas where definitive data are lacking, andparticularly in identifying areas of critical uncertainty.

Completeness of safety reporting in randomized trialsMethod: survey

Value: emphasis on quality at a procedural/organizational level

Ioannidis and Lau recently addressed the quality ofsafety reporting in clinical trials.24 They surveyed192 randomized studies within seven topics ofinternal medicine and assessed the adequacy ofreporting of adverse effects. Principal measureswere: (i) the severity of adverse clinical events andlaboratory abnormalities and (ii) the frequency andreasons for withdrawals due to toxic effects. Thestudy found that the quality and quantity of safetyreporting varied, but was generally inadequate. Theseverity of clinical adverse events and laboratory-determined toxicity was adequately defined in only39% and 29% of trial reports, respectively. Only46% of trials stated the frequency of specific reasonsfor discontinuation of study treatment due totoxicity. The authors concluded that the standard ofcollection, analysis and reporting of safety data inclinical trials lags behind that for efficacy data. Theysuggested that the outcome measures used in thisstudy could be incorporated into the recommend-ations of the Consolidated Standards of ReportingTrials statement.25

This study demonstrates the diversity of importantoutcomes and measures of quality. Measures ofquality may be applied to areas beyond the directpatient–practitioner focus.

Randomized trial comparing traditional Chinese medical acupuncture, therapeutic massage, and self-care education for chronic low back painMethod: overlap between the traditional clinical trial and outcomes research

Value: assessment and development of interventions to increase well-being and increased certainty regarding the benefit of interventions

The study of Cherkin et al. randomly allocatedpatients with chronic lower-back pain to traditionalChinese medical acupuncture, therapeutic massage orself-care education.26 The primary outcomes weresymptoms and dysfunction. Secondary outcomesincluded disability, health-care use and cost. Patientswere assessed 4, 10 and 52 weeks after a 10-weekintervention period. At baseline and at each follow-upvisit patients were asked how ‘bothersome’ back pain,leg pain, numbness and tingling had been in thepreceding week, each on a scale from ‘0’ to ‘10’.Patients also completed the Roland Disability Scale.Automated health-care use data were collected,including a record of all provider visits, medicationsdispensed, imaging procedures, operations and hospit-alizations. With respect to primary end-points, thestudy found that therapeutic massage was effective forpersistent lower-back pain and appeared to conferlong-lasting benefit, as assessed by 1-year follow-up.

This study exemplifies the interface between tradi-tional clinical trials and outcomes research. The studyused patient-focused outcome data, includingmeasures of health perception (symptom panel) andmeasures of function (disability scale), in a rand-omized clinical trial. Quality of life is the mostimportant outcome in a clinical scenario such as this.The study also illustrates the application of adminis-trative data to explore important unconventionaloutcome measures in the rigorous context of a rand-omized trial.

SOME TECHNIQUES/METHODOLOGIES IN OUTCOMES RESEARCH

Because of the breadth of outcomes research, a dis-cussion of all approaches and methodologies isimpossible. However several approaches are com-monly used, and these are briefly discussed below.

Measures used to assess well-being and satisfactionClancy and Eisenberg have provided a useful descrip-tion of the broad aspects of health-related quality



of life.27 Health perceptions may be assessed throughsymptom panels. Several validated symptommeasures have entered routine clinical practice. Forexample the American Urological Associationsymptom index scale is used by >80% of practisingurologists in the USA.28 Functional measures may beused to assess the impact of health interventions on arange of domains, including physical, mental, socialand role function. Preference-based measures convey themeaning that a person places upon their individualhealth status. Patient satisfaction may also be directlyassessed. This may include interpersonal and tech-nical aspects of care.

Instruments may be generic or disease-specific.29

Generic instruments allow comparison betweendifferent conditions and between different interven-tions. They may also detect the differential effectsthat an intervention has on different aspects of healthstatus. Specific instruments focus on: (i) singledisease states (such as asthma), (ii) patient groups(such as the frail elderly), (iii) areas of function (suchas sleep) or (iv) a particular problem (such as pain).29

Specific instruments are often considered to be moreclinically sensible. They may also be more sensitive tochanges in specific aspects over time. The validity ofmany scales has been established. Acknowledging thetension between ‘generalizeability’ and specificity,several groups have recommended a modularapproach, which incorporates instruments that recon-cile these conflicting needs by adding specific items toa generic core.30–33

Economic analysisEconomic analysis includes techniques such ascost outcome, cost-effectiveness and cost-utilityanalyses.34

Cost-outcome studies are descriptive and indicate thecosts associated with a particular disease or treatmentstrategy. The lack of information about efficacy andthe lack of a suitable comparison often limit theusefulness of such studies. For example, the mediancost in Canada of treating women with advancedovarian cancer with second-line and subsequentchemotherapy was estimated to be approximately$Can37 000; however, costs and outcomes in theabsence of such treatment were not available.35

Cost-effectiveness studies compare the incremental costof one treatment over another with the incrementalbenefit in terms of a single common outcome.34 Fortreatments that influence survival, the cost can beexpressed in terms of life-years gained. While this can

be useful for comparing therapies with similaroutcomes, it is not possible to compare interventionswith different outcomes using this method. In addi-tion, it can be difficult to take into account more thanone component of a health strategy, such as theadverse effects of drug therapy.

Cost-utility analyses account for the possibility thatnot all outcomes are valued equally by weightingoutcomes according to their perceived value (utility).Years in good health are likely to be valued more thanyears with sickness. For example, gains in survivalmay be in varying states of imperfect health, due tothe effects of the disease or treatment. To account forthis, the life-years gained in a particular state may bemultiplied by the utility of that health-state to giveQALY. Utility refers to the value attributed to aparticular health-state (outcome). Utility measuresare an example of a generic measure of health-relatedquality of life because they provide a summary ofquality of life that can be compared across diseases,conditions and populations.

Cost-utility analyses are expressed in terms of dollarsper QALY gained. For example, the incremental costin the USA of adjuvant chemotherapy for node-positive breast cancer in pre-menopausal women hasbeen estimated to be about $US10 000 per QALYgained.36

Decision analysisSarasin describes decision analysis as the ‘quantitativeapplication of probability and utility theory todecision making under conditions of uncertainty’.37

Decision analysis involves modelling a problem toguide decision-making. The model builds onavailable clinical information such as: (i) prevalenceof disease, (ii) effectiveness of interventions,(iii) incidence of side-effects, (iv) associated costs and(v) outcome measures, such as patient-assignedutility values. Thus, decision analysis is able to linkresearch results, patient preferences and populationdata. Sensitivity analysis involves varying the assump-tions made within the model. For example, the risk ofside-effects may be higher (or lower), or the effective-ness of an intervention may be lower (or higher) thanin the baseline model. Similarly, patient preferences(reflected in utility values) may vary. Sensitivityanalysis allows one or many factors to be adjusted,alone or simultaneously, to determine whether thisaffects the apparent best choice; that is, how sensitivethe decision is to the assumptions on which the modelis based.

116 Jefford et al.


Decision analysis can be particularly useful wherethere is inadequate information available from clinicalstudies to guide treatment or policy recommen-dations. Its greatest benefit here is to identify keyareas of uncertainty requiring further research.Kassirer et al. also suggest that decision analysis canbe used: (i) to address health policy questions aboutscreening and prevention, (ii) to weigh trade-offsbetween tests and treatments and (iii) for the inter-pretation of clinical data where there is uncertainty.38

Data sources in outcomes researchThe use of databases facilitates data explorationthrough the analysis of interventions and outcomesand, thus, through the generation of novel insights. Awide variety of health-information sources exists,including administrative databases, clinical databases,disease registries and clinical-trial databases. Thesedatasets may be linked with information from othersources, such as census or electoral role data. Issuessurrounding databases and outcomes research werediscussed at the Regenstrief Conference MeasuringQuality, Outcomes, and Cost of Care Using Large Data-bases and reported in a supplement to Annals ofInternal Medicine in 1997.39

Administrative databases

Administrative data are obtained from health-careadministration, member enrolment and the collectionof payments for services.40 Producers of such datainclude federal and state governments and privatehealth insurers. Clinical content is often low and mayonly include demographic details and coding infor-mation concerning diagnoses and procedures.Nevertheless, administrative data are importantbecause they are readily available, inexpensive toacquire, computer-readable and, typically, cover largepopulations.

Clinical databasesClinical databases may include laboratory, radiology,pharmacy and surgery scheduling systems. Theycontain detailed clinical information, covering abroad population. However, a major disadvantage ofthese datasets is that the information is often recordedin a non-standardized manner and is difficult toaccess. Thus, it is often necessary to manually extractdata elements. This is expensive, time-consuming andprone to error.41

Disease registersDisease registers, such as cancer registers, provideincidence data and, frequently, detailed clinical infor-

mation. They are normally centralized and, withmandatory reporting, offer the potential for thorough,accurate reporting. One disadvantage is the frequentlack of a denominator or reference population.

Clinical trial databasesThese contain detailed clinical information regardingtrial participants. Because of this, detailed and multi-variate analysis within the data set may be possible.Disadvantages include the potential difficulty ingeneralizing to a broader population and the time andexpense required to collect high-quality data.

CensusesCensuses provide information on the demographiccomposition of large populations with respect to suchvariables as age, gender, ethnicity, place of residence,employment and education. Importantly, these datamay be used as a denominator for other data (such ascancer incidence) to determine rates. However,collecting census data is expensive, logistically diffi-cult and limited to the collection of a relatively smallamount of information.

Linking datasets may overcome the individual limit-ations of the various data sources. Differentinformation sources may be combined for modellingin decision analysis. Inferences may also be madewhen different information sources are juxtaposed,which may serve to generate hypotheses or promptfurther research.

PUBLIC REPORTING OF MEDICAL OUTCOMES

At first glance, the public reporting of health out-comes seems an important social responsibility. Inthe Bristol case in the United Kingdom, it is possiblethat more public presentation of outcomes mighthave avoided deaths.42 The New York State Depart-ment of Health collects data on quality of care pro-vided to patients undergoing coronary artery bypassgrafting.43 The publication of surgeon-specific deathrates raised many concerns. Many physicians lack theskill to interpret and critically evaluate medical liter-ature,44 which leads us to ask, Can consumers beexpected to appraise outcomes data? Concerns suchas independent data validation and appropriate riskadjustment may not be adequately considered. Publicreporting may lead to hospitals or physicians avoidinghigh-risk patients in an attempt to lower risk-adjustedmortality rates. Regarding the public release of infor-mation on organizational performance, Epstein sug-gests that attention be given to determining what



constitutes a valid assessment of quality and how toensure the integrity of reported data.45 He states that,‘it has become clear that providing information toconsumers in a way that is understandable and allowsthem actually to use it is at least as formidable a taskas developing reliable and valid quality indicators’.45

THE AUSTRALIAN SITUATION

Australia, like other developed countries, is conf-ronted with the challenge of providing an equitable,efficient, affordable and high-quality health system.Cost containment is necessary. The shift to casemix-based funding has produced gains in throughput andhas reduced costs.46 Casemix classification systemshave also encouraged inter-hospital comparisons onissues such as the planning of bed and staff numbers,utilization review and funding allocations. What ismissing from casemix is an assessment of quality ofcare (including the process) and cogent measures ofpatient outcomes.

In Australia, the quality of health-care delivery ispartly regulated by legislation (i.e. compulsory regis-tration of health-care providers) and also regulated bythe requirement for accreditation of health-care facil-ities. Australia has embraced the importance ofalternative outcome measures in the process ofapproval of drugs to the Pharmaceutical BenefitsScheme (PBS). All applications include a cost-effec-tiveness analysis. The standard of health-relatedeconomic evaluations in Australia was questioned bySalkeld et al. in 1995.47 Although Hill et al. suggestthat there are significant problems with manypharmacoeconomic analyses, they report that theintensive evaluation process used in the PBS allowsfor identification and correction of many problems.48

Although outcomes research in Australia has not beenfunded as generously as in the USA, Australianoutcomes research and resulting publications areincreasing. Australian clinical trialists have pioneeredand led the incorporation of patient-based measuresand economic analyses into clinical trials.49,50

CONCLUSIONS

Outcomes research aims to provide information to allstakeholders (including patients, health-careproviders, health-care managers and government) toallow more rational, evidence-based decision-making.Outcomes research complements clinical trials andother more traditional forms of clinical research.Outcomes research also offers the prospect ofimproving the quality of other aspects of health care

by applying a scientific approach to the evaluation ofa wider range of clinical, management and organi-zational problems.

REFERENCES

1 Lee SJ, Earle CC, Weeks JC. Outcomes research in oncology: history, conceptual framework, and trends in the literature. J Natl Cancer Inst 2000; 92: 195–204.

2 National Library of Medicine. Medical Subject Headings [cited 2002 September 18]. Available from: URL: http://www.nlm.nih.gov/mesh/

3 Agency for Healthcare Research and Quality. Outcomes Research Fact Sheet. AHRQ Publication no. 00-P011. Rockville: Agency for Healthcare Research and Quality; 2000. Available from: URL: http://www.ahrq.gov/clinic/outfact.htm

4 Codman EA. The product of a hospital. Surg Gynecol Obstet 1914; 18: 491–6.

5 Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q 1966; 44: 166–206.

6 Kelman HR, Willner A. Problems in measurement and evaluation of rehabilitation. Arch Phys Med Rehab 1962; 43: 172–81.

7 Donabedian A. The quality of care. How can it be assessed? JAMA 1988; 260: 1743–8.

8 Wennberg JE, Gittelsohn A. Small area variations in health care delivery. Science 1973; 182: 1102–8.

9 Wennberg JE, Gittlesohn A. Variations in medical care among small areas. Sci Am 1982; 246: 120–34.

10 Wennberg JE, Barnes BA, Zubkoff M. Professional uncertainty and the problem of supplier-induced demand. Soc Sci Med 1982; 16: 811–24.

11 McPherson K, Wennberg JE, Hovind OB, Clifford P. Small-area variations in the use of common surgical procedures. an international comparison of New England, England, and Norway. N Engl J Med 1982; 307: 1310–4.

12 Chassin MR, Kosecoff J, Park RE, Winslow CM, Kahn KL, Merrick NJ et al. Does inappropriate use explain geographic variations in the use of health care services? A study of three procedures. JAMA 1987; 258: 2533–7.

13 Wennberg JE, Freeman JL, Culp WJ. Are hospital services rationed in New Haven or over-utilised in Boston? Lancet 1987; 1: 1185–9.

14 Wennberg JE, Freeman JL, Shelton RM, Bubolz TA. Hospital use and mortality among Medicare beneficiaries in Boston and New Haven. N Engl J Med 1989; 321: 1168–73.

15 McNeil BJ, Weichselbaum R, Pauker SG. Speech and survival: tradeoffs between quality and quantity of life in laryngeal cancer. N Engl J Med 1981; 305: 982–7.

16 Ellwood PM. Shattuck lecture – outcomes management. A technology of patient experience. N Engl J Med 1988; 318: 1549–56.

17 Roper WL, Winkenwerder W, Hackbarth GM, Krakauer H. Effectiveness in health care. An initiative to evaluate and improve medical practice. N Engl J Med 1988; 319: 1197–202.

18 Relman AS. Assessment and accountability: the third revolution in medical care. N Engl J Med 1988; 319: 1220–2.

19 Laine C, Davidoff F. Patient-centered medicine. A professional evolution. JAMA 1996; 275: 152–6.

20 Agency for Health Care Policy and Research. Medical Treatment Effectiveness Research. Rockville: Agency for Health Care Policy and Research; 1990.

118 Jefford et al.


21 Pronovost PJ, Kazandjian VA. A new learning environment: combining clinical research with quality improvement. J Eval Clin Pract 1999; 5: 33–40.

22 Schrag D, Gelfand SE, Bach PB, Guillem J, Minsky BD, Begg CB. Who gets adjuvant treatment for stage II and III rectal cancer? Insight from surveillance, epidemiology, and end results – Medicare. J Clin Oncol 2001; 19: 3712–8.

23 Cheng CH, Sanders GD, Hlatky MA, Heidenreich P, McDonald KM, Lee BK et al. Cost-effectiveness of radiofrequency ablation for supraventricular tachycardia. Ann Intern Med 2000; 133: 864–76.

24 Ioannidis JP, Lau J. Completeness of safety reporting in randomized trials. an evaluation of 7 medical areas. JAMA 2001; 285: 437–43.

25 Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I et al. Improving the quality of reporting of randomized controlled trials: the CONSORT Statement. Jama 1996; 276: 637–9.

26 Cherkin DC, Eisenberg D, Sherman KJ, Barlow W, Kaptchuk TJ, Street J et al. Randomized trial comparing traditional Chinese medical acupuncture, therapeutic massage, and self-care education for chronic low back pain. Arch Intern Med 2001; 161: 1081–8.

27 Clancy CM, Eisenberg JM. Outcomes research: measuring the end results of health care. Science 1998; 282: 245–6.

28 Barry MJ, Fowler FJ Jr, O’Leary MP, Bruskewitz RC, Holtgrewe HL, Mebust WK. Measuring disease-specific health status in men with benign prostatic hyperplasia. Measurement Committee of the American Urological Association. Med Care 1995; 33: AS145–55.

29 Guyatt GH, Feeny DH, Patrick DL. Measuring health-related quality of life. Ann Intern Med 1993; 118: 622–9.

30 McHorney CA. Generic health measurement: past accomplishments and a measurement paradigm for the 21st century. Ann Intern Med 1997; 127: 743–50.

31 Cella DF, Tulsky DS, Gray G, Sarafian B, Linn E, Bonomi A et al. The Functional Assessment of Cancer Therapy Scale: development and validation of the general measure. J Clin Oncol 1993; 11: 570–9.

32 Aaronson NK, Ahmedzai S, Bergman B, Bullinger M, Cull A, Duez NJ et al. The European Organization for Research and Treatment of Cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst 1993; 85: 365–76.

33 Aaronson NK, Bullinger M, Ahmedzai S. A modular approach to quality-of-life assessment in cancer clinical trials. Recent Results Cancer Res 1988; 111: 231–49.

34 Detsky AS, Naglie IG. A clinician’s guide to cost-effectiveness analysis. Ann Intern Med 1990; 113: 147–54.

35 Doyle C, Stockler M, Pintilie M, Panesar P, Warde P, Sturgeon J et al. Resource implications of palliative chemotherapy for ovarian cancer. J Clin Oncol 1997; 15: 1000–7.

36 Hillner BE, Smith TJ. Efficacy and cost effectiveness of adjuvant chemotherapy in women with node-negative breast cancer. A decision-analysis model. N Engl J Med 1991; 324: 160–8.

37 Sarasin FP. Decision analysis and its application in clinical medicine. Eur J Obstet Gynecol Reprod Biol 2001; 94: 172–9.

38 Kassirer JP, Moskowitz AJ, Lau J, Pauker SG. Decision analysis: a progress report. Ann Intern Med 1987; 106: 275–91.

39 Anonymous. Measuring quality, outcomes, and cost of care using large databases. Proceedings of the 6th Regenstrief Conference. Marshall, Indiana, 4–6 September 1996. Ann Intern Med 1997; 127: 665–774.

40 Iezzoni LI. Assessing quality using administrative data. Ann Intern Med 1997; 127: 666–74.

41 McDonald CJ, Overhage JM, Dexter P, Takesue BY, Dwyer DM. A framework for capturing clinical data sets from computerized sources. Ann Intern Med 1997; 127: 675–82.

42 Bolsin SN. Professional misconduct: the Bristol case. Med J Aust 1998; 169: 369–72.

43 Chassin MR, Hannan EL, DeBuono BA. Benefits and hazards of reporting medical outcomes publicly. N Engl J Med 1996; 334: 394–8.

44 Anonymous. How to read clinical journals: II: to learn about a diagnostic test. Can Med Assoc J 1981; 124: 703–10.

45 Epstein AM. Rolling down the runway: the challenges ahead for quality report cards. JAMA 1998; 279: 1691–6.

46 Health Solutions Pty Ltd. Independent Assessment of Casemix in Victoria. Melbourne: Health Solutions Pty Ltd; 1994.

47 Salkeld G, Davey P, Arnolda G. A critical review of health-related economic evaluations in Australia: implications for health policy. Health Policy 1995; 31: 111–25.

48 Hill S, Henry D, Mitchell A. Problems in pharmacoeconomic analyses. JAMA 2000; 284: 1922–4.

49 Coates A, Gebski V, Bishop JF, Jeal PN, Woods RL, Snyder R et al. Improving the quality of life during chemotherapy for advanced breast cancer. A comparison of intermittent and continuous treatment strategies. N Engl J Med 1987; 317: 1490–5.

50 Stewart RA, Sharples KJ, North FM, Menkes DB, Baker J, Simes J. Long-term assessment of psychological well-being in a randomized placebo-controlled trial of cholesterol reduction with pravastatin. The LIPID Study Investigators. Arch Intern Med 2000; 160: 3144–52.

outcomes research: what is it and why does it matter?

Documents