nhs r&d hta programmepure-oai.bham.ac.uk/ws/files/17501744/deeks... · random samples of...

Health Technology Assessm

ent 2005;Vol. 9: No. 26

Indirect comparisons of com

peting interventions

Indirect comparisons of competinginterventions

AM Glenny, DG Altman, F Song, C Sakarovitch, JJ Deeks, R D’Amico, M Bradburn and AJ Eastwood

Health Technology Assessment 2005; Vol. 9: No. 26

HTAHealth Technology AssessmentNHS R&D HTA Programme

The National Coordinating Centre for Health Technology Assessment,Mailpoint 728, Boldrewood,University of Southampton,Southampton, SO16 7PX, UK.Fax: +44 (0) 23 8059 5639 Email: [email protected]://www.ncchta.org ISSN 1366-5278

FeedbackThe HTA Programme and the authors would like to know

your views about this report.

The Correspondence Page on the HTA website(http://www.ncchta.org) is a convenient way to publish

your comments. If you prefer, you can send your comments to the address below, telling us whether you would like

us to transfer them to the website.

We look forward to hearing from you.

July 2005

Copyright notice

© Queen's Printer and Controller of HMSO 2005 HTA reports may be freely reproduced for the purposes of private research and study and may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising Violations should be reported to [email protected] Applications for commercial reproduction should be addressed to HMSO, The Copyright Unit, St Clements House, 2–16 Colegate, Norwich NR3 1BQ

How to obtain copies of this and other HTA Programme reports.An electronic version of this publication, in Adobe Acrobat format, is available for downloading free ofcharge for personal use from the HTA website (http://www.hta.ac.uk). A fully searchable CD-ROM isalso available (see below).

Printed copies of HTA monographs cost £20 each (post and packing free in the UK) to both public andprivate sector purchasers from our Despatch Agents.

Non-UK purchasers will have to pay a small fee for post and packing. For European countries the cost is£2 per monograph and for the rest of the world £3 per monograph.

You can order HTA monographs from our Despatch Agents:

– fax (with credit card or official purchase order) – post (with credit card or official purchase order or cheque)– phone during office hours (credit card only).

Additionally the HTA website allows you either to pay securely by credit card or to print out yourorder and then post or fax it.

Contact details are as follows:HTA Despatch Email: [email protected]/o Direct Mail Works Ltd Tel: 02392 492 0004 Oakwood Business Centre Fax: 02392 478 555Downley, HAVANT PO9 2NP, UK Fax from outside the UK: +44 2392 478 555

NHS libraries can subscribe free of charge. Public libraries can subscribe at a very reduced cost of £100 for each volume (normally comprising 30–40 titles). The commercial subscription rate is £300 per volume. Please see our website for details. Subscriptions can only be purchased for the current orforthcoming volume.

Payment methods

Paying by chequeIf you pay by cheque, the cheque must be in pounds sterling, made payable to Direct Mail Works Ltdand drawn on a bank with a UK address.

Paying by credit cardThe following cards are accepted by phone, fax, post or via the website ordering pages: Delta, Eurocard,Mastercard, Solo, Switch and Visa. We advise against sending credit card details in a plain email.

Paying by official purchase orderYou can post or fax these, but they must be from public bodies (i.e. NHS or universities) within the UK.We cannot at present accept purchase orders from commercial companies or from outside the UK.

How do I get a copy of HTA on CD?

Please use the form on the HTA website (www.hta.ac.uk/htacd.htm). Or contact Direct Mail Works (seecontact details above) by email, post, fax or phone. HTA on CD is currently free of charge worldwide.

The website also provides information about the HTA Programme and lists the membership of the variouscommittees.

HTA


AM Glenny,1* DG Altman,2 F Song,3

C Sakarovitch,2 JJ Deeks,2 R D’Amico,2

M Bradburn2 and AJ Eastwood4

In collaboration with the International Stroke Trial Collaborative Group

1 Cochrane Oral Health Group, Dental School, University of Manchester, UK

2 Cancer Research UK, Centre for Statistics in Medicine, Wolfson College Annexe, Oxford, UK

3 School of Medicine Health Policy and Practice, University of East Anglia,Norwich, UK

4 Centre for Reviews and Dissemination, University of York, UK

*Corresponding author

Declared competing interests of authors: none

Published July 2005

This report should be referenced as follows:

Glenny AM, Altman DG, Song F, Sakarovitch C, Deeks JJ, D’Amico R, et al. Indirectcomparisons of competing interventions. Health Technol Assess 2005;9(26).

Health Technology Assessment is indexed and abstracted in Index Medicus/MEDLINE,Excerpta Medica/EMBASE and Science Citation Index Expanded (SciSearch®) and Current Contents®/Clinical Medicine.

NHS R&D HTA Programme

The research findings from the NHS R&D Health Technology Assessment (HTA) Programme directlyinfluence key decision-making bodies such as the National Institute for Health and Clinical

Excellence (NICE) and the National Screening Committee (NSC) who rely on HTA outputs to help raisestandards of care. HTA findings also help to improve the quality of the service in the NHS indirectly inthat they form a key component of the ‘National Knowledge Service’ that is being developed to improvethe evidence of clinical practice throughout the NHS.

The HTA Programme was set up in 1993. Its role is to ensure that high-quality research information onthe costs, effectiveness and broader impact of health technologies is produced in the most efficient wayfor those who use, manage and provide care in the NHS. ‘Health technologies’ are broadly defined toinclude all interventions used to promote health, prevent and treat disease, and improve rehabilitationand long-term care, rather than settings of care.

The HTA programme commissions research only on topics where it has identified key gaps in theevidence needed by the NHS. Suggestions for topics are actively sought from people working in theNHS, the public, consumer groups and professional bodies such as Royal Colleges and NHS Trusts.

Research suggestions are carefully considered by panels of independent experts (including consumers)whose advice results in a ranked list of recommended research priorities. The HTA Programme thencommissions the research team best suited to undertake the work, in the manner most appropriate to findthe relevant answers. Some projects may take only months, others need several years to answer theresearch questions adequately. They may involve synthesising existing evidence or designing a trial toproduce new evidence where none currently exists.

Additionally, through its Technology Assessment Report (TAR) call-off contract, the HTA Programme isable to commission bespoke reports, principally for NICE, but also for other policy customers, such as aNational Clinical Director. TARs bring together evidence on key aspects of the use of specifictechnologies and usually have to be completed within a limited time period.

The research reported in this monograph was commissioned by the HTA Programme as project number96/51/99. The contractual start date was in February 1999. The draft report began editorial review in May2002 and was accepted for publication in November 2004. As the funder, by devising a commissioningbrief, the HTA Programme specified the research question and study design. The authors have beenwholly responsible for all data collection, analysis and interpretation, and for writing up their work. TheHTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like tothank the referees for their constructive comments on the draft document. However, they do not acceptliability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTAProgramme or the Department of Health.

Editor-in-Chief: Professor Tom WalleySeries Editors: Dr Peter Davidson, Professor John Gabbay, Dr Chris Hyde,

Dr Ruairidh Milne, Dr Rob Riemsma and Dr Ken SteinManaging Editors: Sally Bailey and Caroline Ciupek

ISSN 1366-5278

© Queen’s Printer and Controller of HMSO 2005

This monograph may be freely reproduced for the purposes of private research and study and may be included in professional journals providedthat suitable acknowledgement is made and the reproduction is not associated with any form of advertising.

Applications for commercial reproduction should be addressed to NCCHTA, Mailpoint 728, Boldrewood, University of Southampton, Southampton, SO16 7PX, UK.

Published by Gray Publishing, Tunbridge Wells, Kent, on behalf of NCCHTA.Printed on acid-free paper in the UK by St Edmundsbury Press Ltd, Bury St Edmunds, Suffolk.

Criteria for inclusion in the HTA monograph seriesReports are published in the HTA monograph series if (1) they have resulted from work commissionedfor the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the refereesand editors.

Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search,appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit thereplication of the review by others.

M

Objectives: To survey the frequency of use of indirectcomparisons in systematic reviews and evaluate themethods used in their analysis and interpretation. Alsoto identify alternative statistical approaches for theanalysis of indirect comparisons, to assess theproperties of different statistical methods used forperforming indirect comparisons and to compare directand indirect estimates of the same effects withinreviews.Data sources: Electronic databases.Review methods: The Database of Abstracts ofReviews of Effects (DARE) was searched for systematicreviews involving meta-analysis of randomisedcontrolled trials (RCTs) that reported both direct andindirect comparisons, or indirect comparisons alone. Asystematic review of MEDLINE and other databaseswas carried out to identify published methods foranalysing indirect comparisons. Study designs werecreated using data from the International Stroke Trial.Random samples of patients receiving aspirin, heparinor placebo in 16 centres were used to create meta-analyses, with half of the trials comparing aspirin andplacebo and half heparin and placebo. Methods forindirect comparisons were used to estimate thecontrast between aspirin and heparin. The wholeprocess was repeated 1000 times and the results werecompared with direct comparisons and also theoreticalresults. Further detailed case studies comparing theresults from both direct and indirect comparisons ofthe same effects were undertaken.Results: Of the reviews identified through DARE,31/327 (9.5%) included indirect comparisons. A furtherfive reviews including indirect comparisons were

identified through electronic searching. Few reviewscarried out a formal analysis and some based analysison the naive addition of data from the treatment armsof interest. Few methodological papers were identified.Some valid approaches for aggregate data that could beapplied using standard software were found: theadjusted indirect comparison, meta-regression and, forbinary data only, multiple logistic regression (fixedeffect models only). Simulation studies showed that thenaive method is liable to bias and also produces over-precise answers. Several methods provide correctanswers if strong but unverifiable assumptions arefulfilled. Four times as many similarly sized trials areneeded for the indirect approach to have the samepower as directly randomised comparisons. Detailedcase studies comparing direct and indirect comparisonsof the same effect show considerable statisticaldiscrepancies, but the direction of such discrepancy isunpredictable.Conclusions: Direct evidence from good-quality RCTsshould be used wherever possible. Without thisevidence, it may be necessary to look for indirectcomparisons from RCTs. However, the results may besusceptible to bias. When making indirect comparisonswithin a systematic review, an adjusted indirectcomparison method should ideally be used employingthe random effects model. If both direct and indirectcomparisons are possible within a review, it isrecommended that these be done separately beforeconsidering whether to pool data. There is a need toevaluate methods for the analysis of indirectcomparisons for continuous data and for empiricalresearch into how different methods of indirect


iii

© Queen’s Printer and Controller of HMSO 2005. All rights reserved.

Abstract

Indirect comparisons of competing interventions

AM Glenny,1* DG Altman,2 F Song,3 C Sakarovitch,2 JJ Deeks,2 R D’Amico,2

M Bradburn2 and AJ Eastwood4

In collaboration with the International Stroke Trial Collaborative Group

1 Cochrane Oral Health Group, Dental School, University of Manchester, UK

2 Cancer Research UK, Centre for Statistics in Medicine, Wolfson College Annexe, Oxford, UK3 School of Medicine Health Policy and Practice, University of East Anglia, Norwich, UK4 Centre for Reviews and Dissemination, University of York, UK *Corresponding author

comparison perform in cases where there is a large treatment effect. Further study is needed intowhen it is appropriate to look at indirect comparisons and when to combine both direct andindirect comparisons. Research into how evidence from indirect comparisons compares to that from

non-randomised studies may also be warranted.Investigations using individual patient data from a meta-analysis of several RCTs using different protocols and anevaluation of the impact of choosing different binaryeffect measures for the inverse variance method wouldalso be useful.

Abstract

iv


v

List of abbreviations .................................. vii

Executive summary .................................... ix

1 Background ............................................... 1

2 Research questions .................................... 5

3 Indirect comparisons in published systematic reviews ..................................... 7Methods ...................................................... 7Results ........................................................ 8Summary .................................................... 14

4 Statistical methods for indirect comparisons ............................................... 17Systematic review of the literature ............. 17Statistical methods for indirect comparisons ............................................... 18Comments .................................................. 25

5 An empirical investigation of the properties of different statistical methods used forperforming indirect comparisons .............. 27The International Stroke Trial (IST) ......... 27Adaptations of the trial .............................. 27Analyses ...................................................... 28Some theoretical results ............................. 29Results of empirical studies ........................ 29Summary .................................................... 40

6 Statistical discrepancy between the direct and indirect estimate: empirical evidence from published meta-analyses ................... 43Methods ...................................................... 43Results ........................................................ 44Commentary ............................................... 44Summary .................................................... 49

7 Detailed case studies ................................. 51Five cases with significant discrepancy ....... 51Relative efficacy of antimicrobial prophylaxis in colorectal surgery ............... 55Summary .................................................... 63

8 Discussion ................................................... 65Indirect comparisons based on randomisedtrials: current practice ................................ 65Performance of different methods of making indirect comparisons ..................... 65

Quality of evidence from RCTs (directcomparisons) .............................................. 66What to do when direct evidence is available but insufficient ............................ 67What to do when there is no direct evidence ...................................................... 67Are indirect comparisons of RCTs preferable to direct comparisons from non-randomised trials? .............................. 68Comparing multiple interventionssimultaneously ............................................ 69

9 Conclusions ................................................ 71Implications for practice of systematic reviews ........................................................ 71Implications for clinical research ............... 71Recommendations for methodological research ...................................................... 71

Acknowledgements .................................... 73

References .................................................. 75

Appendix 1 Reviews of effectivenessprescreening form ...................................... 81

Appendix 2 Data extraction form ............. 85

Appendix 3 Table of systematic reviews making indirect comparisons ..................... 87

Appendix 4 List of excluded reviews ........ 109

Appendix 5 Search strategy development ............................................... 111

Appendix 6 Additional electronic searchstrategies ..................................................... 115

Appendix 7 Illustrative analyses ............... 119

Appendix 8 Identified meta-analyses providing sufficient data for both direct and indirect comparisons ........................... 127

Health Technology Assessment reportspublished to date ....................................... 135

Health Technology Assessment Programme ................................................ 145

Contents

ACE angiotensin-converting enzyme

AIC adjusted indirect comparison

AIIA angiotensin II antagonist

ALT alanine transaminase

AP aerolised pentamidine

APSAC anisoylated plasminogenstreptokinase activator complex

ASA aminosalicylate

ATC Antiplatelet Trialists’ Collaboration

CAD coronary artery disease

CDSR Cochrane Database of SystematicReviews

Cefot-M cefotaxime plus metronidazole

Cefur-M cefuroxime plus metronidazole

CI confidence interval

Co-A co-amoxiclav

CRD Centre for Reviews andDissemination

CT chemotherapy

ctrl control

D dapsone

DARE Database of Abstracts of Reviews ofEffects

DC direct comparison

Dip dipyridamole

DP D-penicillamine

DVT deep vein thrombosis

EE ethinyl estradiol

ENRIS Evaluating Non-RandomisedIntervention Studies

ESRD end-stage renal disease

FTT Fibrinolytic Therapy Trialists

5-FU 5-fluorouracil

GI gastrointestinal

GLMM generalised linear mixed model

GORD gastro-oesophageal reflux disease

H2RA H2-receptor antagonist

IBS irritable bowel syndrome

IC indirect comparison

IPD individual patient data

IST International Stroke Trial

IVE important vascular events

LMWH low molecular weight heparin

MAP mean arterial pressure

NA not applicable

NNT number needed to treat

NRT nicotine replacement therapy

ns not significant

NSAID non-steroidal anti-inflammatorydrug

OA osteoarthritis

OGD oesophagogastroduodenoscopy

OR odds ratio

P pyrimethamine

PE pulmonary embolism

PEP polyestradiol phosphate

PPI proton pump inhibitor

RA rheumatoid arthritis

RCT randomised controlled trial

RemRR remedication rate ratio

ResRR response rate ratio

RPA reteplase

RR relative risk

RT radiotherapy

continued


vii


List of abbreviations

List of abbreviations continued

SD standard deviation

SE standard error

SEM standard error of the mean

SK streptokinase

SMX sulfamethoxazole

SPID sum of pain intensity difference

SSRI selective serotonin reuptakeinhibitor

SSZ sulfasalazine

SWI surgical wound infection

TMP trimethoprim

TOTPAR total pain relief

t-PA tissue plasminogen activator

UFH unfractionated heparin

List of abbreviations

viii

All abbreviations that have been used in this report are listed here unless the abbreviation is well known (e.g. NHS), or it has been used only once, or it is a non-standard abbreviation used only in figures/tables/appendices in which case the abbreviation is defined in the figure legend or at the end of the table.


ix


BackgroundThe randomised controlled trial (RCT) is the mostvalid design for evaluating the relative efficacy ofhealthcare technology. However, many competinginterventions have not been directly compared inRCTs and indirect methods have been commonlyused in meta-analyses. Such indirect comparisonsare subject to greater bias (especially selectionbias) than head-to-head randomised comparisons,as the benefit of randomisation does not holdacross trials. Therefore, it is essential to evaluatesuch bias that may lead to inaccuracies in theestimates of treatment effects and result ininappropriate policy decisions.

ObjectivesThe objectives of this study were:

� to survey the frequency of use of indirectcomparisons in systematic reviews and evaluatethe methods used in their analysis andinterpretation

� to identify alternative statistical approaches forthe analysis of indirect comparisons

� to assess the properties of different statisticalmethods used for performing indirectcomparisons

� to carry out empirical work comparing directand indirect estimates of the same effects withinreviews.

MethodsThe Database of Abstracts of Reviews of Effects(DARE) (1994 to March 1999) was searched forsystematic reviews involving meta-analysis of RCTsthat reported both direct and indirectcomparisons, or indirect comparisons alone. Asystematic review of MEDLINE (1966 to February2001) and other databases was carried out toidentify published methods for analysing indirectcomparisons.

Study designs were created using data from theInternational Stroke Trial. Random samples ofpatients receiving aspirin, heparin or placebo in16 centres were used to create meta-analyses, with

half of the trials comparing aspirin and placeboand half heparin and placebo. Methods forindirect comparisons were used to estimate thecontrast between aspirin and heparin. The wholeprocess was repeated 1000 times and the resultswere compared with direct comparisons and alsotheoretical results.

Further detailed case studies comparing the resultsfrom both direct and indirect comparisons of thesame effects were undertaken.

ResultsOf the reviews identified through DARE thatincluded meta-analyses of two or more RCTs,31/327 (9.5%) included indirect comparisons. Afurther five reviews including indirect comparisonswere identified through electronic searching. Fewreviews carried out a formal analysis. Some reviewsbased analysis on the naive addition of data fromthe treatment arms of interest. Interpretation ofindirect comparisons was not always appropriate.

Few methodological papers were identified. Somevalid approaches for aggregate data that could beapplied using standard software were found: theadjusted indirect comparison, meta-regressionand, for binary data only, multiple logisticregression (fixed effect models only).

Simulation studies showed that the naive methodis liable to bias and also produces over-preciseanswers. Several methods provide correct answersif strong but unverifiable assumptions are fulfilled.Four times as many similarly sized trials areneeded for the indirect approach to have the samepower as directly randomised comparisons.

Detailed case studies comparing direct andindirect comparisons of the same effect showconsiderable statistical discrepancies, but thedirection of such discrepancy is unpredictable.

ConclusionsWhen conducting systematic reviews to evaluatethe effectiveness of interventions, direct evidence

Executive summary

x

from good-quality RCTs should be used wherever possible. If little or no such evidenceexists, it may be necessary to look for indirectcomparisons from RCTs. The reviewer needs,however, to be aware that the results may besusceptible to bias.

When making indirect comparisons within asystematic review, an adjusted indirect comparisonmethod should ideally be used using the randomeffects model. If both direct and indirectcomparisons are possible within a review, it isrecommended that these be done separatelybefore considering whether to pool data.

Recommendations for researchThere is a need for evaluation of methods foranalysis of indirect comparisons for continuousdata.

There is a need for empirical research into how different methods of indirect comparison

perform in cases where there is a large treatment effect.

Further research is required to consider how todetermine when it is appropriate to look atindirect comparisons and how to judge when tocombine both direct and indirect comparisons.Research into how evidence from indirectcomparisons compares to that from non-randomised studies may also be warranted.

Empirical investigations were based on one large,multicentre trial with a common protocol acrosseach centre. It would be useful to repeat theinvestigations using individual patient data from ameta-analysis of several RCTs using differentprotocols.

The odds ratio was used as the measure of effectwithin this simulation study. Although logisticregression calls for the effect measure to be theodds ratio, it would be interesting to evaluate theimpact of choosing different binary effectmeasures for the inverse variance method.

Executive summary

Well-designed randomised controlled trials(RCTs) generally provide the most reliable

evidence of effectiveness as observed differencesbetween the trial arms can, in general, beconfidently attributed to differences in thetreatment(s) being evaluated.1,2 However, in manyareas, available trials may not have directlycompared the specific treatments or regimens ofinterest. A common example is where there is aclass of several drugs, each of which has beenstudied in placebo-controlled RCTs, but there areno trials (or very few) in which the drugs havebeen directly compared with each other. Forexample, in a systematic review of antibioticprophylaxis for preventing surgical woundinfections after colorectal surgery, only a limitednumber of antibiotics or combinations ofantibiotics (more than 70 regimens in total) weredirectly compared (head-to-head comparisons) in147 randomised trials.3 With the increasing useand development of meta-analytical techniques,comparisons of arms of different RCTs are beingundertaken. Such indirect comparisons are subjectto greater bias (especially selection bias) thanhead-to-head randomised comparisons, as thebenefit of randomisation does not hold acrosstrials. It is vital, therefore, to evaluate such biasesthat may lead to inaccuracies in the estimates oftreatment effects and result in inappropriatepolicy decisions. By identifying the presence,magnitude and, if possible, methods to overcomesuch bias, the accuracy and interpretation ofestimates of the effectiveness of health

technologies will be enhanced. In addition, suchevaluations may give an insight into theusefulness of indirect comparisons in cases wheredirect comparisons are impossible, for example,when examining the placebo effects ofsubcutaneous and oral medication.4

In a review exploring the types of evidence used tosupport the prescribing of one drug over another,McAlister and colleagues5 suggest a hierarchy ofevidence for grading studies that compare a drugwith another of the same class (Table 1).

As expected, direct comparisons from head-to-head RCTs measuring clinically importantoutcomes (referring to long-term efficacy data) areat the top of the hierarchy of evidence at level 1,followed by head-to-head RCTs using validatedsurrogate outcomes (level 2). Comparisons madeacross placebo-controlled RCTs of different drugsare, however, also classified at level 2, despite theincreased threats to validity due to likelydifferences between trials in terms of end-pointdefinitions, inclusion criteria, patientcharacteristics, setting or baseline risk ofoutcomes. McAlister and colleagues5 acknowledgethat the strength of inference from such indirectcomparisons is limited.

It could be argued that since the randomisationelement of the RCT is not (or not fully) usedduring indirect comparison, such methods mayhave important implications on the use of data


1


Chapter 1

Background

TABLE 1 Suggested levels of evidence for comparing the efficacy of drugs within the same class

Level Comparison Study patients Outcomes

1 Within a head-to-head RCT Identical (by definition) Clinically important

2 Within a head-to-head RCT Identical (by definition) Validated surrogate

2 Across RCTs of different drugs Similar or different (in disease status Clinically important or vs placebo and risk factor status) validated surrogate

3 Across subgroup analyses from RCTs Similar or different Clinically important or of different drugs vs placebo surrogate

3 Across RCTs of different drugs vs Similar or different Unvalidated surrogateplacebo

4 Between non-randomised studies Similar or different Clinically important

Adapted from McAlister et al.5

from other types of non-randomised(‘observational’) study design.

Indirect comparisons are not only used to makecomparisons between drugs in the same class orduring subgroup analyses when comparing, forexample, different dosages of the same drug.Comparisons between very different interventions,such as pharmacological interventions versussurgery, or between different classes of drugs havealso been made using indirect comparisons.Examples of each type of comparison can be seenin the health research literature.

For example, an indirect comparison of differentclasses of drugs was undertaken in a meta-analysisof second-line drugs used to treat rheumatoidarthritis.6 The drugs compared were antimalarialdrugs, auranofin, injectable gold, methotrexate, D-penicillamine and sulfasalazine. Sixty-six trialsexamining the efficacy of second line drugs wereincluded. For each drug the results were combinedacross treatment groups. Means for each drugtreatment were generated by weighting eachtreatment group by size at the end of the trial. Tocompare drugs, analysis of variance wasundertaken, weighted by treated group size,multiplied by study quality and adjusted forcovariates shown to have significant associationwith each outcome. A fixed effects model wasused. The outcomes of interest were the tenderjoint count, the erythrocyte sedimentation rateand grip strength. For each outcome, resultsshowed that auranofin tended to be weaker thanother second line drugs. No attempt, however, wasmade to provide data from direct comparisons.

An example of indirect comparisons being used tocompare drugs within a specific class can be seenin the systematic review by Garg and Yusuf.7 Theauthors of the review used indirect comparisons toevaluate the effects of angiotensin-convertingenzyme (ACE) inhibitors on mortality andmorbidity in patients with symptomatic congestiveheart failure. Thirty-two placebo-controlled trialsof ACE inhibitors, including 7105 patients, wereused to make adjusted indirect comparisons ofestimates of effect for the different agentsevaluated. The authors state that “Similar benefitswere observed with several different ACEinhibitors, although the data were largely based onenalapril maleate, captopril, ramipril, quinaprilhydrochloride, and lisinopril”.

Indirect comparisons of the effect of differentdoses of a drug can be illustrated with the reviewby the Homocysteine Lowering Trialists’

Collaboration.8 The review included onlyrandomised trials (with an untreated controlgroup) that assessed the effects of different dosesof folic acid, with or without the addition ofvitamin B12 or B6, on blood homocysteineconcentrations. Twelve trials, with data on 1114patients, were included in the review. Trials weregrouped according to the folic acid regimen used(<1, 1–3 or >3 mg daily). The proportionalreductions in blood homocysteine in the treatedgroups compared with the control groups wereevaluated by an analysis of covariance whichestimated the difference in the post-treatment,log-transformed homocysteine values afteradjustment for baseline homocysteine levels. Thismodel was extended to allow the extent of thisadjustment to vary between studies according tofactors such as folic acid dose, additional vitaminB6 or B12, age, gender or duration of treatment.After adjusting for pretreatment bloodconcentrations of homocysteine and folate, therewas no significant difference between daily doses.Such indirect comparison of results from subgroupanalyses is not uncommon in systematic reviewsand meta-analyses. Boersma and colleagues9

evaluated the effect of delayed thrombolytictreatment following acute myocardial infarctionand short-term mortality. They included 22randomised trials of over 50,000 patients. Thetrials were grouped according to time tofibrinolytic therapy (0–1, ≥ 1–2, ≥ 2–3, ≥ 3–6,≥ 6–12, ≥ 12–24 hours) rather than dosage. Theresults of the subgroup analysis were used toconclude that the “beneficial effect of fibrinolytictherapy is substantially higher in patientspresenting within two hours after symptom onsetcompared to those presenting later”.

The above examples illustrate both unadjusted(naive) and adjusted indirect comparisons. In thenaive approach6 data are pooled across treatmentarms, ignoring the fact that the studies are RCTs(and discarding data from some treatmentgroups). Given that comparisons are being madeacross trial arms in the naive indirect comparisons,negating the randomised nature of the trial, theexclusion of non-randomised or observationalstudies can be questioned. Such methods mayhave important implications for the use of datafrom other types of non-randomised(observational) study design.

In the adjusted indirect comparison, thecomparison of the interventions of interest isadjusted by the results of their direct comparisonwith a common control group (e.g. placebo),partially using the strength of the RCT. The

Background

2

methods used to undertake such adjusted indirectcomparisons vary. In the simplest case, one may beinterested in comparing two interventions B andC. Indirect evidence can be obtained from RCTs ofeither B or C versus a common comparator,perhaps placebo (RCT 1 and RCT 2, Figure 1).One could use the results from one or more trialsto estimate the effect of each intervention relativeto control. An indirect measure of the relativeefficacy of B and C could be made subjectively orusing some statistical procedure. This informationcould be combined with data from any direct Bversus C comparison, again either qualitatively orstatistically. In more complex situations, one maybe interested in trying to estimate simultaneouslythe (relative) effectiveness of several treatments(e.g. various �-blockers and other preventivetreatments for patients who have had a myocardialinfarction, or numerous analgesic drugs used in avariety of pain relief settings).

Several ad hoc approaches have been adopted tohandle such situations. For example, in the casewhere there are a large number of placebo-controlled trials of various drugs a commonapproach is to carry out separate meta-analyses ofthe trials of each drug. The estimated effects arethen compared explicitly or implicitly, ignoringthe fact that the studies (or the patients in them)may not be strictly comparable. This procedure islike producing a league table, and results of thistype appear increasingly in Bandolier10 andelsewhere.11 Such comparisons may be of little useif tables are ranked according to number neededto treat (NNT) estimates in which no-treatmentgroups, placebo-controlled trials and head-to-headcomparisons are included.12

Meta-analyses using both randomised and non-randomised studies have also been undertaken.For example, in a review of thromboprophylaxis

after total hip replacement, Murray and colleaguesanalysed treatment arms from both controlled anduncontrolled studies.13 Such indirect comparisonsbetween non-randomised groups are made on theassumption that these groups are similar acrossthe different studies. However, this may not be areasonable assumption, so analyses of this kind areopen to several sources of bias. For example, asthe efficacy of a treatment may vary amongsubpopulations of patients, differences in baselinecharacteristics between groups within differenttrials (variations in case-mix) may lead to biasedestimates of treatment effect. Even when patientcharacteristics are similar, other aspects may varybetween trials, such as ancillary treatment or otheraspects of patient care (the actual treatment mayvary too).

There is a need to investigate the properties ofsuch procedures and to address the ways in whichsuch meta-analyses can best be carried out. Mostobviously, if trials have all used a common controlintervention (maybe placebo) then there is thepotential to use the control groups as astandardising factor. Bucher and colleagues14

presented a model for undertaking indirectcomparisons that preserves the randomisation ofthe originally assigned patient groups. Theirmodel was tested using a meta-analysis of RCTscomparing two experimental regimens against thestandard regimen for the prevention ofPneumocystis carinii pneumonia in patients withHIV infection. Trials providing a direct estimate ofthe relative effectiveness of the two experimentaldrugs were also identified, and an overall estimateof effect of these trials was calculated. Both theindirect and direct comparisons favouredtrimethoprim–sulfamethoxazole overdapsone/pyrimethamine, but the magnitude ofdifference was less in the direct comparison. Theodds ratio (OR) from the indirect comparison was0.37 [95% confidence interval (CI) 0.21 to 0.65]favouring trimethoprim-sulfamethoxazole,compared with 0.64 (95% CI 0.45 to 0.90) for thedirect comparison using the fixed effects model.Using the random effects model the odds ratiofrom the direct comparison was 0.43 (95% CI 0.21to 0.89), which was similar to that from theindirect comparison. The model presented mayprotect against some of the biases that arisethrough the use of indirect comparisons. Theauthors concluded that only where directcomparisons are unavailable should indirectcomparison meta-analysis be carried out. In suchcases the limitations of such procedures should beexamined thoroughly. However, the approachclearly needs further evaluation.


3


RCT 1 RCT 2

Placebo PlaceboB C

Indirectcomparison

FIGURE 1 Indirect comparison

Indirect comparisons are being used in systematicreviews to evaluate the relative effectiveness ofalternative interventions even though such indirectcomparisons may be less accurate than head-to-head randomised comparisons. It is vital,

therefore, to evaluate the properties of differentstatistical approaches to indirect comparisons toensure that inaccuracies in the estimates oftreatment effects do not result in inappropriatepolicy decisions.

Background

4

The aims of the project were:

� to survey the frequency of use of indirectcomparisons in systematic reviews and evaluatethe methods used in their analysis andinterpretation

� to identify alternative statistical approaches for the analysis of indirect comparisons

� to assess the properties of different statisticalmethods used for performing indirectcomparisons

� to carry out empirical work comparing directand indirect estimates of the same effects withinreviews.

To achieve these aims the project involved a reviewof the literature and methodological and empiricalinvestigations.

Chapter 3 presents a survey of published indirectcomparisons. Chapter 4 contains a systematicreview of the literature identifying differentstatistical approaches to the analysis of indirectcomparisons. Chapters 5 and 6 present themethodological and empirical investigations.Detailed case studies are given in Chapter 7. Eachchapter contains a description of the methodsused, the results and a brief overview of thefindings. Chapter 8 gives an overall discussion,drawing on the findings from Chapter 3–7, andthe implications are discussed in Chapter 9.


5


Chapter 2

Research questions

This chapter summarises the findings of asurvey of frequency of use of indirect

comparisons in published systematic reviews.

MethodsSearch strategyTo survey the frequency of use of indirectcomparisons in published systematic reviews it wasunnecessary to conduct a comprehensive search ofthe research literature. Rather, a careful search ofdatabases of systematic reviews was performed. Inaddition, it was considered impossible to develop asearch strategy to identify relevant publishedreviews.

Two databases were readily available which providea source of meta-analyses: the Database ofAbstracts of Reviews of Effects (DARE) [availablethrough a number of sources, including the Centrefor Reviews and Dissemination (CRD) websitehttp://www.york.ac.uk/inst/crd/darehp.htm and theCochrane Library] and the Cochrane Database ofSystematic Reviews (CDSR) (on the CochraneLibrary, available to all members of the NHSthrough the National Electronic Library forHealth, www.nelh.nhs.uk). It was felt appropriateto focus initially on a search of DARE. DARE is adatabase of quality-assessed systematic reviews.The reviews are identified through regularsearching of a number of electronic databases(including MEDLINE, CINAHL, CurrentContents Clinical Medicine and BIOSIS), byhandsearching key major journals and scanninggrey literature. To be included on DARE, thereviews undergo a rigorous quality assessment andmust meet set criteria with regard to the reviewquestion, the search strategy, validity assessment ofthe primary studies and the presentation andpooling of the primary studies. A hard copy of allreviews published on DARE (1994 to December1998) was obtained, and each review screenedaccording to the inclusion criteria (see below).

Inclusion criteriaAll systematic reviews including at least one meta-analysis were assessed to see whether they used:

1. RCTs2. indirect comparisons3. direct comparisons.

The following types of meta-analysis wererecorded:

� those incorporating elements 1 and 2, with thepossibility of comparing them with other studiesmaking direct comparisons of the sameinterventions. A note was made of whether theauthors interpreted the results as if directcomparisons had been made

� those incorporating all three elements, toexamine the differences in estimates of effectobtained using direct and indirect comparisons

� those incorporating elements 1 and 3, andproviding sufficient data to try to undertake anindirect comparison of specific interventions.

All other systematic reviews were excluded.

Assessment of relevanceAll systematic reviews listed on DARE wereassessed using a piloted prescreening form (seeAppendix 1) designed to allow for identification ofarticles meeting the above inclusion criteria andalso those for the Evaluating Non-RandomisedIntervention Studies (ENRIS) project.15 Tworeviewers independently assessed each article anddisagreements were resolved by discussion. Alldata were managed using Microsoft Excel 97.

Data extractionFor all reviews assessed as relevant, data wereextracted using a predefined data extraction form(Appendix 2). This process was undertakenindependently by two reviewers and discrepancieswere resolved through discussion. The results ofthe data extraction process were tabulated. Themethod used to carry out the indirect comparisonand the appropriateness of the interpretation ofresults were noted. The interpretation was deemedappropriate if the findings of direct comparisonswere given greater weight in the conclusions, orthe potential biases associated with the findings ofthe indirect comparisons were discussed.


7


Chapter 3

Indirect comparisons in published systematic reviews

ResultsIn total, 734 systematic reviews were available onDARE for screening in March 1999. Of these, 327included a meta-analysis of RCTs. The initialscreening identified ten reviews that were coded asbeing definite examples of indirect comparisonsand 46 reviews that were coded as possiblyincluding an indirect comparison. A further 12potentially relevant reviews were located throughthe electronic searches for the systematic review(Chapter 4).

After detailed assessment, 13 of these 68 reviewswere identified as including both direct andindirect comparisons of competing interventionsand 23 included indirect comparisons only. Thirty-one of the reviews were identified from the searchof DARE and five from the electronic searches.Detailed summaries of the 36 included reviewsappear in Appendix 3. The remainder of thereviews did not include suitable data and so wereexcluded (see Appendix 4).

Reviews including both direct andindirect comparisonsThirteen of the identified systematic reviews usedboth direct and indirect comparisons.16–28 Tenpresented adjusted indirect comparisons, firstestimating the pooled effects of each treatmentarm against placebo or a control group and thencomparing the two estimates. The interpretationof the results was appropriate in each case, withconclusions being based largely on the directcomparisons, or problems associated with indirectcomparisons being discussed.16–20,22,24,26–28 Theother three reviews presented naive, unadjusted,indirect comparisons.21,23,25 The interpretation ofthe results was classed as appropriate in two of thereviews, with emphasis given to results from directcomparisons when available.21,25 There was someuncertainty surrounding the appropriateness ofthe interpretation of the results presented in thethird review23 (Table 2).

Each of these reviews is discussed in the followingsections.

Adjusted indirect comparisons (see Appendix 3,Table 19)Adjusted indirect comparisons were considered tohave been used when estimates of effect werecompared for interventions that had beenevaluated using a common comparator (e.g.placebo). Three of the reviews undertaking anadjusted indirect comparison were conducted bythe Antiplatelet Trialists’ Collaboration andexamined the effects of antiplatelet therapy onvarious outcomes measures (including theprevention of death, myocardial infarction, stroke,venous thrombosis and pulmonary embolism) indifferent categories of patients.16–18 All threereviews were rigorous in their methodology. Thereviews all presented adjusted indirectcomparisons of aspirin plus dipyridamole andaspirin alone, using a common control group. Theresults for the indirect comparisons werepresented separately from those of the directcomparison, and used to enhance the findings ofthe reviews. Conclusions were drawn cautiously ineach review.

A fourth review comparing aspirin plusdipyridamole with aspirin alone was conducted byLowenthal and Buyse, examining the effectivenessof the drugs for the secondary prevention ofcerebrovascular accidents.19 The outcomes ofinterest were total mortality, vascular mortality,total strokes, fatal strokes, and a composite end-point consisting of vascular death or non-fatalstroke or non-fatal myocardial infarction(‘important vascular events’). Double-blind RCTscomparing aspirin with placebo, aspirin plusdipyridamole with placebo, or aspirin plusdipyridamole with aspirin alone were included inthe review. Adjusted indirect comparison wasmade using the placebo groups. Overall oddsratios for aspirin versus placebo were plottedalongside odds ratios for aspirin plus


8

TABLE 2 Reviews using both direct and indirect comparisons

No. of Appropriate interpretation of results Agreement between IC and DC studies (estimate of effect in same direction)

Yes No Uncertain Yes No Uncertain

Adjusted IC 10 10 – – 7 2 1Naive IC 3 2 – 1 1 1 1

DC, direct comparison; IC, indirect comparison.

dipyridamole versus placebo for all outcomes. Therisk reduction for each outcome was consistentlybetter for the combination of drugs whencompared with aspirin alone. Chi-squared testswith one degree of freedom were calculated to testwhether the differences observed could beascribed to chance alone, and a statisticallysignificant difference was demonstrated in favourof the combination for three of the five outcomesmeasured [important vascular events (riskreduction 18% versus 40%; �2 = 7.30, p = 0.007),all strokes (risk reduction 17% versus 42%;�2 = 7.15, p = 0.007) and fatal strokes (riskreduction –10% versus 43%; �2 = 4.60, p = 0.03)].No such statistically significant differences werenoted in the trials comparing the two treatmentregimens directly. The authors of the reviewinterpret the results with caution. They state thatthe results “suggest that the combination therapy ofaspirin with dipyridamole may be superior toaspirin alone”. However, they discuss that theresults from the indirect comparisons may reflectdifferences in selection criteria or otherconfounding factors, rather than a truly greatertreatment effect of combination therapy.

Zhang and Li Wan Po conducted a systematicreview to assess the efficacy of paracetamol and itscombination with codeine or caffeine incomparison to paracetamol alone.27 An adjustedindirect comparison was made by estimating thepooled effects of each treatment arm againstplacebo and then by comparing the two estimates.Second, a direct comparison was made betweenparacetamol–dextropropoxyphene andparacetamol using head-to-head trials (ignoringthe placebo group in the three-armed studies).The results were expressed as the difference inpercentage improvement of total pain relief(TOTPAR%) and the sum of pain intensitydifference. The proportions of patients obtainingmoderate to excellent pain relief relative toplacebo and the ratio of patients requiringanalgesic remedication were also estimated. Theresults of the indirect comparison were similar tothose of the head-to-head comparisons anddemonstrated an enhanced analgesic efficacywhen codeine (60 mg) was used in addition toparacetamol (600 mg) (using TOTPAR% as theoutcome measure).

The efficacy of paracetamol and its combinationwith codeine was also assessed by Moore andcolleagues.24 They discussed the results of theindirect comparison (between paracetamol pluscodeine and paracetamol alone, using placebo asthe common control) and direct comparison

separately, making no attempt to combine them.In both cases an increased response rate was notedfor the combination therapy, although the indirectcomparison gave a greater estimate of effect.

Li Wan Po and Zhang conducted a systematicreview of RCTs to evaluate the comparativeefficacy and tolerability ofparacetamol–dextropropoxyphene combinationand paracetamol alone.20 The main outcomemeasures used in this review were the sum ofdifference in pain intensity, the response rate ratioand difference in response rate; and the rate ratioand rate difference of side-effects. Theparacetamol–dextropropoxyphene combinationwas compared with paracetamol alone bothdirectly and indirectly (adjusted indirectcomparison), as in their previous review.27 Theresults of the indirect comparison were used tosupport the findings of the direct comparisons.The mean (95% CI) difference in the sum ofdifference in pain intensity, as illustrated by thedirect comparisons, was 7.3% (–0.2 to 14.9%)(fixed effect model), in favour of the combination.The results of the indirect comparisons wereconsistent with the head-to-head comparisons andthe authors concluded that on the evidence ofboth direct and indirect comparisons “there islittle objective evidence to support prescribing acombination of paracetamol anddextropropoxyphene in preference to paracetamolalone in moderate pain such as that after surgery”.

The benefits of a longer duration interferonregimen (3 MU three times per week for 12months) in comparison to the standard 6-monthregimen for patients with chronic hepatitis C wasevaluated by Poynard and colleagues.26 Theypresented results of an adjusted indirectcomparison, comparing the pooled odds ratios forthe different regimens in comparison to undefinedcontrols. For example, the odds ratio for thesustained alanine transaminase (ALT) versuscontrol at the end of the 12-month regimen was0.35 (95% CI 0.28 to 0.43). This was shown to begreater than for the 6-month regimen, whichproduced an OR of 0.21 (95% CI 0.13 to 0.28),although the difference was not statisticallysignificant (p = 0.06). Direct comparison of a 12-month (or more) regimen and a 6-month regimenshowed a statistically significant duration effect onthe sustained response rate at 3 MU, OR 0.16(95% CI 0.9 to 0.23) (p < 0.01) in favour of the12-month regimen.

Matchar and colleagues did not make anyreference to the use of indirect comparisons within


9


their review of medical treatments for strokeprevention, but the pooled relative risks forwarfarin and aspirin versus a control/placebogroup were compared, as were different doses ofaspirin versus placebo.22 The findings of theindirect comparisons were similar to, and used toreinforce, those of the direct comparison(although only one head-to-head study of warfarinversus aspirin was included).

Piccinelli and co-workers examined the efficacy ofdrug treatment in obsessive–compulsivedisorders.28 The review found considerabledifferences between the results from the indirectcomparisons and direct comparisons. The authorsrecognise that the increase in improvement ratewas greater for clomipramine than for selectiveserotonin reuptake inhibitors (SSRIs) whencompared with placebo, but highlight the fact thatdirect (head-to-head) comparisons showed similartherapeutic efficacy on obsessive–compulsivesymptoms. Possible reasons for discrepancies arecovered in Chapter 6.

Naive indirect comparisons (see Appendix 3,Table 20)Three other reviews included both direct andindirect comparisons, but did not use an adjustedindirect comparison.21,23,25 Marshall and Irvineundertook a systematic review to establish the roleof rectal corticosteroids in the management ofactive distal ulcerative colitis.21 They includedRCTs that assigned patients to two or moretreatment groups, with rectal corticosteroids in atleast one arm. Pooled response rates for each typeof corticosteroid and control therapy werecalculated across all trials. The pooled responserates for the conventional rectal corticosteroids,the topically active corticosteroids,aminosalicylates and placebo were compared.Conventional rectal and topically activecorticosteroids produced similar response rates forsymptoms, endoscopy, histology and remission.The aminosalicylates showed improvements in allresponse rates. The authors of the review sensiblyfocused on the results of direct comparisons intheir discussions, concluding that rectal 5-ASA (anaminosalicylate) is superior to rectalcorticosteroids in the management of distalulcerative colitis. Although the authors did notdraw heavily on the findings from the indirectcomparisons, the results of these are fairlysupportive of the conclusions.

Pope and colleagues also used a naive indirectcomparison technique to investigate thehypertensive effects of non-steroidal anti-

inflammatory drugs (NSAIDs) and ranked them bymagnitude of change in mean arterial pressure(MAP).25 They did this by extracting data fromeach NSAID treatment arm across all trials. Dataon possible confounders (including age, trialquality, dietary salt intake, hypertensive ornormotensive patients) were recorded andadjusted for in the calculation of the averagechange in MAP for each NSAID. The results of theindirect comparison are used greatly within thereview, but are compared with the results of head-to-head comparisons when available. The findingsfrom the direct and indirect comparisons were notalways consistent.

The results from indirect comparisons are notalways used cautiously. The comparativetolerability and rate of withdrawal from clinicaltrials of roxithromycin and erythromicin inpatients with lower respiratory tract infectionswere examined in a systematic review, again usingboth direct and indirect comparisons.23 ThreeRCTs included in the review were head-to-headcomparisons of roxithromycin and erythromicin.All other trials (n=22) compared eitherroxithromycin or erythromicin with anothermacrolide or other agent commonly used as firstline therapy for patients with lower respiratorytract infections. Summary statistics of reportedadverse events and withdrawals based on datafrom arms of all trials (both head-to-head andindirect comparisons) were calculated and used toformulate the review’s conclusions. Although theresults of the three head-to-head trials arepresented, summary statistics for these trials aloneare not calculated. The authors do discuss thepossibility of potential confounding factors, butconclude that there is no significant differencebetween groups in terms of clinical efficacy, age,gender, settings, duration of treatment, indicationsor year of publication.

Reviews using indirect comparisonsaloneOf the 23 reviews presenting results of indirectcomparisons only, 15 used an adjusted method ofcomparison,7–9,11,29–39 and eight performed anaive indirect comparison6,40–46 (Table 3).

Adjusted indirect comparisons (see Appendix 3,Table 21)Of the 15 studies including an adjusted indirectcomparison, only six of these interpreted theresults in an appropriate way.8,29–33 For example,Zalcberg and colleagues undertook a systematicreview of RCTs to determine the effect of 5-fluorouracil (5-Fu) dose in the adjuvant therapy


10

of colorectal cancer.29 The trials included in thereview compared a 5-FU-containing regimen witha no-chemotherapy control group. For each study,the observed and expected number of deaths ontreatment was calculated and an estimated oddsratio of mortality obtained for combined studiesusing a fixed effect model. Forest plots werepresented for trials comparing a 5-FU-containingregimen (separated into four categories: ≥ 10 g,between 8 and 10 g, <8 g and oral chemotherapy)with no-treatment controls, and also for trialscomparing 5-FU and levamisole or another 5-FUregimen with no-treatment controls. The authorswere cautious in their interpretation of the resultsstating that, owing to the fact that findings werebased on indirect comparisons, confounding bythe type of patient being studied in each trial is apossibility.

A review by the Homocysteine Lowering Trialists’Collaboration also used a no-treatment controlgroup as the common comparator to enableindirect comparisons to be made for differentdoses of folic acid.8 The aim of the review was todetermine the size of reduction in homocysteineconcentrations produced by dietarysupplementation with folic acid and with vitaminB12 or B6. Twelve trials were included in thereview, with individual data on 1114 patients. Aforest plot was used to illustrate the reductions inblood homocysteine concentrations with varyingdoses of folic acid. The findings suggest that awide range of doses (0.5–5 mg) is similarlyeffective. The authors of the review providedetailed implications for future research in thisarea.

In a comparison of the efficacy of home-administrated low molecular weight heparin(LMWH) in the treatment of deep vein thrombosis(DVT) with that of hospital-administered LMWH,Leizorovicz used an unfractionated heparin (UFH)group as the common comparator.30 Twosubgroups of studies were identified (LMWHadministered at home versus UFH and LMWHadministered at hospital versus UFH) and the

odds ratios for recurrent thromboembolic events,mortality and major haemorrhage were presentedin a forest plot. The authors report similar efficacyresults for the recurrence of thromboembolicevents or death, but acknowledge that thisapproach reduced the statistical power for eachsubgroup.

Pignon and co-workers conducted a meta-analysisusing individual patient data from RCTscomparing chemotherapy alone withchemotherapy combined with thoracicradiotherapy.31 They clearly acknowledge thatindirect comparisons were used to compare trialsof early versus late radiotherapy, and also trialswith or without sequential radiotherapy. Thecomparisons did not reveal any optimal time fortreatment. The authors conclude that in order toidentify the optimal combination of chemotherapyand radiotherapy, further trials, of directcomparisons, are required.

Similarly, a meta-analysis of individual patientdata to evaluate the effect of different cytotoxicchemotherapy regimens on patients with non-small cell lung cancer stated that from the trialsincluded it was not possible to recommend oneparticular regimen over another, and that “Furtherrandomised trials are needed to determine whichregimens are the most effective of the modernchemotherapies studied”.32

Rossouw examined angiographic trials to assess theoverall effects of lipid reduction on angiographicoutcomes and clinical events.33 The trials includedin the review compared various interventions(lifestyle changes, resins, statins, combinations ofdrugs, and surgery) with control groups. Theauthor did not discuss the possibility of biasoccurring due to the use of indirect comparisons.However, the review demonstrated no evidence ofa class effect, with all classes of interventionappearing to have beneficial effects on bothangiographic and cardiovascular outcomes.

A review of antiemetic drugs for the prevention ofvomiting following paediatric strabismus surgerywas conducted by Tramer and colleagues.11 Thereview included 24 RCTs that were placebocontrolled or no-treatment controlled, or includedan unspecified control group. The drugsexamined were droperidol (varying doses),metoclopramide (varying doses), dixyrazine,ondansetron, lignocaine, hyoscine, atropine,lorazepam and propofol. Only three of the drugs(droperidol, metoclopramide and propofol) wereused to draw conclusions, owing to insufficient


11


TABLE 3 Reviews using indirect comparisons alone

No. of Appropriate interpretation studies of results

Yes No Unclear

Adjusted IC 15 6 3 6Naive IC 8 – 7 1

data for the remaining drugs. Adjusted indirectcomparisons were made for these three drugs,with odds ratios and 95% confidence intervalsbeing calculated using a fixed effects model, andNNT also calculated. The findings for droperidolsuggested a dose–response relationship(10–75 �g kg–1), with 75 �g kg–1 being the onlydose to have an odds ratio greater than one. Theauthors do interpret the result with caution;however, this caution is mainly due to the smallsample sizes being examined and not the potentialbiases that can occur through indirect comparisons.

The appropriateness of the conclusions is unclearin five other systematic reviews also presentingadjusted indirect comparisons. For example, Kochand colleagues34 conducted a review of H2 blockersand misoprostol. The trials included had to have aplacebo arm to act as the common comparator. Therate difference for each active drug in comparisonto placebo was calculated, as were NNTs. Forestplots were presented. The authors claimed that“gastric ulcer was found to be significantly reducedby misoprostol – both in short-term and long-termNSAID treatment – but not by H2 blockers”. In thediscussion, the authors did not explicitly discussbiases that can occur through indirectcomparisons, but they did highlight the fact thatthere were differences in the characteristics ofpatients studied in the misoprostol and H2-blockertrials (those included in the misoprostol trials wereat higher risk of gastric ulcer).

Holme reviewed the association between totalmortality outcome or coronary artery disease(CAD) incidence and the amount of cholesterolreduction in randomised cholesterol-loweringtrials that were performed before and afterinclusion of statin trials.35 An adjusted indirectcomparison was undertaken comparing diet,statins and hormones. The common comparatorwas fibrates. Multiple regression analysis wasundertaken, adjusted by cholesterol changes andbaseline risk of CAD. The statistical analyses weredone by weighted multiple linear regressionmodels with a fixed effects variance assumption.Diet versus fibrates gave an odds ratio of 0.975 fortotal mortality and 1.07 (p < 0.05) for CADincidence. Odds ratios of 1.088 and 1.185(p < 0.05) were obtained for hormones versusfibrates, and statins versus fibrates showed an oddsratio of 0.833 (p < 0.05) and 0.875 (p < 0.05) fortotal mortality and CAD incidence, respectively.The authors concluded that “Fibrate trials as agroup had the least favourable outcome profilesfor CAD and all cause mortality of all other drugtrials (except hormones)”.

Garg and Yusuf7 conducted a systematic reviewevaluating the effect of ACE inhibitors on mortalityand morbidity in patients with symptomaticcongestive heart failure. Thirty-two placebo-controlled trials of ACE inhibitors, including 7105patients, were used to make adjusted indirectcomparisons of estimates of effect for the differentagents evaluated. Trials making head-to-headcomparisons of different ACE inhibitors wereexcluded from the review unless they also includeda placebo group. The authors state that “Similarbenefits were observed with several different ACEinhibitors, although the data were largely based onenalapril maleate, captopril, ramipril, quinaprilhydrochloride, and lisinopril”.

A systematic review to compare the effectivenessand safety of oral tramadol with standardanalgesics using a meta-analysis of individualpatients’ data included 3453 postoperativepatients.36 Tramadol (50, 75, 100, 150 and 200 mg), codeine (60 mg), aspirin (650 mg) pluscodeine (60 mg) and acetaminophen (650 mg)plus proxyphene (100 mg) were all compared withplacebo, and then adjusted indirect comparisonsmade. Relative risks and NNT were presented with95% confidence intervals, and illustratedgraphically. Again, the authors do not discuss thepotential biases associated with indirectcomparisons. Head-to-head comparisons werepossible in certain cases, although data were notpresented.

The sixth study for which it was unclear whetherthe interpretation of the results was appropriate ornot was conducted by Lefering and Neugebauer.39

They compared studies of low-dose corticosteroidversus control with studies of high-dosecorticosteroid versus control. The outcomeassessed was mortality. The control groups wereunspecified, and there was wide variation in themortality rates among the control groups (7–69%).Pooled rate differences were calculated for the low-dose and high-dose studies, and the resultscompared. Low-dose corticosteroids showed aneffect of –1.9% (95% CI –20.0 to 16.2%), whilehigh-dose corticosteroid trials had a pooled effectof 3.6% (95% CI –2.5 to 9.8%). The findings wereinconclusive and the authors state that “Neitherthe type of steroid used nor the separation intolow-dose or high-dose regimen indicated aremarkable difference between the steroid groupand control group.” No discussion about the roleof indirect comparisons was presented.

Boersma and co-workers examined earlythrombolytic treatment in acute myocardial


12

infarction and used indirect comparisons toexamine the relationship between time totreatment, from onset of symptoms, and mortalityup to 35 days.9 Odds ratios were calculated andpresented graphically. Both linear and non-linearregression analyses were undertaken. The authorsconclude that the beneficial effect of fibrinolytictherapy is substantially higher in patientspresenting with 2 hours after symptom onsetcompared with those presenting later. They didnot mention the potential problems of indirectcomparisons. However, a previous meta-analysis ofthe same studies argued that “if patient categoriescan be arranged in some meaningful order then… may be reasonably reliably informative …”.47

The reviews illustrate the usefulness of indirectcomparisons when there is a lack of directcomparison information.

The purpose of the systematic review conductedby Aro37 was to compare the cardiovascular andall-cause mortality of two oestrogen regimens[polyestradiol phosphate (PEP) alone, or incombination with oral ethinyl estradiol (EE)] andorchidectomy with those of the Finnish malepopulation. The review also compared the age-standardised mortality for all three treatmentforms. Only two trials were included in the review,Finnprostate I (PEP plus EE versus orchidectomy)and Finnprostate II (PEP versus orchidectomy).Age-specific person-years at risk were computedfor each treatment group at 5-year intervals. Theauthors concluded that “intramuscular PEPmonotherapy is associated with low cardiovascularmortality and with an all-cause and prostaticcancer mortality equal to orchidectomy”. Theresults and conclusions of the review arequestioned in an article by Ekbom and Taube,48

who highlight the fact that the two RCTs wereconducted in different years and with differentinclusion criteria. Indeed, there were statisticallysignificant differences between patientcharacteristics in the two trials. Ekbom and Taubesuggested that the observed low mortality in PEPalone may be due to “flaws in the methodology”.48

The objective of the meta-analysis carried out byPoynard and colleagues was to comparelansoprazole with raniditine or famotidine in acuteduodenal ulcer, and to compare indirectlylansoprazole with other drugs (omeprazole,nizatidine, cimetidine and sucralfate).38 Raniditineor famotidine was used as a common comparator.Four-week healing rates (OR, 95% CI) werecalculated for each drug in comparison to thecommon comparator, and ranked according toefficacy. The authors mention an RCT directly

comparing lansoprazole versus omeprazole inacute duodenal ulceration; however, this trial isnot included in the meta-analysis.

Naive indirect comparisons (see Appendix 3,Table 22)Eight studies, presenting results of indirectcomparisons alone, used the naive approach.6,40–46

None of the studies presented results from directcomparisons (even if this were possible), andpotential biases associated with indirectcomparisons were rarely discussed.

Coulter and colleagues41 reviewed RCTs of drugsused to treat menorrhagia. A variety of NSAIDs,antifibrinolytics, hormones and intrauterinedevices was examined. The drugs were listedaccording to the percentage reduction inmenstrual blood loss. The review did not presentthe data from the head-to-head comparisons madewithin some of the RCTs. Half of the trialsincluded a placebo group and could have beenused to carry out an adjusted indirect comparison,but the results of the placebo controls were notreported.

The comparative efficacy and toxicity of secondline drugs in rheumatoid arthritis was examinedby Felson and colleagues.6 For each trial, thetreatment arms of interest were identified anddata extracted. To compare the drugs they usedanalysis of variance, weighted by treated groupsize, multiplied by study quality and adjusted forthose covariates that had a significant associationwith each outcome (tender joint count, erythrocytesedimentation rate and grip strength). For eachoutcome auranofin was shown to be weaker thanthe other second line drugs (methotrexate,injectable gold, D-penicillamine, sulfasalazine andantimalarial drugs). It is unclear whether theadjustment for covariates actually reduced orincreased biases. Within-study comparisons were ignored, as were studies comparing drugsdirectly.

The review by Felson and colleagues6 was quotedby Imperiale and Speroff45 in support of usingnaive indirect comparisons of RCT data. Theyundertook a meta-analysis to examine the efficacyof thromboprophylaxis following total hipreplacement. The naive indirect comparison usedin the review was based on the premise that thetreatment groups were clinically homogeneous incomposition. The authors give the impression thattheir conclusions were based on evidence fromRCTs, even though only between-studycomparisons were made.


13


Bansal and Beto42 compared the efficacy oftherapeutic agents used in the treatment of lupusnephritis using outcomes of end-stage renaldisease (ESRD) and total mortality, using the samemethod of indirect comparison as Imperiale andSperoff.45 They undertook a simple indirectcomparison of results of different arms across thestudies included in the review. Theappropriateness of pooling the data was examinedby two methods (z-score and heterogeneity test),but the results of the tests were not presented. Thereview included RCTs and quasi-RCTs, but theresults of the studies were used to make between-study comparisons only, therefore losing the powerand rigour of the randomisation process. Again,the authors do not mention the potentialproblems associated with such indirectcomparisons, and no evidence from directcomparisons is provided.

Similar problems occur in a review ofantihypertensive agents to reduce left ventricularhypertrophy.43 The study had strict inclusioncriteria in that only double-blind RCTs wereincluded. However, analysis was undertaken bycombining all treatment arms of the same drugclass and weighting them according to the numberof patients in each individual study, thus breakingthe randomisation procedure. The authors dodiscuss the importance of randomisation and thevalidity of studies, although this is not taken intoaccount in the analysis. Within-study comparisonswere ignored and no attempt was made to carryout an adjusted indirect comparison using acommon placebo group.

Unge and Berstad44 studied anti-Helicobacter pyloriregimens. They included both RCTs andobservational data, and pooled data into groupsaccording to the combination of drugs used,regardless of, for example, dosage or duration.The authors argued that “a formal meta-analysis isof limited value due to the substantial variation oftherapeutic options”. There was no discussion ofpotential biases, and direct comparisons were notincluded in the review.

Chiba and colleagues40 conducted a meta-analysisto evaluate the speed of healing and symptomrelief in grade II–IV gastrooesophageal refluxdisease (GORD). The review included onlyrandomised trials, but did not directly comparedifferent interventions. Data from the includedstudies were grouped by drug class, decided apriori to be placebo, proton pump inhibitors(PPIs) and H2-receptor antagonists (H2RAs). Foreach study arm the overall healing proportion

reported at the final evaluation time-point wasused to calculate the overall healing proportion to12 weeks. Data were pooled within each drug classirrespective of dose, duration of treatment orspecific drug. Groups were then compared usinganalysis of variance (no further details given).Within-study comparisons were not made,although sufficient data to make directcomparisons were presented (see Chapter 6 forfurther details).

A review comparing the antihypertensive efficacyof available drugs in the angiotensin II antagonist(AIIA) class included 43 trials.46 Most of the trialswere placebo controlled, although some werehead-to-head trials. Analysis was based ontreatment arms, regardless of which othertreatments were included in the trials. Theestimate of blood pressure reduction assessedwithin the review is likely to be overestimatedbecause of regression to the mean effect, as theplacebo group (or other comparators) wasignored. The review did not include anyrecognition of the weakness of this approach.

SummaryIndirect comparisons are commonly used forevaluating the relative effectiveness of alternativeinterventions. Approximately 9.5% (31/327) ofmeta-analyses of RCTs identified through DAREincluded some form of indirect comparison. Afurther five reviews including some form ofindirect comparison were identified. The majorityof the reviews included in this chapter werepublished before 1998, owing to the nature of thesearching. An update search was not undertakenas it was felt the sample of reviews was sufficientlylarge. In addition, there was no reason to supposethat the frequency with which indirectcomparisons are used in meta-analyses has altered.

The methods used for the indirect comparisonsincluded the naive indirect comparison (11/36,31%) and the adjusted indirect comparison (25/36,69%). Although the identified meta-analyses oftenincluded only randomised trials, the strength ofthe randomisation procedure is completely brokenwhen making naive or unadjusted indirectcomparisons, thus providing data that are perhapsequivalent or even inferior to those obtainedthrough non-randomised studies. Such indirectcomparisons may be subject to bias (especiallyselection bias) compared with head-to-headrandomised comparisons as the benefit ofrandomisation does not hold across trials. In 50%


14

(18/36) of the systematic reviews examined therewas no mention of the potential biases associatedwith the findings of the indirect comparisons.

Results obtained through indirect comparisonwere not always consistent with the findingsobtained by direct comparisons. Thirteen (36%) ofthe identified systematic reviews used both directand indirect comparisons. The results of directand indirect comparisons within these 13 reviewswere different in three meta-analyses and similar(same direction but not necessarily magnitude ofeffect) in eight meta-analyses. Because data fromthe same trials have often been used in both directand indirect comparisons in a review, the

difference between the direct and indirectestimates may have been underestimated in thereviews.

The indirect comparisons were sometimes carriedout implicitly and the results of indirectcomparisons interpreted as if from directcomparisons within randomised trials. Further, the findings of direct comparisons were sometimes ignored, even when data wereavailable. The misuse of indirect methods andinappropriate interpretation of results of indirectcomparison may result in misleading assessmentsof relative efficacy of competing healthcareinterventions.


15


Systematic review of the literatureA systematic review of the research literature wasundertaken to identify and evaluate statisticalapproaches to the analysis of indirectcomparisons. The CRD guidelines were used as astarting point for the systematic review protocol.49

However, as the review was a methodologicalreview rather than a review of the effectiveness ofan intervention, it was noted that these guidelineswould not be strictly applicable and would needadapting. In particular, the review was notrestricted to consideration of publicationsidentified by the formal searching.

Search strategyA thorough search was undertaken, including bothcomputerised and manual searching, to identifyrelevant literature. It was recognised that relevantmaterial may also be published in textbooks.Constructing a literature search strategy formethodological papers is problematic owing to thelack of suitable indexing terms in the electronicdatabases. The development of any search strategyis essentially an iterative process whereby theinitial strategy is refined and developed accordingto its recall and precision. The development of thesearch strategy in this instance involved five roundsof searching MEDLINE, reviewing the results andadapting the search strategy. The MEDLINE searchwas run from 1966 to March 1999. It was updatedto include records published by February 2001.Details of this process are given in Appendix 5.

Search strategy 5 (Appendix 5) was used as a basisfrom which to develop strategies to use in otherdatabases. Amendments were made regardingthesaurus terms and subject indexing whereappropriate. In general, the strategies used reliedheavily on the use of free text terms because of alack of adequate MeSH or other thesaurus termsto describe the concepts of research methodology.The strategies for PsycLIT (1887 to February2001), EMBASE (1980 to April 1999), ERIC (1966to February 1999) and MathSCI (1940 toSeptember 1999) are listed in Appendix 6.

The project team considered that it might beuseful to carry out searches of some of the

databases covering the agricultural literature.Potential databases to search were identified asbeing BIOSIS, Agricola, Agris International andCAB Abstracts. Some test search strategies wererun on the BIOSIS database, but the recordsretrieved were studies reporting the results ofsystematic reviews rather than articles discussingmethodological issues. It was decided not topursue this source further.

In addition to the electronic searches, thefollowing key journals were initially handsearched:

� Statistics in Medicine (1984–1999)� Controlled Clinical Trials (1984–1999)� Journal of Clinical Epidemiology (1991–1999)� Psychological Bulletin (1995–1999)� Psychological Methods (1995–1999).

The searches of Statistics in Medicine, ControlledClinical Trials and the Journal of ClinicalEpidemiology were subsequently updated to July2004. The reference lists of all relevant articleswere examined to identify further studies andattempts made to uncover grey and unpublishedliterature. Contact was also made with thoseworking in the field, both nationally andinternationally, including the Cochrane MethodsGroups for Empirical Methodological Studies (nowthe Cochrane Methodology Review Group),Statistics, and Individual Patient Data Meta-analyses. Papers identified ad hoc were alsoincluded.

Finally, given the difficulty in identifying relevantarticles through electronic searching, an updatesearch was conducted in the form of a citationsearch run on all of the relevant papers previouslyidentified (April 2004).

Inclusion criteriaThe searches were used to identify all papers thataddressed the following issues:

� methodology for carrying out indirectcomparisons

� methodology for identifying and assessingbiases that arise from indirect comparisons

� methodology for avoiding (or even adjustingfor) biases arising from indirect comparisons.


17


Chapter 4

Statistical methods for indirect comparisons

To provide examples for the empirical work (seeChapter 5), any reviews meeting the followingcriteria were also identified:

� systematic reviews reporting both direct andindirect evidence

� examples of systematic reviews where treatmentarms from different RCTs are compared (i.e.randomisation has been broken).

Assessment of relevanceAll titles and abstracts of articles identifiedthrough the searches were screened independentlyby two reviewers. All papers considered to berelevant by at least one reviewer were retrieved forfurther appraisal. Retrieved articles were againassessed independently by two reviewers and anydisagreements taken to a third party.

Results of searchesIn total, 3034 titles and abstracts were identifiedthrough the electronic searches. Followingscreening 29 full text articles were obtained, andfurther screening resulted in the identification ofonly six papers for inclusion.14,50–54 The majorityof papers included in this chapter, including twobook chapters,55,56 were identified on an ad hocbasis.

As noted below, the problem of indirectcomparisons is closely related to other problemsfor which articles were not specifically beingsought: meta-regression, subgroup analysis, andactive–control equivalence trials. Although thesearch certainly did not detect all relevantpublications, it is probable that all of the mainmethods of analysis were covered.

Statistical methods for indirectcomparisonsBackgroundAs noted in Chapter 1, an indirect comparisoninvolves the comparison of the results from sets ofstudies making different treatment comparisons. Itis thus a combination of two (or more) meta-analyses and thus shares all the methodologicaldifficulties associated with the use of meta-analysisto combine the data from several studies. Multiplestudies may vary in numerous ways including, butnot restricted to, the precise interventions beingcompared, characteristics of participants,methodological quality (including aspects oftreatment allocation and blinding), concomitantinterventions, length of follow-up, outcomemeasures and amount of loss to follow-up. In

addition to concerns about the comparability oftrials making the same comparison, thecomparability of different sets of trials must alsobe considered. Two further issues are the choice ofsummary outcome measure57 and theheterogeneity of the results of the trials, issues thatmay be related.

The statistical methods for carrying out anindirect comparison can be derived from methodsfor investigating heterogeneity in a meta-analysis.In meta-analysis it is common to examineseparately subgroups of trials, for example,defined by the clinical or demographiccharacteristics of participants. Such analysis mayserve as a means of exploring and possiblyexplaining statistical heterogeneity of results.58–60

Subgroups may also be defined by characteristicsof one of the treatments. For example, trials mayhave used two or more different doses of an activetreatment, or members of a class of drugs, or mayhave compared a single treatment against differenttypes of standard or inert treatment. Although it iscommon to perform separate meta-analyses foreach subgroup, a formal approach requires acomparison of the treatment effects in each subsetof trials, which assesses the treatment by subgroupinteraction. When subgroups define differenttreatments the analysis is exactly the same as anindirect comparison. Comparison of twoindependent estimates is a standard statisticalanalysis, yielding an estimate of the comparativeeffect in the subgroups, with a confidence interval,and a p-value. The calculation is slightly morecomplex when the analyses have been of relativemeasures (odds ratio, relative risk) and thus on thelog scale.61 Alternatively, a regression model canbe used to examine whether heterogeneity isexplained by one or more study characteristics,known as meta-regression.60,62 The adjustedindirect comparison can thus be seen to be thesimplest form of meta-regression, with a singlebinary trial factor.

Indirect comparisons are inherent in the use ofactive–control equivalence drug trials, in which theaim is to demonstrate that a new active drug isequivalent to (i.e. not very different in efficacyfrom) an already available active drug, which itselfhas been shown to be superior to placebo.63,64

Although indirect comparisons can arise in thesedifferent contexts, the statistical analysis optionsare the same whichever scenario applies. Theinitial focus is on the simplest, and common, casein which results are available from one or moreRCTs comparing A versus B (AvB) and one or


18

more RCTs comparing AvC, and the aim is toestimate the difference in effect of BvC. Thefollowing five sections consider different statisticalapproaches that have been suggested for indirectcomparisons. First, statistical methods usingaggregate data are discussed for each study,mainly simple two-step methods. Second,modelling approaches based on individual patientdata (IPD) are considered. Although full IPD arerarely available, in the case of binary outcomes thefrequencies in a 2 × 2 table are effectivelyindividual observations and thus such studies areamenable to a wide range of possible analysismethods. In these first two sections the emphasisis on classical frequentist methods that are in wideuse and do not require specialist software. Third,Bayesian and likelihood-based methods areconsidered. Fourth, the assumptions that underlieall indirect comparisons are explored. Finally,various extensions of the approach, including thecombination of direct and indirect evidence in asingle analysis, are discussed.

The deeply flawed ‘naive method’, in which therandomisation is broken (described in Chapter 1)is not considered in this chapter, although itsperformance is evaluated in Appendix 7 and inChapter 5.

Classical methods using aggregate dataMeta-analysis using summarised (or aggregate)data extracted from published studies is a two-stage process involving the extraction orcalculation of an appropriate summary statistic foreach of a set of studies, followed by the weightedcombination of these statistics to provide anoverall estimate of effect (i.e. the contrast betweentwo treatments). Many familiar methods of meta-analysis are of this type, includingMantel–Haenszel methods for binary data and thegeneric inverse variance method, used for anytype of data.60

For an indirect comparison these ideas areeffectively extended to a three-step approach, inwhich the third step is to combine the results oftwo separate meta-analyses into an overallcomparison. A standard statistical result is that thevariance of the difference between twoindependent estimates is the sum of the twovariances (the variance is the square of thestandard error).65,66 This relation underlies thetwo-sample t-test, for example. It also applies toan indirect comparison, as the two sets of data arefrom different studies. Thus, given two estimatedeffects �AB and �AC for comparisons of AvB andAvC, respectively, the effect for the comparison

BvC is estimated as �BC = �AB – �AC, and var(�BC)= var(�AB) + var(�AC). A 95% confidence intervalfor �BC is obtained as �BC ± 1.96√[var(�BC)]. Theestimates of effect, denoted �, relate to the scaleon which the data would be analysed; examplesare risk difference, log risk ratio and log oddsratio for binary data, means for continuous data,and log hazard ratio for time-to-event data. Themethod in which two separate meta-analyses arecombined is referred to as an adjusted indirectcomparison. The adjustment here is for a commoncomparison group. The possibility of furtheradjustment for covariates is discussed later.

Because the basic method relies on a standardstatistical result, it is not possible to say who firstapplied it in the context of indirect comparison.The earliest methodological discussion found hereis by Eddy and colleagues in 1992,67 but earlierapplied examples are likely to exist, especially inthe framework of comparing subgroups of trials ina meta-analysis.

The first methodological article explicitlydiscussing adjusted indirect comparisons seems tobe that of Bucher and co-workers.14 For trials witha binary outcome they suggested combining oddsratios from separate meta-analyses, ORAB andORAC, so that logORBC is estimated as logORAB –logORAC, and its variance as var(log ORBC) =var(logORAB) + var(logORAC). From thesecalculations it is simple to obtain a confidenceinterval for logORBC and hence, by transformation,an estimate of ORBC with a confidence interval.The adjusted indirect comparison method is quitegeneral, and this formulation is clearly a specificexample of the general method described above.The approach has been used to combine trialsusing other effect measures, such as risk ratios,68

risk differences,68 hazard ratios or means.69

Fisher and colleagues discussed the use of adjustedindirect comparisons to address the specificquestion of estimating the superiority of a drug toplacebo when no placebo-controlled trials havebeen done.70 This situation arises when one drughas been shown to be effective and it becomesunethical to conduct further placebo-controlledtrials of new agents. Baker and Kramer71 presentthe same idea from a conceptual perspective. Allof their examples are hypothetical and they do notdiscuss the specifics of analysis.

In a meta-regression analysis, the estimatedtreatment effect is modelled as a function of oneor more study characteristics as predictorvariables.59,62 Least squares regression or a


19


maximum likelihood method is used and eacheffect estimate is weighted by the reciprocal of itsvariance. For a single binary study factor, thisanalysis is exactly the same as an adjusted indirectcomparison. The method can be used to compareresults from trials comparing AvB and AvC wherethe study characteristic is the choice ofcomparator. The estimated effect comparing BvCis the coefficient for the indicator variabledenoting which comparison was made.

All of the authors cited used a fixed effect meta-analysis to combine the results of trials of the samecomparison, but the same method can be applied tocombine estimates from two random effects meta-analyses. Meta-regression can also be performedallowing estimation of a random effect to describeresidual heterogeneity. Only Fisher and co-workersdiscussed this possibility, and they gave variousreasons for preferring a fixed effect meta-analysis.70

In all meta-analyses a choice has to be madebetween a fixed effect or random effects analysis; itis not specific to indirect comparisons or meta-regression. However, in this more complex situationthere is more than one way to implement a randomeffects analysis. In a random effects analysis thedifferent trials are assumed to be estimatingdifferent, but related quantities that are distributedaround some typical value with a variance that isestimated from the data (�2). In an indirectcomparison, for some models �2 is set the same foreach component comparison of two treatments,while for other models it is estimated separately.

Methods using generalised linear(mixed) modelsThe previous section described the analysis of anadjusted indirect comparison in terms of weightedcombinations of meta-analyses of sets of trialsmaking different treatment comparisons. Anindirect comparison can also be analysed in aregression framework using generalised linearmodels, but this approach requires the availabilityof IPD.

For a binary outcome, the 2 × 2 frequency tablefor a trial effectively reflects IPD and so such datacan be analysed using logistic regression, asdiscussed by Hasselblad.52

The same principles apply to meta-analyses ofstudies with continuous or time-to-event outcomes,but IPD will need to be obtained from the authors,which is generally a major undertaking.72

Statistical methods for these outcomes have beenconsidered by Higgins and colleagues73 and TudurSmith and colleagues.74

For binary and other outcomes, random effectsanalysis requires fitting generalised linear mixedmodels (GLMMs),75 which can be done forexample in Stata (Release 8.0; Stata Corporation,Texas, USA, 2003) using the program gllamm, orin SAS (version 9.1) using PROC MIXED (for acontinuous outcome) or PROC NLMIXED (for abinary outcome).

These general methods can be used to handleboth simple indirect comparisons and morecomplex networks of treatment comparisons, asdiscussed below (section ‘Extensions to morecomplex situation’, p. 22).

Bayesian and likelihood-based methodsSome authors have presented flexible Bayesianmethods that can be used to analyse indirectcomparisons as well as more complex datastructures. Higgins and Whitehead53 used aBayesian approach to investigate the relativeeffectiveness of �-blockers and sclerotherapy toprevent bleeding in patients with cirrhosis.Twenty-four trials had compared one of the activetreatments against control, and two trials hadthree arms.

The confidence profile method67 is a very generalmethod for combining almost any evidencerelating to a particular question. As well asincorporating studies making different treatmentcomparisons it can encompass different designs,outcomes and measures of effect, and allowsexplicit modelling of biases. Although oftenconsidered as a fully Bayesian model, it can alsobe formulated without prior distributions andfitted using maximum likelihood.

Ades76 recently proposed a Bayesian Markov chainMonte Carlo method to handle the general case.His method of multiparameter evidence synthesisis also very general in allowing other types ofevidence to be included in the model. Earlier,Dominici and colleagues51 presented ahierarchical Bayes grouped random effects model.They used Markov chain Monte Carlo simulationto apply this approach to a highly complex set of46 trials evaluating treatments for migraine fromthree classes: �-blockers, calcium channel blockersand biofeedback therapy.

Sutton and colleagues77 give an overview of theadvantages and disadvantages of Bayesianmethods in meta-analysis. One advantage of fullyBayesian methods is that they place a distributionon the heterogeneity �2 and do not assume thatthe estimate of �2 is without error.78 This issue is


20

especially relevant when there are rather few trialsmaking a particular comparison.

Fully Bayesian models offer the greatest flexibility,particularly in modelling random effects. Theyrequire specialist software and a deep statisticalunderstanding, taking them beyond the scope ofmany research groups. Attention in this study isfocused on simpler approaches, which can be usedfor many of the more common scenarios, but thereviewers indicate later where complex methodsare needed.

Assumptions Many study and patient factors (case-mix)influence the outcome of patients in a specifictreatment group in a particular trial. The essenceof the indirect comparison is that the effect for thecomparison BvC is estimated using a commoncomparator A by contrasting the estimated effectsfor comparisons of AvB and AvC. Implicit here isthe notion that the variation in the observedresults for patients treated with the commoncomparator, treatment A, will (to some extent)account for differences between studies inmethodology, case-mix, and so on. The validity ofthe indirect comparison will thus depend on theextent to which this assumption is reasonable.Adjustment for a common comparator (A) will beexpected to reduce, if not remove bias in thecomparison of B and C.

The underlying assumption of a fixed effect meta-analysis is that the various studies are allestimating the same effect, such as the effect of Arelative to that of B. The same considerationsapply to trials comparing AvC. An indirectcomparison shares with other observationalepidemiological investigations the risk of bias byconfounding.62 The key additional assumption ofan indirect comparison using the results of trialsof AvB and AvC is that there should be noimportant differences between the two sets of trialswith respect to aspects that could influence (bias)the estimated treatment effect of BvC, that is,there is no confounding of the comparison bysome trial characteristic. As an example, Lim andcolleagues79 reported a comparison of low- andmedium-dose aspirin therapy after coronarysurgery using an indirect comparison of placebo-controlled trials, with outcome evaluated byangiography. There were three randomised trialsof low-dose aspirin, with patients receivingangiography at an average of 10, 130 and 180days, and two trials of medium-dose aspirin, withangiography at an average of 363 and 367 days.Subsequent correspondence80 considered the

possible relation between the time to angiographyand the observed effect of aspirin.

Another formulation of this argument is that thetwo sets of trials should be exchangeable, in thesense that there is no reason to suppose that theresults as a whole would be different had thevarious trialists kept the same protocol andpatients, but chosen to study a different treatmentcomparison.81 Baker and Kramer71 showgraphically that the validity of the indirectcomparison depends on the consistency of thetreatment effect over settings with different eventrates (different case-mix). Phillips82 consideredsome ways in which this assumption might fail.Clearly, one important way in whichexchangeability could fail is when the treatmenteffect is influenced by some factor that itself variesacross the different treatment comparisons, suchas the clinical setting or length of follow-up.Adjustment for study covariates can help to reducesuch an effect and make exchangeability morelikely. The analysis is only possible when the factorvaries within each set of trials.

Similar issues have been discussed in the contextof non-inferiority trials, in which a new treatmentis compared with a standard treatment with theaim of showing that the new treatment is no lesseffective than the standard treatment, within someprespecified margin (these trials are also calledactive–control equivalence trials).64,83,84 Often thestandard treatment will previously have beenshown to be better than placebo. In essence, thereis an implicit indirect comparison when inferringfrom a trial that shows non-inferiority that the newtreatment is better than placebo. It is recognisedthat the validity of such an inference relies onaspects of the non-inferiority trial being closelysimilar to the prior placebo-controlled trials of thestandard treatment. Particular examples includepatient characteristics, treatment dose, outcomemeasures and length of follow-up.85

In addition to concerns about the averagetreatment effect, the issue of heterogeneity has tobe considered. Hirotsu and Yamada54 presented amethod that is equivalent to adjusted indirectcomparison using inverse variance meta-analysis.They suggested testing for homogeneity of trialresults both within treatment comparison andbetween direct and indirect estimates beforepooling.54 However, an indirect comparison doesnot require homogeneity, and modellingapproaches exist that include random effects termsthat estimate the degree of heterogeneity in thecomparisons. The model of Higgins and


21


Whitehead assumes that the degree of heterogeneityis similar in all comparisons,53 whereas othermethods allow separate estimates of heterogeneityfor each comparison. Further, in most situationsthere are too few trials for each paired comparisonto allow reliable assessment of whether there isexcess heterogeneity, or estimation of separaterandom effects for each component comparison.

As with any meta-analysis, there is the possibilityof fitting random effects models to take account ofbetween-trial heterogeneity. Under a randomeffects model the treatment effect is allowed tovary among trials making the same treatmentcomparison (usually assuming that the distributionof effects is normal). Models that apply therandom effect to each study arm rather than eachcomparison between arms are not sensible. Inmany situations it may be reasonable to assumethat the variability of the treatment effects is thesame for all paired treatment comparisons.53

Several of the papers describing the analysis ofindirect comparisons have given rather littleconsideration to the underlying assumptions, andthose statements that have been made are oftenquestionable. For example, the only assumptionmentioned by Hasselblad and Kong68 is that thepopulations of the individual studies contain somesubpopulation in common. While this is sensible,and echoes concerns about equivalence trials, it isby no means a sufficient requirement. Bucher andcolleagues14 noted that “The only requirement isthat the magnitude of the treatment effect isconstant across differences in the populations’baseline characteristics”. In their appendix theymentioned the additional assumption of notreatment by study interaction in each set of trials,but as noted above, the requirement is rather oneof equal heterogeneity.

The following section considers various morecomplex situations involving multiple comparisonsof multiple treatments. It is clear that theassumptions become more numerous and tenuousin relation to increasing complexity of the datastructure. Most obviously, the notion ofexchangeability may need to extend across manysets of trials (which may have been carried outacross a long period).

Lastly, in a meta-analysis of trials with a binaryoutcome, it is well known that the choice of effectmeasure may have a considerable impact on theanalysis, and also on the degree of observedheterogeneity. Empirical studies show that forbinary outcomes measures of relative effect (odds

ratios and relative risk) are more likely to beconsistent across trials than measures of absoluteeffect (risk difference).65,86 This issue is of majorrelevance to indirect comparisons of two or moresets of trials. None of the papers cited aboveconsidered the impact of the choice of effectmeasure in making indirect comparisons, with theodds ratio used in almost all cases. Anotherassumption of an indirect comparison meta-analysis, therefore, is that the effect measure isappropriate.

Extensions to more complex situationsThis section considers six ways in which theavailable data may be more complex than thesimple case outlined above. First, three furthersituations are considered where all trials still havetwo treatment arms, and then two cases where atleast one trial has more than two arms. Lastly, thecase where study or patient characteristics aretaken into account is considered.

More than one common comparison treatment(e.g. AvB, AvC, DvB and DvC)All methods for adjusted indirect comparisonsextend simply to the case where there are two ormore sets of indirect evidence for the comparisonof interest, such as comparing B and C using datafrom trials comparing AvB and AvC and also trialscomparing DvB and DvC. Separate estimates ofBvC from adjusted indirect comparisons using twodifferent sets of two-arm trials can be combinedusing inverse variance weighting. The analysis ofsuch a set of trials is illustrated in Chapter 8.87

Lumley88 suggests comparing the results derivedfrom the different comparators, and suggests aparameter to measure the ‘incoherence’ of thesystem, which considers the consistency of a specificestimated contrast between two treatments withthe rest of the system (here the estimate from theother route). As with other tests in this context,there may be too few trials of each treatmentcomparison for this test to have much power.

Combining direct and indirect evidence (e.g. AvB,AvC and BvC)Often there is both direct and indirect evidence.Traditionally, meta-analysts focus only on directevidence and do not seek indirect evidence, orthey disregard it. Indirect comparisons havemainly been used when there is no directevidence, or perhaps very little. Thus, there arefew examples of indirect and direct evidence beingcombined. Such a combined analysis is consideredhere; whether such an analysis is sensible isconsidered in Chapter 8.


22

The adjusted indirect comparison method doesnot explicitly generalise to allow inclusion of directevidence (RCTs of BvC). It is simple, however, tocombine the estimate from the adjusted indirectcomparison with data from one or more directcomparisons using inverse variance weighting.The separate meta-analysis estimates need notalso be obtained from inverse variance meta-analyses.

Hasselblad showed how logistic regression couldbe used to analyse a mixture of direct and indirectcomparisons.52 The approach works well for fixedeffect models. However, because of the way themodel is set up, the random effects model hedescribed allows the outcome per treatment armto vary randomly across trials rather than thetreatment effect itself, so that this approach is notrecommended. A proper random effects combinedanalysis requires the use of mixed models, asdiscussed above.75 This issue is also considered inAppendix 7.

Hirotsu and Yamada presented a fixed effectmethod for estimating odds ratios from a mixtureof direct and indirect comparisons, noting that theproblem can be seen as an example of anincomplete block design.54 Although presentedrather differently, their method is equivalent toadjusted indirect comparison (possibly combinedwith direct comparisons) using inverse variancemeta-analysis.

A sequence of comparisons (e.g. AvB, BvC, CvDand DvE)The case of a sequence, or chain, of pairedcomparisons may arise in the drug developmentprocess, particularly where the emphasis is onestablishing equivalence of new drugs rather thansuperiority. If A and B are found to bebioequivalent (often defined as an effect ratio inthe range 0.8–1.25), and from separate studies soare B and C and C and D, what can be said aboutthe equivalence of A and D?63

Such questions may arise primarily in smallbioequivalence studies looking at uptake of drugsvia measures such as total and peak exposure,63

but in principle can also arise in studies of clinicalequivalence, and may arise in superiority trialstoo. An example of such a chain is shown inTable 18 in Chapter 8.

A simple chain such as AvB, BvC, CvD and DvEcan be handled by most of the methods of analysisdescribed. For this example, the contrast AvE canbe evaluated via a combination of multiple

applications of the simple adjusted indirectcomparison.68

While such an analysis is clearly easy to carry out,it extends the assumption of exchangeabilityacross three or more sets of trials in the chain,which many may consider to be rather tenuous.

One or more multiarm trials (e.g. AvB, AvC andAvBvC)An extension of the previous case is when thereare multiarm trials comparing all three of thetreatments of interest. Although the standardapproaches to meta-analysis relate to two-armtrials, trials with three or more treatment arms aresurprisingly common, making up about a quarterof parallel group trials in a MEDLINErepresentative sample.89 Multiarm trials also causedifficulties in conventional meta-analysis.

The adjusted indirect comparison method cannottake account of trials with three or more armswithout either splitting or discarding groups.Multiarm trials can be handled using logisticregression, as this method treats the treatmentarm as the unit of data,52 and by more complexmethods.

Gleser and Olkin considered the case where thereis a set of RCTs in each of which one or more ofseveral active treatments has been evaluatedagainst a common control.55 They proposedregression models for obtaining estimates of eitherthe risk difference or (log) odds ratio betweeneach treatment and control, and also for contrastsbetween the active treatments. Their method isconceptually similar to the adjusted indirectcomparison.

When a single multiarm trial is used to estimatetwo or more paired comparisons those estimateswill be correlated when they contain a commonarm (e.g. AvB and BvC). This correlation wasincluded in the models presented by Gleser andOlkin55 and Higgins and Whitehead;53 it is nottaken into consideration in logistic regression.

An arbitrary set of all combinations The most general case allows for trials comparingvarious sets of two, three or more of manytreatments. (The further possibility of theinclusion of other trial designs, such as cluster orcross-over trials, is not considered here, althoughin principle these could also be added.) Anexample is used in Appendix 7 as the basis forillustration of some of the methods of analysis.This scenario goes beyond the main focus of this


23


report, but such data sets can be analysed usingseveral very general methods, such as that of Eddyand colleagues.67

Lumley88 mentioned the possibility of includingmultiarm trials, but did not show how his networkmeta-analysis method extends to that case. Theassumptions of a single analysis of a complex setof trials are considerable, and the inner features ofthe data will be obscured. If such an approach isused it seems desirable also to examine simpleranalyses of subsets of treatments and to exploreconsistency of results.

Adjustment for study and/or patientcharacteristics Concerns about the validity of the assumptionsmay lead to the wish to incorporate study-levelcovariates in the analysis. These may relate toaspects of the study, such as length of follow-up orwhether or not double blind, or study-levelsummaries of patient characteristics such as meanage. Such adjustment may be felt to remove orreduce meta-confounding and make the studiesmore likely to conform to the assumption ofexchangeability. Adjustment will be more usefulfor study characteristics as there is very poorpower to detect the effect of patient characteristicsusing study-level summaries.90,91

Regression methods, including logistic regression,can be used to adjust for study characteristics. Inthe rare case where full individual patient data areavailable, then the whole analysis could beperformed using individual patient characteristics,for example using multilevel (hierarchical)modelling.92

Summary of available analysesIt has been noted that some complex statisticalmethods are available to analyse any set of trials.Such methods are inaccessible to manyresearchers, so it is useful to summarise the typesof data that can be analysed using simpler, widelyavailable methods.

The method of adjusted indirect comparison canbe extended to any case where all of the trials havejust two treatment arms, where necessary by usinginverse variance weighted combination of separateindirect estimates. For example, separate indirectand direct estimates of a common treatment effectevidence can be combined. Likewise, indirect anddirect estimates can also be combined using theresults from separate meta-regression analyses.Adjusted indirect comparison and meta-regressioncan be used to combine fixed or random effects

estimates from subsets of trials, but they cannothandle trials with more than two treatment arms(multiarm trials). However, an advantage of bothapproaches is that they can be used for any type ofoutcome measure, including odds ratios, relativerisk and risk difference for binary data as well asmeans or hazard ratios.

For a binary outcome, logistic regression can beused to perform a fixed effect analysis for all ofthe cases discussed in the previous section,including multiarm trials. Here the odds ratio isthe only available effect measure.

When there is at least one multiarm trial, randomeffects analysis requires hierarchical (mixed)models, and these can also be used for continuousoutcomes. Such models appropriately consider therandom effects to apply to the treatment effect, incontrast to some of the models discussed above.Their use ideally requires expert statisticalassistance.

As noted earlier, several more flexible but complexapproaches can be used to handle any mixture ofdata structures, although not all can deal with anytype of outcome measure.

Related literature The focus of interest here was the combination ofresults from RCTs, and the authors would notadvocate including data from uncontrolled studies.However, methods are available to combine datafrom RCTs with results of uncontrolled studies.Begg and Pilote93 proposed a random effectsmodel for combining controlled and uncontrolledstudies in the context of incorporating historicalcontrols into a meta-analysis of comparativestudies. Their method assumes a distribution forthe control group event rate (rather than for thetreatment effect, as in conventional random effectsmeta-analysis) and a fixed treatment effect. Therelative weight given to the controlled studiesdepends on the observed degree of heterogeneity.They illustrated their method using fourcontrolled trials (274 patients) and 12uncontrolled studies (1708 patients) comparingbone marrow transplantation and chemotherapyfor acute non-lymphocytic leukaemia. The methodwas extended by Li and Begg.94 Raghunathan95

presented a conceptually similar approach forcombining the results of case–control studies withdata from controls in previous studies.

Berkey and colleagues50 presented a method forcombining multiple outcomes in a meta-analysis,to allow a composite analysis when not all


24

outcomes were assessed (reported) in all trials.Their generalised least squares regression modelalso extended to analysis of single outcomes fromtrials comparing multiple treatments, not all ofwhich were studied in each trial. They presentedan analysis comparing gold and auranofin forreducing tender joints in patients with rheumatoidarthritis. Of the 44 randomised trials, only ninecompared directly the two treatments of interest.Their method does not preserve the link betweentreatment arms in a given trial and thus cannot berecommended.

Lastly, Büchner and colleagues96,97 described theprospective design of several trials with a commonstandard arm, to allow indirect comparison via thecommon treatment. However, they suggest that ineach trial only a small proportion of patients arerandomised to the common arm and all thepatients across the various trials are used as thecommon comparator. This method mixesrandomised and non-randomised comparisonsand runs a serious risk of bias.98

CommentsAlthough indirect comparisons are common,surprisingly few methodological papers haveproposed or discussed methods for handling suchdata. Searching for such papers was hampered bythe lack of recognised terminology for indirectcomparisons; other terms used in the papersidentified include cross-trial (or cross-study)comparison,82,99 connected comparativeexperiment,54 network meta-analysis,88 mixedcomparison,76 and virtual comparison.100 Hardlyany of the papers identified cited any of theothers. It is thus quite possible that othermethodological papers were missed, although it isunlikely that there are important omissionsregarding methods of analysis.

Some valid approaches were identified foraggregate data that could be applied to simpleproblems using standard software: the adjustedindirect comparison, meta-regression and, forbinary data only, multiple logistic regression (butonly fixed effect models).

A particular advantage of a regression modellingapproach is the possibility to adjust for othervariables available for each study. The hope wouldbe that such adjustment may help to explain someof the heterogeneity within and between groups oftrials making the same comparisons. Adjustmentfor aggregated patient-level variables (such asaverage age) would be expected not to have mucheffect in comparison with adjustment with thesame variables at the individual level.90,91 All suchanalyses are open to all of the problems inherentin meta-regression analysis.62,91 The regressionapproach may be seen as a simple version of someof the more complex approaches that can be used.

The use of simple adjusted indirect comparisonsto combine two-arm trials is being used moreoften. Some of these methods can be extended tomore complex situations. At the extreme, someadvanced methods exist that can handle generalproblems of networks of comparisons, andsometimes also combine different outcomes, studydesigns and perhaps external information.Whether it is sensible to include such variedinformation in a single analysis is open toquestion. It would seem desirable that, if used,such complex analyses are supplemented bysimpler analyses.

Chapter 5 reports on empirical investigations intothe performance of several methods of makingindirect comparisons, using both fixed andrandom effects, with and without adjustment forstudy covariates.


25


Comparisons of indirect and directcomparisons, while interesting, cannot

provide reliable information about the propertiesof the indirect approaches. Apart from theproblem of small numbers of trials, the maindifficulty is the fact that the difference inestimated treatment effect between the two sets oftrials may be confounded with differencesbetween the studies.

To compare direct and indirect estimates withoutsuch confounding an extensive empiricalinvestigation was carried out using data from alarge multicentre randomised controlled trial, theInternational Stroke Trial (IST).101 The centres inthe study were grouped by region to representseparate trials and results from these ‘trials’ usedto generate both direct and indirect estimates ofthe same treatment comparison.

As all of the centres used exactly the sameprotocol, including eligibility criteria, the results ofthe studies can be said to represent a somewhatoptimistic case as, in general, further variabilitywill be introduced by variation in protocols acrossstudies.

The International Stroke Trial(IST)The aim of the trial was to assess the separate andcombined effect of aspirin (300 mg daily) and ofsubcutaneous heparin [5000 IU twice daily (lowdose) and 12,500 IU twice daily (medium dose)]administered for 14 days. Between January 1991and May 1996 19,435 patients with suspectedacute ischaemic stroke entering 467 hospitals in 36countries were assigned centrally to a treatmentusing minimisation within 48 hours of symptomsonset.

Details of the study methods and interventionshave been described elsewhere.101 In brief, using afactorial design, half of all patients were randomly

allocated to receive aspirin and half to ‘avoidaspirin’. For each of these two groups, half of thepatients were allocated heparin (low or mediumdose) and half to ‘avoid heparin’. Placebos werenot used.

The primary outcomes were death within 14 daysand death or dependency at 6 months. At6 months fewer patients in the aspirin group weredead or dependent [62.2% versus 63.5%, p = 0.07;a difference of 13 (SD 7) per 1000]. Afteradjustment for baseline stroke severity, the benefitfrom aspirin was statistically significant [14 eventsprevented per 1000 patients (SD 6), p = 0.03].Heparin was not found to have any effect (eventrate 62.9% in groups who did or did not receiveheparin). There was no detectable interactionbetween aspirin and heparin in the mainoutcomes.

Adaptations of the trialOutcome and treatmentDeath and dependence at 6 months wasconsidered as the main outcome. Aspirin versusheparin (either dose) was taken as the comparisonof interest. The four treatment groups werelabelled as follows:

� no treatment: A � aspirin: B� heparin: C� aspirin and heparin: D.

CountriesData from the different countries in the trial wereused to represent multiple studies in a meta-analysis. Only countries with more than 100patients for each arm were considered. As Italyand the UK enrolled a large number of patientsboth were split into three new ‘countries’ byregion. Thus, data from 16 countries wereanalysed. As a consequence of omitting somesmaller centres, the overall comparison between


27


Chapter 5

An empirical investigation of the properties of different statistical methods used for performing

indirect comparisons

aspirin and heparin is slightly different from thatobtained in the actual trial.

Sampling schemeFor each comparison, a sample of 100 patientswas drawn at random with replacement from each relevant treatment arm in each country.Thus, the samples drawn were all of the same sizeregardless of the number of patients actually inthe trial in that country. This eliminatedcomplications arising from variation in samplesize across trials.

For indirect comparisons, random samples ofpatients receiving aspirin (B), heparin (C) orneither (A) were used to create meta-analyses withk ‘trials’ comparing AvB and k comparing AvC,where k = 8, 4, 2 or 1. Various methods forindirect comparisons were used to estimate thecontrast between aspirin and heparin. Randomsamples comparing BvC were also taken from thesame 2k countries. The whole process wasrepeated 1000 times.

All analyses were done in Stata 6.0 (StataCorporation, Texas, USA, 2000). Meta-analyseswere undertaken using the metan command.102,103

Random effects meta-regression was fitted usingthe metareg command.102,104

AnalysesThe researchers were interested in thecomparison of aspirin and heparin (BvC).Situations were examined where the comparisonbetween aspirin and heparin could be estimatedusing indirect comparisons. The first situation(case 1) is examined in most detail as it is themost likely to occur. However, it was recognisedthat indirect comparisons can occur in manydifferent ways. Three possibilities are examined incases 2–4 using the same basic approach as incase 1. Each of these cases also used data frompatients in the fourth arm of the IST trial whohad received both aspirin and heparin (denoted D).

Case 1: Indirect comparisons by oneroute (trials of AvB and AvC)Estimates based on indirect comparison (using ktrials of AvB and k trials of AvC) were comparedwith estimates from direct comparisons from thesame 2k countries.

The specific methods used to estimate the BvCtreatment effect were as follows.

� Indirect comparison of BvC using each of thefollowing methods

Method Short name used in tables

Fixed effect adjusted indirect Adjusted indirect (fixed effect)comparison14 (ratio of odds ratios for AvB and AvC)

Random effects adjusted Adjusted indirect (random indirect comparison effects)(DerSimonian and Laird)

Logistic regression Logistic regression(fixed effect)

Random effects meta- Meta-regression (random regression effects)

Naive method (adding Naivenumerators and denominators for treatment arms)

� Direct comparison of BvC in the same 2kcountries used for indirect comparison

Method Short name used in tables

Inverse variance meta-analysis Meta-analysis (fixed effect)

Logistic regression Logistic regression

Random effects meta-analysis Meta-analysis (random (DerSimonian and Laird) effects)

In addition, logistic regression analyses wereperformed for both direct and indirectcomparisons adjusting for the three covariatesgender, age and risk score, both at the individuallevel and using study-level summaries.

Analyses were performed for k = 8, 4, 2 or 1, but some methods could not be used reliably forsmall k.

The scenario just described is asymmetricregarding the three treatments, focusing on BvCas the comparison of interest. As all of the datawere available, it was also possible to set up thesame analyses with, in turn, AvB or AvC as thecomparison of interest. The extent to which thesethree analyses would indicate the sameperformance of different methods of analysisdepends on the similarity of the amount ofheterogeneity between studies for the differentcomparisons.

Additional analyses were performed to examinethe effect of adjusting for patient covariates.Logistic regression was used with adjustment forpatients’ gender, age and symptom score. Analyseswere done using either individual patient data

An empirical investigation of the properties of different statistical methods used for performing indirect comparisons

28

(k = 8, 4, 2 or 1) or trial-level summary statistics(k = 8 or 4).

Case 2: Indirect comparison of BvC bytwo routes (trials of AvB, AvC and DvB,DvC)The same basic approach to sampling was used,with k trials of each of the four comparisons.Indirect comparison of BvC was made usinglogistic regression. Analyses were performed fork = 4, 2 or 1.

Case 3: Indirect comparisons of BvC bytwo steps (trials of BvD, DvA and AvC)Here, k trials of each of three comparisons weregenerated, and indirect comparisons madecombining two adjusted indirect comparisons.Thus, trials of BvD and DvA were combined toobtain an indirect estimate of the comparisonBvA, and then this estimate was combined withthe estimate for AvC to give an estimate of thecomparison BvC. Analyses were performed fork = 4, 2 or 1.

Case 4: Indirect comparisons of BvCfrom multiarm trials (trials of AvBvDand AvCvD)Here, k trials of each of two three-armcomparisons were generated, and indirectcomparisons made using logistic regression.Analyses were performed for k = 8, 4, 2 or 1.

Some theoretical resultsPrecision of indirect estimatesSuppose that a set of trials has been performed onsets of patients randomly sampled from the samepopulation. The symbol � was used to indicate theestimated log odds ratio and a subscript in squarebrackets to indicate the number of trials beingcombined to obtain this estimate. For one trial,suppose that the estimated treatment effect � hasvariance �2 (= [SE(�)]2). Then for a meta-analysisof 2k trials, all of the same size and assuming acommon true treatment effect, an inverse variancemeta-analysis would provide an estimated varianceof the treatment effect of [SE(�[2k])]

2 = �2/2k. Theexpected variance from an indirect comparisonbased on k trials for each comparison is

�2 �2 2�2

[SE(�AB[k] – �AC[k])]2 = — + — = ——.

k k k

Thus, one directly randomised trial is as precise asan indirect comparison based on four randomisedtrials of the same size. Put differently, four times

as many similarly sized trials are needed for theindirect approach to have the same power asdirectly randomised comparisons. This relationwill be approximately true when � is estimatedfrom k trials of varying sizes.

The 4:1 ratio depends on certain assumptions,including equal variances for comparisons AvB,AvC and BvC. Using �2 to denote the between-trialvariance, one requires

�2AB + �2

AB = �2AC + �2

AC = �2BC + �2

BC

This assumption cannot be tested unless one hasdata from all three two-way comparisons, and eventhen there are unlikely to be enough trials to allowreliable estimation of all of these variances.

Fixed effect methodsAs with standard meta-analysis, fixed effectsmethods for indirect comparisons (the method ofBucher and colleagues14 and logistic regression)will underestimate SE2(�) if there is excessheterogeneity. For the indirect comparisonbetween B and C,

2�2

[SE(�AB[k] – �AC[k])]2 = —– + �2

AB + �2AC

k

rather than 2�2/k given earlier. Heterogeneity in atleast one of the sets of trials will lead to theestimated SE2(�) being too small. Random effectsanalyses would therefore seem to be a safer optionfor indirect comparisons.

The naive methodThe naive method treats the data as if they camefrom a single trial and completely ignores between-trial variance. To take the simplest case with noexcess heterogeneity, SE2(�) for a single trial of sizen is �2, as before. A naive comparison between karms of treatment A and k arms of treatment B isequivalent to a single trial of size kn, so that SE2(�)would be estimated as �2/k, which is half of thevariance from the adjusted indirect comparison bythe method of Bucher and colleagues.14 Thus, inthe case with no heterogeneity the naive methodwill give standard errors that are too small by afactor of 1/√2; that is, about 30% too small. Whenthere is between-trial heterogeneity, theunderestimation will be even greater.

Results of empirical studiesTables 4–10 summarise results of analyses of 1000simulated data sets, constructed as described inthe previous section.


29


The content of the columns in Tables 4–6 isdescribed in the box below. Tables 7–10 contain thesame information apart from the estimated �2.Tables 8–10 also do not show estimated coverage.In the tables the notation k + k is used to denoteindirect comparisons based on two sets of k trials,and similarly for three or four sets of trials inTables 8 and 9.

Case 1: Indirect comparisons by oneroute (trials of AvB and AvC)Table 4 summarises direct and indirectcomparisons of aspirin and heparin (BvC) fordirect comparisons based on 16, 8, 4 and 2 trials

and indirect comparisons based on 8 + 8, 4 + 4,2+2 and 1 + 1 trials. Thus, for example, thedirect comparison involving 16 trials of BvC isbased on exactly the same amount of data as theindirect comparison derived from eight trials ofAvB and 8 of AvC. For the direct comparisonsresults are shown for an inverse variance meta-analysis, logistic regression and random effectsmeta-analysis (DerSimonian and Laird). For theindirect comparisons analyses were made using the same methods (the inverse variancemeta-analysis approach is that suggested byBucher and colleagues14), and also by the naivemethod. Random effects analyses with fewer thanfour trials per treatment comparison were notperformed.

Several features of the results can be noted:

� The estimated odds ratio is effectively the samewhichever method was used. The samplingprocedures used should lead to this result.

� As predicted, the results from indirectcomparisons were less precise than those fromdirect comparisons. As noted above, a fixedeffect indirect comparison of k+k trials wouldbe expected to give estimates with twice thevariance as a direct comparison based on 2ktrials (all of the same size). Thus, the standarderror of the indirect estimate is expected to beabout √2 = 1.41 larger than that of the directestimate.

� Similarly, the standard error for the indirectcomparison of k+k trials would be expected tohave the same precision as the direct estimatefrom k/2 trials. Table 4 shows that the standarderrors of the fixed effect indirect estimates from8+8 trials are indeed very similar to those from4 direct trials, and likewise for indirect 4+4 anddirect 2.

� The standard error obtained from the fixedeffect analysis will be too small if there isbetween-trial heterogeneity (beyond randomvariation). Column 6 shows that there was suchheterogeneity in all cases except for k<4. Forboth direct and indirect fixed effect estimates(first two rows of each section) the empiricalstandard deviations of the 1000 estimated logodds ratios (column 5) exceed the averagestandard errors (column 4). An exception hereis the direct comparisons using all 16 trials, forwhich there is no obvious explanation.

� Equivalent random effects methods seem toover-correct for heterogeneity, giving standarderrors that are larger than the empiricalstandard deviations of estimated log oddsratios.105

Column

2 Estimated odds ratio (obtained as exponential ofthe value in column 3)

3 Estimated log odds ratio (mean of 1000 estimatedvalues), denoted logOR

----------

4 Standard error of the estimated log odds ratio (mean of 1000 estimated values), denoted

SE––

(logOR) –––––––

5 Standard deviation of 1000 estimates log oddsratios (provides a non-parametric estimate ofuncertainty of the estimated log odds ratio),denoted SD(logOR

----------)

Comparison with the standard error (previouscolumn) indicates the extent to which the methodof analysis (e.g. logistic regression) underestimatesthe imprecision of the estimated treatment effect

6 Median value of �2 from 1000 analyses (forrandom effects models only). For indirect values, this is the median of the average of thetwo values of �2 for the two componentcomparisons Positive values indicate heterogeneity abovechance variation between studies. Differencesbetween values for different comparisons (e.g. comparing Tables 4 and 5) indicate non-constant heterogeneity for differentcomparisons

7,8 Coverage of the 95% confidence intervals:percentages of 1000 confidence intervals for thelog odds ratio that were wholly below or whollyabove the value in column 2 For a good method there should be about 2.5%of confidence intervals that lie wholly below orwholly above the true value

9,10 Percentages of 1000 trials for which thecomparison of two treatments was statisticallysignificant (p < 0.05) in favour of each treatmentThe percentages for indirect comparisons can be compared with the direct comparisons toshow overoptimistic or conservative methods


30

� The results for the adjusted indirect inversevariance meta-analysis are almost identical tothose using logistic regression. Likewise, theresults from the random effects methods(DerSimonian and Laird meta-analysis andrandom effects meta-regression) agree closelyfor these data.

� The naive method gives standard errors that aremuch smaller than those from all of the validmethods. Further, the empirical standarddeviations using the naive method were verymuch larger than those using properapproaches. As a consequence, the apparentstandard error of the estimates from the naive method is only about 40% of the correct size.

� Fixed effect direct and indirect methods do notgive the right coverage: the proportion ofoccasions when 95% confidence intervals do not include the correct value (i.e. the directestimate based on all the trials) exceeds 5% as a consequence of the excess heterogeneity. By contrast, for random effects models thecoverage is about 5%. Again, only the direct analyses based on 16 trials do not follow this pattern. The coverage of the naive method is awful, with over 40% ofconfidence intervals not including the correctvalue.

� The probability of obtaining a significant result (power) in favour of aspirin was similar for indirect and direct comparisons of equivalent strength (e.g. direct 4 and indirect 8+8). Random effects analyses had a rather lower power than fixed effectmethods, but also a reduced risk of a significant result in favour of heparin. As the number of studies reduces, as expected, the power falls and the probability of asignificant result in favour of heparin increases.

� The naive method not only gave a muchinflated probability of a significant result, butalso gave a high risk, 12–20%, of a significantresult in favour of heparin. About half of theanalyses using the naive method werestatistically significant in one direction or the other, for all numbers of trialsconsidered.

Tables 5 and 6 show the same information as inTable 4, but for analyses of no treatment versus heparin and no treatment versus aspirin,respectively. Although the broad picture is the same as in Table 4, some differences arisefrom variation in the degree of heterogeneity of the different comparisons. In particular,

for the direct comparison between no treatmentand heparin (Table 5) the between-trialheterogeneity was much less than for the othertwo comparisons. In fact, �2 for this comparisonwas about one-quarter of �2 for the other twocomparisons, indicating that the spread (standard deviation) of the distribution ofestimates across trials was twice as great for aspirin versus heparin and no treatment versusaspirin as for no treatment versus heparin. This disparity leads to the underestimation ofstandard errors of estimates being greatest for no treatment versus heparin. In most realapplications one would not be able to estimate �2 for all three two-way comparisons, so any such variability would be hidden.

The effect of adjustment for patient characteristicswas also explored. Table 7 shows for the aspirinversus heparin comparison the results ofadditional logistic regression analyses in whichadjustment was made for three patient variables:age, gender and prognostic score. The scoreindicates the number of symptoms (deficits) thatpatients presented at randomisation. Onlysymptoms that increased the risk at the 5% level ofstatistical significance in univariate analysis wereconsidered. The score was thus based on thefollowing deficits: face, arm/hand, leg/foot,dysphasia, hemianopia and visuospatial. Separate adjusted analyses were made usingindividual data and also using trial-levelsummaries (only for analyses of at least 8 trials).For comparison purposes, the results ofunadjusted logistic regression analyses arerepeated from Table 4. Only fixed effects analyseswere used.

Adjustment for individual patient data had a smalleffect on the estimated log odds ratio in bothdirect and indirect analyses. There was also a small inflation of the standard error of the logodds ratio. Adjustment for study-level summarydata had a rather greater effect on the estimatedlog odds ratio in both direct and indirect analyses, giving results nearer to no effect (OR=1)than the analyses based on individual data. There was also a further inflation of the standarderror of the log odds ratio in comparison to the analyses using individual data. The standarderrors for analyses with adjustment for study-level summaries were about 50% larger than the standard errors of unadjusted estimates.In all cases the standard errors were smaller than the empirical standard deviations ofestimates.


31



32 TA

BLE

4Re

sults

of d

irect

and

indi

rect

com

paris

ons

of a

spiri

n ve

rsus

hep

arin

usin

g da

ta fr

om d

iffer

ent

num

bers

of c

ount

ries

as s

epar

ate

tria

ls (r

esul

ts b

ased

on

1000

resa

mpl

ings

bas

ed o

n th

e IS

Tst

udy;

pat

ient

out

com

e w

as d

eath

or d

epen

denc

e)

Cov

erag

e of

95%

CI

% R

esul

ts w

ith

p<

0.05

OR

logO

R---

------

---SE

(log

OR

)SD

(log

OR

------

------

)M

edia

n �

2B

elow

tru

e A

bove

tru

e Fa

vour

s Fa

vour

sva

lue

(%)

valu

e (%

)as

piri

nhe

pari

n

Dir

ect

16 t

rial

sM

eta-

anal

ysis

(fixe

d ef

fect

)0.

88–0

.131

0.07

50.

070

2.0

1.9

41.7

0Lo

gist

ic r

egre

ssio

n 0.

88–0

.132

0.07

50.

071

2.0

2.4

42.5

0M

eta-

anal

ysis

(ran

dom

effe

cts)

0.88

–0.1

300.

097

0.07

20.

058

0.6

0.6

21.7

0

Indi

rect

(8)

+ (

8)a

tria

lsA

djus

ted

indi

rect

(fix

ed e

ffect

)0.

88–0

.131

0.14

90.

171

4.8

4.5

17.2

0.4

Logi

stic

reg

ress

ion

0.87

–0.1

340.

148

0.17

14.

24.

317

.30.

8M

eta-

regr

essio

n (r

ando

m e

ffect

s)0.

87–0

.138

0.18

10.

171

0.03

71.

03.

512

.10.

1A

djus

ted

indi

rect

(ran

dom

effe

cts)

0.88

–0.1

290.

189

0.17

10.

050

1.2

2.2

9.1

0.2

Nai

ve

0.89

–0.1

150.

102

0.25

522

.320

.138

.112

.1

Dir

ect

8 tr

ials

Met

a-an

alys

is (fi

xed

effe

ct)

0.88

–0.1

330.

106

0.12

33.

43.

927

.70.

7Lo

gist

ic r

egre

ssio

n0.

88–0

.128

0.10

50.

123

5.3

3.8

26.9

0.5

Met

a-an

alys

is (r

ando

m e

ffect

s)0.

88–0

.131

0.13

70.

125

0.04

82.

32.

716

.20.

2

Indi

rect

(4)

+ (

4) t

rial

sA

djus

ted

indi

rect

(fix

ed e

ffect

)0.

88–0

.131

0.21

10.

248

5.6

4.1

12.4

1.7

Logi

stic

reg

ress

ion

0.88

–0.1

240.

210

0.25

45.

14.

014

.22.

1M

eta-

regr

essio

n (r

ando

m e

ffect

s)0.

88–0

.125

0.25

90.

245

0.02

72.

52.

57.

50.

9A

djus

ted

indi

rect

(ran

dom

effe

cts)

0.88

–0.1

310.

273

0.25

80.

048

2.5

2.8

7.2

0.6

Nai

ve

0.88

–0.1

310.

145

0.38

023

.323

.235

.213

.9

Dir

ect

4 tr

ials

Met

a-an

alys

is (fi

xed

effe

ct)

0.88

–0.1

260.

149

0.18

46.

14.

918

.51.

0Lo

gist

ic r

egre

ssio

n0.

87–0

.134

0.14

90.

189

6.3

6.2

21.3

1.1

Met

a-an

alys

is (r

ando

m e

ffect

s)0.

88–0

.130

0.19

40.

187

0.02

83.

23.

512

.20.

8

Indi

rect

(2)

+ (

2) t

rial

sA

djus

ted

indi

rect

(fix

ed e

ffect

)0.

88–0

.123

0.29

90.

356

5.6

5.2

10.4

1.8

Logi

stic

reg

ress

ion

0.88

–0.1

310.

298

0.35

54.

84.

79.

42.

9M

eta-

regr

essio

n (r

ando

m e

ffect

s)0.

88–0

.128

0.37

90.

357

0.00

13.

13.

76.

61.

7N

aive

0.

89–0

.116

0.20

80.

545

21.5

22.0

28.6

17.3

cont

inue

d


33


TA

BLE

4Re

sults

of d

irect

and

indi

rect

com

paris

ons

of a

spiri

n ve

rsus

hep

arin

usin

g da

ta fr

om d

iffer

ent

num

bers

of c

ount

ries

as s

epar

ate

tria

ls (r

esul

ts b

ased

on

1000

resa

mpl

ings

bas

ed o

n th

e IS

Tst

udy;

pat

ient

out

com

e w

as d

eath

or d

epen

denc

e) (

cont

’d)

Cov

erag

e of

95%

CI

% R

esul

ts w

ith

p<

0.05

OR

logO

R---

------

---SE

(log

OR

)SD

(log

OR

------

------

)M

edia

n �

2B

elow

tru

e A

bove

tru

e Fa

vour

s Fa

vour

sva

lue

(%)

valu

e (%

)as

piri

nhe

pari

n

Dir

ect

2 tr

ials

Met

a-an

alys

is (fi

xed

effe

ct)

0.88

–0.1

300.

212

0.27

06.

46.

414

.52.

6Lo

gist

ic r

egre

ssio

n0.

88–0

.132

0.21

10.

273

5.4

6.5

16.2

2.1

Indi

rect

(1)

+(1

) tr

ials

Adj

uste

d in

dire

ct (f

ixed

effe

ct)

0.87

–0.1

390.

426

0.52

96.

75.

29.

94.

2Lo

gist

ic r

egre

ssio

n 0.

87–0

.135

0.42

50.

533

5.6

5.8

8.9

3.5

Naï

ve

0.89

–0.1

160.

302

0.82

424

.121

.031

.119

.8

a(N

umbe

r of

stu

dies

invo

lved

in t

he c

ompa

rison

asp

irin

vers

us n

o tr

eatm

ent)

+ (N

umbe

r of

stu

dies

invo

lved

in h

epar

in v

ersu

s no

tre

atm

ent)

.


34 TA

BLE

5Re

sults

of d

irect

and

indi

rect

com

paris

ons

of n

o tr

eatm

ent

vers

us h

epar

in u

sing

data

from

diff

eren

t nu

mbe

rs o

f cou

ntrie

s as

sep

arat

e tr

ials

(res

ults

bas

ed o

n 10

00 re

sam

plin

gs b

ased

on

the

IST

stud

y; p

atie

nt o

utco

me

was

dea

th o

r dep

ende

nce)

Cov

erag

e of

95%

CI

% R

esul

ts w

ith

p<

0.05

OR

logO

R---

------

---SE

(log

OR

)SD

(log

OR

------

------

)M

edia

n �

2B

elow

tru

e A

bove

tru

e Fa

vour

s no

Favo

urs

valu

e (%

)va

lue

(%)

trea

tmen

the

pari

n

Dir

ect

16 t

rial

sM

eta-

anal

ysis

(fixe

d ef

fect

)0.

95–0

.049

0.07

50.

074

1.8

2.3

9.4

0.7

Logi

stic

reg

ress

ion

0.96

–0.0

450.

074

0.07

32.

42.

09.

20.

3M

eta-

anal

ysis

(ran

dom

effe

cts)

0.96

–0.0

460.

085

0.07

30.

017

1.3

1.3

6.7

0.2

Indi

rect

(8)

+ (

8)a

tria

lsA

djus

ted

indi

rect

(fix

ed e

ffect

)0.

96–0

.040

0.15

00.

190

5.2

6.7

8.8

5.9

Logi

stic

reg

ress

ion

0.96

–0.0

400.

148

0.19

48.

05.

710

.44.

7M

eta-

regr

essio

n (r

ando

m e

ffect

s)0.

95–0

.055

0.19

40.

196

0.05

83.

63.

24.

52.

1A

djus

ted

indi

rect

(ran

dom

effe

cts)

0.95

–0.0

520.

200

0.18

70.

066

2.6

2.1

4.5

0.9

Nai

ve

0.96

–0.0

360.

102

0.24

323

.718

.725

.416

.9

Dir

ect

8 tr

ials

Met

a-an

alys

is (fi

xed

effe

ct)

0.95

–0.0

530.

106

0.11

13.

03.

38.

51.

3Lo

gist

ic r

egre

ssio

n 0.

96–0

.046

0.10

50.

111

3.7

3.3

6.9

1.4

Met

a-an

alys

is (r

ando

m e

ffect

s)0.

96–0

.046

0.12

20.

115

0.01

12.

72.

05.

40.

9

Indi

rect

(4)

+(4

) tr

ials

Adj

uste

d in

dire

ct (f

ixed

effe

ct)

0.94

–0.0

560.

212

0.27

65.

96.

358.

74.

2Lo

gist

ic r

egre

ssio

n 0.

95–0

.049

0.21

00.

275

6.1

7.3

9.9

4.4

Met

a-re

gres

sion

(ran

dom

effe

cts)

0.94

–0.0

580.

277

0.26

70.

045

3.6

3.4

5.0

1.9

Adj

uste

d in

dire

ct (r

ando

m e

ffect

s)0.

96–0

.044

0.28

80.

280

0.06

43.

12.

94.

62.

0N

aive

0.

95–0

.051

0.14

50.

363

21.8

2327

.318

.1

Dir

ect

4 tr

ials

Met

a-an

alys

is (fi

xed

effe

ct)

0.95

–0.0

530.

150

0.16

22.

73.

17.

30.

9Lo

gist

ic r

egre

ssio

n 0.

96–0

.037

0.14

90.

168

5.1

3.0

5.6

2.8

Met

a-an

alys

is (r

ando

m e

ffect

s)0.

96–0

.037

0.17

80.

162

0.00

12.

51.

54.

51.

1

Indi

rect

(2)

+ (

2) t

rial

sA

djus

ted

indi

rect

(fix

ed e

ffect

)0.

94–0

.056

0.30

10.

412

7.2

6.5

10.3

4.6

Logi

stic

reg

ress

ion

0.96

–0.0

410.

298

0.40

28.

56.

88.

16.

6M

eta-

regr

essio

n (r

ando

m e

ffect

s)0.

94–0

.067

0.39

90.

396

0.02

14.

24.

86.

13.

3N

aive

0.

96–0

.039

0.20

80.

509

21.9

19.2

22.5

18.4

cont

inue

d


35


TA

BLE

5Re

sults

of d

irect

and

indi

rect

com

paris

ons

of n

o tr

eatm

ent

vers

us h

epar

in u

sing

data

from

diff

eren

t nu

mbe

rs o

f cou

ntrie

s as

sep

arat

e tr

ials

(res

ults

bas

ed o

n 10

00 re

sam

plin

gs b

ased

on

the

IST

stud

y; p

atie

nt o

utco

me

was

dea

th o

r dep

ende

nce)

(co

nt’d

)

Cov

erag

e of

95%

CI

% R

esul

ts w

ith

p<

0.05

OR

logO

R---

------

--SE

(log

OR

)SD

(log

OR

------

------

)M

edia

n �

2B

elow

tru

e A

bove

tru

e Fa

vour

s no

Favo

urs

valu

e (%

)va

lue

(%)

trea

tmen

the

pari

n

Dir

ect

2 tr

ials

Met

a-an

alys

is (fi

xed

effe

ct)

0.95

–0.0

510.

212

0.24

02.

94.

15.

72.

5Lo

gist

ic r

egre

ssio

n0.

96–0

.046

0.21

00.

236

3.9

3.9

5.8

2.0

Indi

rect

(1)

+(1

) tr

ials

Adj

uste

d in

dire

ct (f

ixed

effe

ct)

0.94

–0.0

660.

426

0.57

26.

56.

98.

85.

8Lo

gist

ic r

egre

ssio

n 0.

95–0

.055

0.42

50.

577

6.0

8.1

8.9

5.2

Nai

ve0.

96–0

.038

0.30

20.

784

22.1

20.9

23.5

20.2

a(N

umbe

r of

stu

dies

invo

lved

in t

he c

ompa

rison

asp

irin

vers

us n

o tr

eatm

ent)

+ (N

umbe

r of

stu

dies

invo

lved

in h

epar

in v

ersu

s as

pirin

).


36 TA

BLE

6Re

sults

of d

irect

and

indi

rect

com

paris

ons

of n

o tr

eatm

ent

vers

us a

spiri

n us

ing

data

from

diff

eren

t nu

mbe

rs o

f cou

ntrie

s as

sep

arat

e tr

ials

(res

ults

bas

ed o

n 10

00 re

sam

plin

gs b

ased

on

the

IST

stud

y; p

atie

nt o

utco

me

was

dea

th o

r dep

ende

nce)

Cov

erag

e of

95%

CI

% R

esul

ts w

ith

p<

0.05

OR

logO

R---

------

---SE

(log

OR

)SD

(log

OR

------

------

)M

edia

n �

2B

elow

tru

e A

bove

tru

e Fa

vour

s no

Favo

urs

valu

e (%

)va

lue

(%)

trea

tmen

tas

piri

n

Dir

ect

16 t

rial

sM

eta-

anal

ysis

(fixe

d ef

fect

)1.

080.

079

0.07

50.

072

2.4

2.2

0.1

16.5

Logi

stic

reg

ress

ion

1.09

0.08

60.

074

0.07

52.

91.

80.

121

.9M

eta-

anal

ysis

(ran

dom

effe

cts)

1.09

0.08

40.

099

0.07

50.

065

0.3

0.8

08.

4

Indi

rect

(8)

+(8

)atr

ials

Adj

uste

d in

dire

ct (f

ixed

effe

ct)

1.08

0.07

40.

150

0.16

35.

03.

41.

39.

1Lo

gist

ic r

egre

ssio

n 1.

090.

090

0.14

90.

170

5.4

3.4

1.3

11.4

Met

a-re

gres

sion

(ran

dom

effe

cts)

1.09

0.08

50.

181

0.16

50.

037

2.5

1.5

0.4

6.2

Adj

uste

d in

dire

ct (r

ando

m e

ffect

s)1.

090.

089

0.18

60.

167

0.04

42.

41.

70.

35.

8N

aive

1.

090.

093

0.10

20.

246

24.3

19.4

11.8

34.5

Dir

ect

8 tr

ials

Met

a-an

alys

is (fi

xed

effe

ct)

1.08

0.07

80.

106

0.12

45.

24.

31.

014

.3Lo

gist

ic r

egre

ssio

n 1.

080.

080

0.10

50.

129

6.1

5.8

1.6

15.0

Met

a-an

alys

is (r

ando

m e

ffect

s)1.

090.

085

0.14

00.

129

0.05

92.

34.

10.

29.

5

Indi

rect

(4)

+(4

) tr

ials

Adj

uste

d in

dire

ct (f

ixed

effe

ct)

1.08

0.07

40.

212

0.23

65.

34.

11.

87.

8Lo

gist

ic r

egre

ssio

n 1.

080.

076

0.21

00.

243

3.9

4.6

1.9

6.9

Met

a-re

gres

sion

(ran

dom

effe

cts)

1.07

0.07

20.

262

0.24

80.

027

3.2

2.8

1.4

6.0

Adj

uste

d in

dire

ct (r

ando

m e

ffect

s)1.

090.

083

0.26

90.

239

0.04

52.

01.

70.

94.

5N

aive

1.

080.

077

0.14

50.

375

23.7

22.9

16.6

31.0

Dir

ect

4 tr

ials

Met

a-an

alys

is (fi

xed

effe

ct)

1.08

0.07

70.

149

0.18

86.

17.

42.

513

.5Lo

gist

ic r

egre

ssio

n 1.

090.

088

0.14

90.

189

7.0

4.6

2.5

14.9

Met

a-an

alys

is (r

ando

m e

ffect

s)1.

080.

077

0.19

80.

192

0.03

92.

84.

21.

39.

8

Indi

rect

(2)

+(2

) tr

ials

Adj

uste

d in

dire

ct (f

ixed

effe

ct)

1.10

0.09

40.

301

0.35

35.

55.

42.

88.

4Lo

gist

ic r

egre

ssio

n 1.

090.

088

0.29

90.

355

5.8

3.7

1.8

8.4

Met

a-re

gres

sion

(ran

dom

effe

cts)

1.09

0.08

80.

377

0.36

10

4.4

2.8

1.9

6.3

Nai

ve

1.08

0.08

10.

207

0.54

222

.222

.018

.027

.2

cont

inue

d


37


TA

BLE

6Re

sults

of d

irect

and

indi

rect

com

paris

ons

of n

o tr

eatm

ent

vers

us a

spiri

n us

ing

data

from

diff

eren

t nu

mbe

rs o

f cou

ntrie

s as

sep

arat

e tr

ials

(res

ults

bas

ed o

n 10

00 re

sam

plin

gs b

ased

on

the

IST

stud

y; p

atie

nt o

utco

me

was

dea

th o

r dep

ende

nce)

(co

nt’d

)

Cov

erag

e of

95%

CI

% R

esul

ts w

ith

p<

0.05

OR

logO

R---

------

---SE

(log

OR

)SD

(log

OR

------

------

)M

edia

n �

2B

elow

tru

e A

bove

tru

e Fa

vour

s no

Favo

urs

valu

e (%

)va

lue

(%)

trea

tmen

tas

piri

n

Dir

ect

2 tr

ials

Met

a-an

alys

is (fi

xed

effe

ct)

1.08

0.08

20.

212

0.28

14.

77.

93.

811

.6Lo

gist

ic r

egre

ssio

n1.

090.

085

0.21

00.

277

7.4

6.1

4.0

12.2

Indi

rect

(1)

+(1

) tr

ials

Adj

uste

d in

dire

ct (f

ixed

effe

ct)

1.10

0.09

20.

428

0.50

54.

25.

43.

77.

3Lo

gist

ic r

egre

ssio

n 1.

080.

080

0.42

70.

499

5.1

4.5

3.1

6.2

Nai

ve1.

100.

094

0.30

00.

821

23.5

22.1

20.7

27.5

a(N

umbe

r of

stu

dies

invo

lved

in t

he c

ompa

rison

asp

irin

vers

us h

epar

in) +

(Num

ber

of s

tudi

es in

volv

ed in

hep

arin

ver

sus

no t

reat

men

t).


38 TA

BLE

7Re

sults

of d

irect

and

indi

rect

com

paris

ons

of a

spiri

n ve

rsus

hep

arin

usin

g lo

gist

ic re

gres

sion

with

and

with

out

adju

stm

ent

for p

atie

nt fa

ctor

s at

stu

dy a

nd in

divid

ual l

evel

Cov

erag

e of

95%

CI

% R

esul

ts w

ith

p<

0.05

OR

logO

R---

------

---SE

(log

OR

)SD

(log

OR

------

------

)B

elow

tru

e A

bove

tru

e Fa

vour

sFa

vour

sva

lue

(%)

valu

e (%

)as

piri

nhe

pari

n

Dir

ect

16 t

rial

sLo

gist

ic r

egre

ssio

n 0.

88–0

.132

0.07

50.

071

2.0

2.4

42.5

0A

djus

ted

for

gend

er, a

ge, s

core

(stu

dy le

vel)

0.92

–0.0

790.

089

0.08

94.

01.

114

.10.

1A

djus

ted

for

gend

er, a

ge, s

core

(ind

ivid

ual l

evel

)0.

89–0

.113

0.08

20.

080

1.7

2.4

27.6

0

Indi

rect

(8)

+ (

8)a

tria

lsLo

gist

ic r

egre

ssio

n 0.

87–0

.134

0.14

80.

171

4.2

4.3

17.3

0.8

Adj

uste

d fo

r ge

nder

, age

, sco

re (s

tudy

leve

l)0.

92–0

.087

0.17

10.

183

3.8

2.8

9.0

1.0

Adj

uste

d fo

r ge

nder

, age

, sco

re (i

ndiv

idua

l lev

el)

0.90

–0.1

080.

164

0.18

74.

95.

612

.51.

1

Dir

ect

8 tr

ials

Logi

stic

reg

ress

ion

0.88

–0.1

280.

105

0.12

35.

33.

826

.90.

5A

djus

ted

for

gend

er, a

ge, s

core

(stu

dy le

vel)

0.92

–0.0

780.

154

0.18

13.

92.

711

.91.

0A

djus

ted

for

gend

er, a

ge, s

core

(ind

ivid

ual l

evel

)0.

89–0

.113

0.11

60.

132

3.5

5.1

20.0

0.2

Indi

rect

(4)

+ (

4) t

rial

sLo

gist

ic r

egre

ssio

n 0.

88–0

.124

0.21

00.

254

5.1

4.0

14.2

2.1

Adj

uste

d fo

r ge

nder

, age

, sco

re (s

tudy

leve

l)0.

93–0

.071

0.31

80.

376

4.9

2.6

5.3

2.0

Adj

uste

d fo

r ge

nder

, age

, sco

re (i

ndiv

idua

l lev

el)

0.89

–0.1

150.

232

0.26

74.

34.

88.

81.

9

Dir

ect

4 tr

ials

Logi

stic

reg

ress

ion

0.87

–0.1

340.

149

0.18

96.

36.

221

.31.

1A

djus

ted

for

gend

er, a

ge, s

core

(ind

ivid

ual l

evel

)0.

89–0

.116

0.16

50.

206

5.2

6.1

16.3

1.4

Indi

rect

(2)

+ (

2) t

rial

sLo

gist

ic r

egre

ssio

n 0.

88–0

.131

0.29

80.

355

4.8

4.7

9.4

2.9

Adj

uste

d fo

r ge

nder

, age

, sco

re (i

ndiv

idua

l lev

el)

0.89

–0.1

130.

331

0.39

75.

45.

68.

72.

8

Dir

ect

2 tr

ials

Logi

stic

reg

ress

ion

0.88

–0.1

320.

211

0.27

35.

46.

516

.22.

1A

djus

ted

for

gend

er, a

ge, s

core

(ind

ivid

ual l

evel

)0.

89–0

.120

0.23

60.

298

4.7

7.3

13.8

2.1

Indi

rect

(1)

+(1

) tr

ials

Logi

stic

reg

ress

ion

0.87

–0.1

350.

425

0.53

35.

65.

88.

93.

5A

djus

ted

for

gend

er, a

ge, s

core

(ind

ivid

ual l

evel

)0.

89–0

.116

0.47

70.

556

4.1

4.7

6.4

2.8

a(N

umbe

r of

stu

dies

invo

lved

in t

he c

ompa

rison

asp

irin

vers

us n

o tr

eatm

ent)

+ (N

umbe

r of

stu

dies

invo

lved

in h

epar

in v

ersu

s no

tre

atm

ent)

.


39


TA

BLE

8Re

sults

of i

ndire

ct c

ompa

rison

s of

asp

irin

vers

us h

epar

in (

BvC)

by

two

rout

es (

tria

ls of

AvB

, AvC

, DvB

and

DvC

) (c

ase

2) (

In p

aren

thes

es a

re s

how

n re

sults

from

Tab

le 4

for t

he s

ame

num

bers

of p

atie

nts

in s

impl

e in

dire

ct c

ompa

rison

)

% R

esul

ts w

ith

p<

0.0

5

OR

logO

R---

------

---SE

(log

OR

)SD

(log

OR

------

------

)Fa

vour

s as

piri

nFa

vour

s he

pari

n

Logi

stic

reg

ress

ion

Indi

rect

4 +

4 +

4 +

4 (i

ndire

ct 8

+ 8

)0.

88–0

.131

(–0.

134)

0.14

8 (0

.148

)0.

176

(0.1

71)

16.6

(17.

3)0.

5 (0

.8)

Indi

rect

2 +

2 +

2 +

2 (i

ndire

ct 4

+ 4

)0.

88–0

.127

(–0.

124)

0.21

1 (0

.210

)0.

259

(0.2

54)

14.6

(14.

2)1.

5 (2

.1)

Indi

rect

1 +

1 +

1 +

1 (i

ndire

ct 2

+ 2

)0.

88–0

.129

(–0.

131)

0.30

0 (0

.298

)0.

375

(0.3

55)

11.3

(9.4

)2.

8 (2

.9)

Adj

uste

d in

dire

ct (

rand

om e

ffect

s)In

dire

ct 4

+ 4

+ 4

+ 4

0.87

–0.1

340.

188

0.18

511

.30.

4

TA

BLE

9Re

sults

of i

ndire

ct c

ompa

rison

s of

asp

irin

vers

us h

epar

in (

BvC)

by

two

step

s (t

rials

of t

rials

of B

vD, D

vA a

nd A

vC)

(cas

e 3)

% R

esul

ts w

ith

p<

0.0

5

OR

logO

R---

------

---SE

(log

OR

)SD

(log

OR

------

------

)Fa

vour

s as

piri

nFa

vour

s he

pari

n

Adj

uste

d in

dire

ct (

fixed

effe

ct)

Indi

rect

4 +

4 +

40.

89–0

.118

0.25

80.

304

11.4

1.8

Indi

rect

2 +

2 +

20.

86–0

.156

0.36

60.

447

10.4

2.6

Indi

rect

1 +

1 +

10.

89–0

.114

0.52

20.

637

8.0

3.3

Adj

uste

d in

dire

ct (

rand

om e

ffect

s)In

dire

ct 4

+ 4

+ 4

0.88

–0.1

230.

335

0.31

75.

70.

5

TA

BLE

10

Resu

lts o

f ind

irect

com

paris

ons

of a

spiri

n ve

rsus

hep

arin

(Bv

C) fr

om m

ultia

rm t

rials

(tria

ls of

AvB

vD a

nd A

vCvD

) by

logi

stic

regr

essio

n (c

ase

4) (

in p

aren

thes

es a

re s

how

n re

sults

from

Tabl

e 4

for t

he s

impl

e in

dire

ct c

ompa

rison

bas

ed o

n Av

B an

d Av

C)

% R

esul

ts w

ith

p<

0.0

5

OR

logO

R---

------

---SE

(log

OR

)SD

(log

OR

------

------

)Fa

vour

s as

piri

nFa

vour

s he

pari

n

Logi

stic

reg

ress

ion

Indi

rect

8+

80.

88–0

.131

(–0.

134)

0.12

8 (0

.148

)0.

152

(0.1

71)

21.4

(17.

3)0.

6 (0

.8)

Indi

rect

4+

40.

88–0

.131

(–0.

124)

0.18

2 (0

.210

)0.

225

(0.2

54)

15.4

(14.

2)1.

5 (2

.1)

Indi

rect

2+

20.

88–0

.124

(–0.

131)

0.25

8 (0

.298

)0.

334

(0.3

55)

12.8

(9.4

)3.

0 (2

.9)

Indi

rect

1+

10.

89–0

.120

(–0.

135)

0.36

9 (0

.425

)0.

468

(0.5

33)

9.7

(8.9

)2.

8 (3

.5)

Case 2: Indirect comparison of BvC bytwo routes (trials of AvB, AvC and DvB,DvC)Tables 4–7 consider the simplest indirectcomparison, in which the contrast BvC isestimated using results from trials of AvB and AvC.Table 8 shows results from analyses that included asecond pathway for the indirect comparison ofBvC, namely trials of DvB and DvC. Results forlogistic regressions are shown for 4, 2 and 1 trialper comparison. For comparison the results fromTable 4 are repeated here relating to a singlepathway of twice as many trials: each of the firstfour rows of the table thus compares analyses ofthe same amount of data. The two sets of resultsare very similar, suggesting that the twoapproaches are equivalent. In theory they are, butin practice additional heterogeneity variances areplaying a part and in general one might expectthat the inclusion of additional links wouldincrease the risk of obtaining an erroneous result.

Table 8 also shows the results of a random effectsanalysis for the case of 4 + 4 + 4 + 4 trials. Aswas seen in Table 4, the standard error of theestimated log odds ratio is larger for the randomeffects analysis than for the fixed effect analysis(first row).

Case 3: Indirect comparisons of BvC bytwo steps (trials of BvD, DvA and AvC)Table 9 shows results from three sets of trialsinvolving four treatments. In effect, this methodinvolves two applications of the adjusted indirectapproach in sequence. These results are broadlycompatible with those in the previous tables.Again, a random effects analysis is shown for thelargest number of trials (4+4+4).

Case 4: Indirect comparisons of BvCfrom multiarm trials (trials of AvBvDand AvCvD)Lastly, Table 10 shows results from indirectcomparisons of two sets of three-arm trials.Although these analyses also make use oftreatment D, in this set-up the comparison isinternal (and randomised), unlike in case 2.Although there is the same number of trials thenumber of patients is 50% greater than in Table 8. As a result, there is a 25% reduction invariance of the estimated log odds ratio compared with the results in Table 8. Similarly,there is a reduction in variance compared with thesimple indirect comparison based on AvB andAvC, which for comparison is repeated here fromTable 4.

SummarySeveral methods have been proposed to estimatethe different effectiveness of treatments notcompared directly in randomised trials. Thisinvestigation focused on variations of widelyavailable methods: the adjusted indirectcomparison (fixed or random effects), meta-regression and logistic regression, and alsoevaluated the inappropriate ‘naive’ method.

The results of the resampling studies show that allof the appropriate methods are unbiased and willgive the right answer on average across many suchapplications. However, the correctness of theestimated standard error will depend on strongbut unverifiable assumptions. Little difference wasfound in the performance of inverse variancemethods (adjusted indirect comparison) andlogistic regression. But, for this data set, theresults support the use of random effects versionsof either of these approaches. Such a finding isnot surprising given that there are (at least) threeheterogeneity variances that could have an impacton the results of even the simplest indirectcomparison.

With binary outcomes, logistic regression mayseem preferable to the adjusted indirectcomparison. The results of the two methods areextremely close when both can be applied. Anadvantage of the adjusted indirect comparison forbinary data is that the meta-analysis is notrestricted to the use of the odds ratio, although itmay be possible to use other types of regressionmodel. The main advantages of logistic regressionare that it can be extended to more complexsituations and it can be used to adjust forindividual- or study-level covariates. In practice,however, IPD are unlikely to be available and theuse of study-level patient covariates (such as meanage) is in general too blunt a method to beinformative. Certainly, in this investigationadjustment for study-level summaries of patientcharacteristics led to biased estimates of treatmenteffect with inflated standard errors. By contrast,the ability to include study-level covariates thatrelate to the study itself can be valuable, as in thestudy by Packer and colleagues.69

This resampling study used the odds ratio as ameasure of effect. While the odds ratio hasdesirable statistical properties, it may not alwaysbe the preferred effect measure for a meta-analysis.57 Adjusted indirect comparisons based oncontrasting separate meta-analyses, such as theapproach of Bucher and colleagues,14 can be


40

extended very simply to risk ratio or riskdifference, but logistic regression and some morecomplex methods do require the effect measure tobe the odds ratio. For continuous data theadjusted indirect comparison approach can beused for either actual or standardised meandifferences, as in the study by Packer andcolleagues.69 The choice here is perhaps simpler,with a strong preference for using the actual meandifference as long as the outcome being measuredis the same in the different studies.

These analyses were all based on the data from asingle large trial, with a unified protocol andhence consistent inclusion criteria. In general,more variation would be expected between

independent component studies contributing toan indirect comparison. In particular, there maybe differences between the participants in thetrials (and aspects of the trials themselves)between the two sets of trials being combined.

In addition, the present analyses compared atreatment with a small benefit (aspirin) with onewith no apparent benefit (heparin). The extent towhich our findings would apply in the morecommon case where indirect comparisons aremade when each of two active drugs is better thanplacebo (or no treatment) is unknown. Also,differences between direct and indirect estimatesmay be less important where the effect of interestwas large.


41


The objective of this chapter is to summariseempirical evidence about the validity of

indirect comparison by measuring discrepancybetween the direct and the indirect estimates in asample of published meta-analyses. It builds onthe findings of the survey of the use of indirectcomparison for evaluating relative efficacy ofcompeting interventions presented in Chapter 3.(Note: data in this chapter have previously beenpublished.106)

MethodsIdentification of relevant reviewsDetailed strategies for identifying relevant meta-analyses have been described in Chapter 3. Inbrief, the hard copies of reviews on DARE (1994 toDecember 1998) were available and read. Inaddition, the reviews on The Cochrane Library(Issue 3, 2000) were screened. Included meta-analyses had to meet two criteria: competinginterventions could be compared both directly andindirectly, and the same primary studies were notused in both the direct and indirect comparison.

Comparison methodsThe relative efficacy of competing interventionswas estimated using three comparative methods:the direct (head-to-head) comparison, the adjustedindirect comparison and the naive indirectcomparison. For the direct comparisons, therelative efficacy (TAB) of intervention A versus Bwas estimated by comparing the result of group Aand the result of group B within a randomisedtrial. When there were two or more similar trialsthat compared the same interventions, results ofindividual trials were weighted by the inverse ofcorresponding variances and then quantitativelycombined. Whenever possible, the random effectsmodel was used for the quantitative pooling.

The naive indirect estimate of relative efficacy wasobtained simply by comparing the result oftreatment A in trial 1 with the result oftreatment B in trial 2. That is, in a naive indirect

comparison, results of individual arms betweendifferent trials are compared as if they are from asingle trial. The variance in the naive indirectcomparison will be similar to the variance in thedirect comparison with the same sample size (seeChapter 5). When more than one study wasavailable for a treatment, the results of individualstudies were weighted by the number ofparticipants in the corresponding arms and thenquantitatively combined (thus, between-trialvariability was not considered).

Adjusted indirect comparisons were conductedusing the method suggested by Bucher andcolleagues.14 In brief, the indirect comparison ofinterventions A and B was adjusted by the resultsof their direct comparisons with a commonintervention C. Suppose TAC is the estimate ofdirect comparison of intervention A versus C, andTBC is the direct comparison of intervention Bversus C. Then the estimate of the adjustedindirect comparison of intervention A versus B(T�AB, e.g. log relative risk, mean difference) isestimated by

T�AB = TAC – TBC

and its variance is

Var(T�AB) = Var(TAC) + Var(TBC)

This adjusted method aims to overcome thepotential problem of different prognostic factorsbetween study participants in different trials, andit is valid if the relative efficacy of interventions isconsistent in patients across different trials. Itshould be noted that the variance in the adjustedindirect comparison using four trials is equivalentto the variance in the direct comparison withinone trial of the same size (see Chapter 5). Multiplestudies used in the adjusted indirect comparisonare combined by a random effects modelwhenever possible.

Measures of discrepancyThe relative efficacy was measured using meandifference for continuous data and log relative risk


43


Chapter 6

Statistical discrepancy between the direct and indirect estimate: empirical evidence from published

meta-analyses

for binary data. The discrepancy between thedirect estimate (TBC) and the adjusted indirectestimate (T�BC) was measured by the difference (X)between the two estimates:

X = TBC – T�BC

and its standard error is

SE(X) = √—————————–— SE(TBC)2 + SE (T�BC)2

where SE(TBC) and SE(T�BC) are the estimatedstandard errors for the direct estimate and theadjusted indirect estimate, respectively. The 95%confidence interval for the estimated discrepancywas calculated by X ± 1.96 * SE(X). The estimateddiscrepancy can also be standardised by itsstandard error to obtain a value of z = X/SE(X).

In addition, the results of meta-analyses werecategorised as statistically non-significant(p > 0.05) or statistically significant (p 0.05).The statistically significant effect can be furtherseparated according to whether intervention B wasless or more effective than intervention C. Thedegree of agreement in statistical conclusionsbetween the direct and indirect method wasassessed by a weighted kappa.

ResultsThe searches identified 28 systematic reviews inwhich both the direct and indirect comparison ofcompeting interventions could be conducted,although indirect comparison was not explicitlyused in many of these meta-analyses (seeAppendix 8). Some systematic reviews assessedmore than two active interventions, and a total of44 comparisons could be made using data fromthe 28 systematic reviews.

Figure 2 summarises the statistical discrepancies(z statistics) between the direct and both naive andadjusted indirect estimates for 43 meta-analyses(note that one meta-analysis in which the naiveindirect comparison was not available was notincluded in Figure 2). There are several significantdiscrepancies between the direct and indirectestimates, but the direction of the difference isinconsistent. The relative efficacy of an interventionmay be overestimated or underestimated by theindirect comparison, as compared with the resultsof the direct comparison.

Significant discrepancy (|z| ≥ 1.96) was observedin three of the 44 comparisons between the direct

and adjusted indirect estimate (7%). Between thedirect and the naive indirect estimate, 11 of the 43comparisons showed statistically significantdiscrepancy (26%). The statistical discrepanciesbetween the direct and the naive indirect estimateare generally greater than those between the directand the adjusted indirect estimate (Figure 2).

Figure 3 shows the relation between the statisticaldiscrepancy (z) and the number of trials used inindirect comparisons. Visually, statisticaldiscrepancies tended to be smaller when thenumber of trials was large (>60) than when thenumber of trials was small (<40). However, suchtendency was not consistent, as the discrepancybetween the direct and indirect estimate may besignificant even when more than 40 trials havebeen used for the indirect comparison (e.g. meta-analysis 44).

Figure 4 summarises the differences (and their 95%confidence intervals) between the direct and theadjusted indirect estimates. Significant discrepancy(p < 0.05) was observed in three of the 44comparisons (i.e. the 95% confidence interval didnot include zero). In four other meta-analyses, thediscrepancy between the direct and the adjustedindirect estimate was borderline significant (i.e.p < 0.1). The relative efficacy of an interventionwas equally likely to be overestimated orunderestimated by the indirect comparison, ascompared with the results of the direct comparison.

There was a moderate agreement in statisticalconclusions between the direct and the adjustedindirect method (weighted kappa = 0.53) (Table 11).In terms of statistical conclusions, 32 of the 44indirect estimates fell within the same categories ofthe direct estimates. According to the directcomparison, 19 of the 44 comparisons suggested astatistically significant difference (p < 0.05) betweencompeting interventions. Compared with directestimates, the adjusted indirect estimates were lesslikely to be statistically significant. Ten of the 19significant direct estimates became statistically non-significant in the adjusted indirect comparison,whereas only two of the 25 non-significant directestimates were significant in the adjusted indirectcomparison. The agreement between the direct andthe naive indirect comparison is much poorer(weighted kappa = 0.28).

CommentaryA wide range of medical topics has been coveredby the 44 meta-analyses used in this study

Statistical discrepancy between the direct and indirect estimate: empirical evidence from published meta-analyses

44


45


6420–2–4–6

z value

FIGURE 2 Statistical discrepancy between the direct and the indirect estimate: empirical evidence from published meta-analyses.Solid bars, discrepancy between the direct and adjusted indirect estimate; blank bars, discrepancy between the direct and naive indirectestimate.

(Appendix 8). The patient categories includethose with an increased risk of vascular occlusion,HIV-infected patients, those with GORD,postoperative pain or dyspepsia, and cigarettesmokers. These meta-analyses were used to obtaindata for examining discrepancies between thedifferent comparative methods. The study did notcritically appraise the methodological quality ofthese meta-analyses and of primary trials in thesemeta-analyses.

The findings presented in this paper suggest thatthe direction of discrepancy between the directand the indirect estimate is unpredictable, but thediscrepancy in the adjusted indirect comparison isgenerally less than that in the naive indirectcomparison. The observed significantdiscrepancies between the direct and the indirectestimates may be explained by many possiblefactors, such as random errors, baseline prognosticcharacteristics, interventions other than those


46

10

8

6

4

2

0

–2

–4

–6

–8

–10

0 15 30 45 60 75 90

No. of trials for indirect comparison

Adjusted

Naivez

valu

e

FIGURE 3 Statistical discrepancy (z value) and the number of trials used in indirect comparison

TABLE 11 Methods of comparison and number of significant findings in 44 meta-analyses of competing interventions

Adjusted indirect estimate Naive indirect estimate

Direct estimate Significant Non- Significant Significant Non- Significant effect (–) significant effect (+) effect (–) significant effect (+)(n = 6) effect (n = 33) (n = 5) (n = 9) effect (n = 19) (n = 15)

Significant effect (–) 5 3 0 3 5 0(n = 8)

Non-significant effect 1 23 1 5 11 8(n = 25 or 24)

Significant effect (+) 0 7 4 1 3 7(n = 11)

Non-significant effect: difference between intervention groups is statistically non-significant (p > 0.05); significant effect(p 0.05) is separated according to whether the intervention A is less (–) or more effective (+) than intervention B. Agreement between the direct and the adjusted indirect estimate: weighted kappa value 0.53; agreement between thedirect and the naive indirect estimate: weighted kappa value 0.28.


47


–2–10 1 2

Met

a-an

alys

es

Discrepancy(95% CI)

123456789

1011121314151617181920212223242526272829303132333435363738394041424344

FIG

UR

E 4

Disc

repa

ncy

betw

een

the

dire

ct a

nd t

he a

djus

ted

indi

rect

com

paris

on: e

mpi

rical

evid

ence

from

44

publ

ished

met

a-an

alys

es. N

umbe

ring

of m

eta-

anal

yses

(1–

44)

follo

ws

the

orde

r as

inAp

pend

ix 8

. The

disc

repa

ncy

betw

een

the

dire

ct a

nd t

he in

dire

ct e

stim

ate

was

def

ined

as

diffe

renc

e in

est

imat

ed lo

g re

lativ

e ris

k (m

eta-

anal

yses

1–3

9), d

iffer

ence

in e

stim

ated

sta

ndar

dise

d m

ean

diffe

renc

e (m

eta-

anal

ysis

40)

or d

iffer

ence

in e

stim

ated

mea

n di

ffere

nce

(met

a-an

alys

es 4

1–44

).

compared, follow-up period and methods foroutcome measurement.

Direct versus naive indirect estimateEvidence from the naive indirect comparisons canbe considered to be equivalent to results fromobservational or non-randomised controlledresearch. To explain the discrepancies between thedirect and the naive indirect estimates, the mostimportant factor that should be considered is lackof comparability between patients in differentstudies. The observed difference betweentreatment groups in the naive indirect comparisonmay be due not to different interventions ofinterest, but to different prognostic characteristicsamong patients in the different studies(confounding).

Random error may be a cause of discrepancybetween the direct and the naive indirect estimate.When |z| = 1.96 is used as the cut-point forstatistical significance, the chance of a type I erroris 5%, that is, observing significant discrepancyeven if the null hypothesis is true. Because thenaive indirect estimates tend to be overprecise (seeChapter 5), the statistical discrepancies betweenthe direct and the naive indirect method will begreat, compared with those between the direct andthe adjusted indirect method.

Eleven of the 43 comparisons show statisticallysignificant discrepancy between the direct and thenaive indirect estimate. In seven of these 11examples with significant discrepancy, the pointestimate by the naive indirect comparison is in theopposite direction to that by the directcomparison. Because of the high frequency ofstatistically significant discrepancy andunpredictable direction of such discrepancy, thenaive indirect method should be avoided in theanalysis of data from randomised trials.

Direct versus adjusted indirect estimateDiscrepancies between the direct and the adjustedindirect estimate may also be due to random errors.However, because of the wider confidence intervalprovided by the adjusted indirect comparison, thediscrepancies between the direct and the adjustedindirect estimate are less likely to be statisticallysignificant than those between the direct and thenaive indirect estimate. Indeed, the statisticallysignificant results by using the direct comparisonoften become statistically non-significant in theadjusted indirect comparison (Table 11).

More importantly, perhaps, prognosticcharacteristics of participants in different studies

have been taken into account partially by theadjusted indirect method. The adjusted indirectmethod has partially preserved the rigour ofrandomisation by considering direct comparisonsof interventions of interest with the same controlintervention. An underlying assumption in thisadjusted method is that the relative efficacy of anintervention is consistent in participants indifferent studies. It is important to examine thegeneralisability of results of trials in the adjustedindirect comparison.

Of the 44 comparisons, three showed significantdiscrepancy (p < 0.05) between the direct andadjusted indirect estimates.27,40,107 These threecases will be further discussed in Chapter 7.

Combination of the direct and theadjusted indirect estimatesIt is often the case that direct evidence is availablebut not sufficient. In such cases, the adjustedindirect comparison may provide supplementaryinformation in evaluating relative efficacy ofcompeting interventions.53 Sixteen of the 44 directcomparisons included are based on onerandomised trial, while the adjusted indirectcomparisons were based on a median of 19 trials(range 2–86). Such a large amount of dataavailable for adjusted indirect comparisons couldbe used to strengthen conclusions based on thedirect comparisons, especially when there areconcerns about the methodological quality of thesingle direct comparison trial.

Results of the direct and the adjusted indirectcomparison could be quantitatively combined toincrease statistical power or precision, when thereis no important discrepancy between the twoestimates. The statistically non-significant relativeeffect estimated by the direct comparison maybecome statistically significant by combining thedirect and the adjusted indirect estimate, forexample, in two of the 44 meta-analyses (Figure 5).

It is also possible that the significant relative effectestimated by the direct comparison would becomenon-significant after being combined with theadjusted indirect estimate. H2RA versus sucralfatefor non-ulcer dyspepsia from a Cochranesystematic review provides an example.108 In thisexample, the direct comparison based on onerandomised trial found that H2RA was lesseffective than sucralfate [relative risk (RR) 2.74,95% CI 1.25 to 6.02], while the adjusted indirectcomparison indicates that H2RA was as effective assucralfate (RR 0.99, 95% CI 0.47 to 2.08). Thediscrepancy between the direct and the adjusted


48

indirect estimate is statistically significant(z = 1.84). The combination of the direct andadjusted indirect estimate provided a statisticallynon-significant relative risk of 1.56 (95% CI 0.93to 2.75). In cases like this, one may questionwhether it is appropriate to combine the twoestimates. If it is inappropriate, then one needs toinvestigate sources of significant discrepancy anddetermine which estimate is more believable.

SummaryForty-four analyses from 28 published meta-analyses were available to compare competinginterventions both directly and indirectly. Therewere considerable statistical discrepancies between the direct and the indirect estimate, but the direction of such discrepancy wasunpredictable. The relative efficacy may be

overestimated or underestimated by the indirectcomparison compared with results of the directcomparison.

The naive indirect comparison should be avoidedowing to its unpredictable nature and very highfrequency of statistically significant discrepanciesbetween the direct and the naive indirect estimate(11/43). In contrast, the adjusted indirect methodhas two advantages: partially taking intoconsideration patient prognostic characteristicsand wider confidence intervals. Empiricalevidence presented in this chapter has confirmedthese theoretical advantages. There is nostatistically significant discrepancy between thedirect and the adjusted indirect estimate in mostcases (41/44). When direct evidence is available butnot sufficient, the direct and the adjusted indirectestimate could be combined to obtain a moreprecise estimate.


49


–2

–1

0

1

2T

reat

men

t effe

ct(9

5% c

onfid

ence

inte

rval

)

Direct estimateAdjusted indirectCombined estimate

Soo et al. Trindade et al.

FIGURE 5 Combining the direct and the adjusted indirect estimates in two meta-analyses

This chapter first examines five comparisonsfrom the sample of meta-analyses included in

Chapter 6.27,40,107–109 The reason for selecting thefive cases is that they showed a statisticallysignificant discrepancy (z > 1.64) between thedirect and the adjusted indirect estimate. It maybe interesting to note that four of these fiveexamples were about the treatment with H2RA orPPI.40,107–109

Following this, further empirical evidence isprovided about indirect comparisons, using asystematic review of antimicrobial prophylaxis incolorectal surgery.3

Five cases with significantdiscrepancyCase 1: Chiba and colleagues (1997)40

This meta-analysis evaluated speed of healing andsymptom relief in grade II–IV GORD. The reviewincluded only randomised trials but did notdirectly compare different interventions, althoughit has presented sufficient data for both direct andindirect comparison of H2RA and PPI.

By pooling data from 13 trials that comparedH2RA and PPI directly, the healing proportion inpatients receiving H2RA was lower than that inpatients receiving PPI (RR 0.56, 95% CI 0.48 to0.66). The adjusted indirect comparison betweenH2RA and PPI was carried out using 11 trials thatcompared H2RA versus placebo and two trials thatcompared PPI versus placebo (Figure 6). Theadjusted indirect estimate (RR 0.26, 95% CI 0.14to 0.48) was statistically significantly different fromthe direct estimate (z = 2.35). However, there maybe no important clinical implication associatedwith this statistically significant discrepancy. Thediscrepancy may be quantitative rather thanqualitative since both the direct and adjustedindirect estimates are in the same direction. In thisexample, the naive indirect estimate is alsostatistically significantly different from the directestimate (z = –2.95) (Figure 6a).

Figure 6(b) presents results of each interventiongroup from trials used in the direct and indirectcomparisons of H2RA and PPI. Patients receiving

placebo had worse outcome in two PPI trials(12.0%) than those receiving placebo in 11 H2RAtrials (35.3%). One explanation for thisobservation may be that the patients in the twoPPI trials were more severe than those in theH2RA trials. However, further investigation isimpossible because detailed data on patientcharacteristics were not available.

Case 2: Rostom and colleagues (2000)107

This systematic review assessed the effectiveness ofinterventions for the prevention of NSAID-induced upper gastrointestinal toxicity. Theinterventions of interest include misoprostol,H2RA and PPI. Although the authors of thisreview did not make any indirect comparisons, thedata from the individual studies are sufficient tocompare PPI and H2RA both directly andindirectly. In a trial that directly compared PPIand H2RA in reducing total endoscopic ulcers,significant difference in favour of PPI wasobserved (RR 0.28, 95% CI 0.15 to 0.51). Theadjusted indirect comparison was conducted usingthree trials that compared PPI versus placebo andsix trials that compared H2RA versus placebo.Significant discrepancy was observed between thedirect estimate and the adjusted indirect estimate,and between the direct and the naive indirectestimate (Figure 7).

In this review the authors concluded that low-doseH2RA was less effective than high-dose H2RA.Because low dose H2RA was used in the trial ofdirect comparison, the indirect comparisons werealso conducted after separating H2RA trials intosubsets of high dose and low dose (according toauthors’ original definition). Results of thesensitivity analysis indicate that the discrepancybetween the direct and the adjusted indirectestimate was reduced but still statisticallysignificant when low-dose H2RA was comparedwith PPI. The naive indirect estimate in thisexample is also significantly different from thedirect estimate. According to the naive indirectestimate, there is no significant difference in totalendoscopic ulcers between the PPI and H2RAtreatment.

Figure 7(b) presents the results of each interventiongroup from the trials included. It can be seen that


51


Chapter 7

Detailed case studies

patients receiving PPI had better outcome whilepatients receiving H2RA had worse outcome in thetrial that directly compared PPI and H2RA,compared with patients receiving the sameintervention in placebo-controlled trials.Table 12 presents some characteristics of studiesinvolved in the direct and indirect comparisons. Itseems that patients in PPI trials and in H2RA trialswere similar. The results of individual trials andmeta-analyses are shown in Figure 8. There was nosignificant heterogeneity across trials for any set oftrials. Thus, the causes of the statisticallysignificant discrepancy were unknown.

The observed statistical discrepancy between thedirect and adjusted indirect estimate may not beclinically important in this case. The results of the

direct and adjusted indirect method bothsuggested that PPI was superior to H2RA. Sincethe direct estimate was based on only one trial thatwas not double blinded, the relative efficacy of PPIversus H2RA in preventing NSAID-inducedendoscopic ulcers may not be as great as had beensuggested by the direct estimate.

Case 3: Soo and colleagues (2000)108

This Cochrane systematic review evaluatedpharmacological interventions for non-ulcerdyspepsia. The authors attempted indirectcomparisons of different drugs and used multipleregression in these (adjusted) indirectcomparisons. Only one comparison had sufficientdata for both a direct and an adjusted indirectestimate, and this is discussed below.


52

100

80

60

40

20

0

Hea

ling

prop

ortio

n (%

)

49.6

84.0

56.9

35.3

78.1

12.0

H2RA vs PPI H2RA vs placebo PPI vs placebo

0.1 1

RR (95% CI)

10

13 (731/884)

11 (1817/959) vs 2 (334/75)

11 (1817/) vs 2 (334)(a)

(b)

FIGURE 6 Chiba et al.:40 H2RA versus PPI for GORD (healing proportion). (a) Solid circle, direct estimate; square, adjusted indirect;cross, naive indirect. The numbers shown in the figure are the number of trials (patients). (b) Results of each intervention group fromtrials used in the direct and indirect comparison of H2RA and PPI.

One trial compared H2RA and sucralfate directlyand observed a significant difference in globalsymptom assessment between the twointerventions (RR 2.74, 95% CI 1.25 to 6.02). Theadjusted indirect comparison involved eightplacebo-controlled H2RA trials and two placebo-controlled sucralfate trials (Figure 9). The adjustedindirect estimate indicates no difference betweenH2RA and sucralfate (RR 0.99, 95% CI 0.47 to2.08). The discrepancy between the direct estimateand the adjusted indirect estimate is statisticallysignificant (z = 1.85) (Figure 10).

Figure 10(b) indicates that the result of sucralfatearm in the direct trial is extremely good comparedwith trials used in the adjusted indirectcomparison. This cannot be explained easily byfactors such as different underlying risk because

the result of H2RA arm in the direct trial wassimilar to that in the trials involved in the adjustedindirect comparison. The scrutiny ofcharacteristics of studies in Table 13 provided noobvious explanation for the observed discrepancybetween the direct and indirect estimates andheterogeneity among trials. In this case, thediscrepancy between the direct and adjustedindirect comparison was clinically important. Theresult of the direct comparison by the small andopen trial may not be more believable than theresult of the adjusted indirect comparison.

Case 4: van Pinxteren and colleagues(2000)109

This Cochrane systematic review evaluated short-term treatment for GORD-like symptoms. For thecomparison of empirical treatment of heartburn


53


FIGURE 7 Rostom et al.:107 PPI versus H2RA for preventing chronic NSAID-induced upper gastrointestinal toxicity (endoscopic ulcers).(a) Solid circle, direct estimate; square, adjusted indirect; cross, naive indirect. The numbers shown in the figure are the number oftrials (patients). (b) Results of each intervention group from trials used in the direct and indirect comparison of H2RA and PPI.

RR (95% CI)

0.1 1 10

1 (210/215)

3 (443/331) vs 6 (645/634)

3 (443) vs 6 (645)

Indirect: adequate H2RA

Indirect: low H2RA

(a)

(b)

60

50

40

30

20

10

0

5.7

20.5

11.1

29.6

10.9

20.2

PPI vs H2RA PPI vs placebo H2RA vs placebo

Tot

al e

ndos

copi

c ul

cers

(%)

remission with PPI or H2RA, the direct estimate(RR 0.67, 95% CI 0.57 to 0.80) was statisticallysignificantly different from the adjusted indirectestimate (RR 0.45, 95% CI 0.31 to 0.66). Thisstatistical discrepancy may have no actual clinicalimportance in this review, since both estimatesindicated a significant benefit of PPI versus H2RAin the empirical treatment for heartburn remission(Figure 11).

Case 5: Zhang and Li-Wan-Po (1996)27

This meta-analysis evaluated the analgesic efficacyof paracetamol plus codeine in surgical pain. (Theauthor, WY Zhang, supplied data for individualstudies that were not available in the publishedarticle.) The direct comparison indicated thatparacetamol plus codeine was more efficacious

that paracetamol alone (mean difference in thesum of pain intensity difference 6.97, 95% CI 3.56to 10.37). The adjusted indirect comparison didnot show significant difference betweenparacetamol plus codeine and paracetamol alone(mean difference –1.16, 95% CI –6.95 to 4.64)(Figure 12).

Results of each intervention group from trialsinvolved in the direct and the indirectcomparisons are presented in Figure 12(b). Theimprovement in pain intensity was much less inpatients receiving placebo in placebo-controlledtrials of paracetamol plus codeine than in placebo-controlled trials of paracetamol alone. It suggeststhat patients in the placebo-controlled trials ofparacetamol plus codeine may be different to


54

Review: The validity of indirect comparisons for estimating the relative efficacy of competing interventionsComparison: 01 PPI vs H2RA for preventing NSAID-induced upper gastrointestinal ulcers (Rostom et al., 2000)Outcome: 01 Total endoscopic ulcers (3–12 months)

Studyor subcategory

Arm 1n/N

Arm 2n/N

RR (random)95% CI

RR (random)95% CI

01 PPI vs H2RA YeomansSubtotal (95% CI)Total events: 12 (Arm 1), 44 (Arm 2)Test for heterogeneity: not applicableTest for overall effect: Z = 4.10 (p < 0.0001)

02 PPI vs placebo Cullen Ekstrom HawkeySubtotal (95% CI)Total events: 49 (Arm 1), 98 (Arm 2)Test for heterogeneity: χ2 = 0.61, df = 2 (p = 0.74), I2 = 0%Test for overall effect: Z = 7.16 (p < 0.00001)

03 H2RA (high dose) vs placebo Hudson Taha Ten WoldeSubtotal (95% CI)Total events: 22 (Arm 1), 53 (Arm 2)Test for heterogeneity: χ2 = 0.39, df = 2 (p = 0.82), I2 = 0%Test for overall effect: Z = 4.03 (p < 0.0001)

04 H2RA (low dose) vs placebo Ehsanullah Levine TahaSubtotal (95% CI)Total events: 48 (Arm 1), 75 (Arm 2)Test for heterogeneity: χ2 = 0.38, df = 2 (p = 0.83), I2 = 0%Test for overall effect: Z = 2.67 (p = 0.007)

12/210 44/215 210 215

3/83 14/85 4/86 15/9142/274 69/155 443 331

10/39 21/39 9/97 24/93 3/15 8/15 151 147

10/151 17/14624/248 34/24814/95 24/93 494 487

0.28 (0.15 to 0.51)0.28 (0.15 to 0.51)

0.22 (0.07 to 0.74)0.28 (0.10 to 0.82)0.34 (0.25 to 0.48)0.33 (0.24 to 0.45)

0.48 (0.26 to 0.87)0.36 (0.18 to 0.73)0.38 (0.12 to 1.15)0.42 (0.27 to 0.64)

0.57 (0.27 to 1.20)0.71 (0.43 to 1.15)0.57 (0.32 to 1.03)0.63 (0.45 to 0.88)

0.1 0.2 0.5 1 2 5 10 Favours arm 1 Favours arm 2

FIGURE 8 Results of individual trials involved in the direct and indirect comparisons in the meta-analysis by Rostom et al.107

those in the placebo-controlled trials ofparacetamol alone. The possible difference inbaseline risk is taken into consideration in theadjusted indirect comparison, which indicates thatthere is no difference between paracetamol pluscodeine and paracetamol alone. When thedifferent baseline risk has not been considered,the naïve indirect comparison yields a result that isopposite to the direct estimate (Figure 12a).

In this case, the significant discrepancy may bedue to different types of surgical pain and/ordifferent doses of paracetamol/codeine used indifferent trials. When the analysis was restricted totrials of dental surgery, the discrepancy betweenthe direct and the adjusted indirect estimate wasstill significant (Table 14). Further scrutiny of theincluded trials found that the majority of the trials(n = 10) in the direct comparison used a dose of600–650 mg for paracetamol and 60 mg forcodeine, whereas many placebo-controlled trials

(n = 29) used a dose of 300 mg for paracetamoland 30 mg for codeine. When the analysisincluded only trials that used paracetamol600–650 mg and codeine 60 mg, the adjustedindirect estimate (5.72, 95% CI –5.37 to 16.81)was no longer significantly different from thedirect estimate (7.28, 95% CI 3.69 to 10.87). Thus,the significant discrepancy between the direct andthe indirect estimate based on all trials could beexplained by the fact that many placebo-controlledtrials used low doses of paracetamol (300 mg) andcodeine (30 mg). This example suggests that thesimilarity of trials involved in the adjusted indirectcomparison should be carefully assessed.

Relative efficacy of antimicrobialprophylaxis in colorectal surgery(Note: this section is based on an article publishedin Controlled Clinical Trials.87)


55


Review: The validity of indirect comparisons for estimating the relative efficacy of competing interventionsComparison: 02 H2RA vs sucralfate for non-ulcer dyspepsia (Soo et al., 2000)Outcome: 01 Global symptom assessment at the end of treatment

01 H2RA vs sucralfate MisraSubtotal (95% CI)Total events: 17 (Arm 1), 7 (Arm 2)Test for heterogeneity: not applicableTest for overall effect: Z = 2.51 (p = 0.01)

02 H2RA vs placebo Delattre Gotthard Hadi Hansen Kelbaek Nesland Saunders SingalSubtotal (95% CI)Total events: 199 (Arm 1), 304 (Arm 2)Test for heterogeneity: χ2 = 27.85, df = 7 (p = 0.0002), I2 = 74.9%Test for overall effect: Z = 2.25 (p = 0.02)

03 Sucralfate vs placebo Gudjonsson KairaluomaSubtotal (95% CI)Total events: 34 (Arm 1), 46 (Arm 2)Test for heterogeneity: χ2 = 3.19, df = 1 (p = 0.07), I2 = 68.7%Test for overall effect: Z = 0.99 (p = 0.32)

2.74 (1.25 to 6.02)2.74 (1.25 to 6.02)

0.53 (0.40 to 0.69)0.74 (0.53 to 1.04)0.03 (0.00 to 0.43)1.20 (0.88 to 1.64)1.19 (0.62 to 2.29)0.75 (0.53 to 1.06)0.50 (0.32 to 0.78)0.57 (0.32 to 0.99)0.70 (0.52 to 0.96)

1.03 (0.57 to 1.86)0.51 (0.32 to 0.83)0.71 (0.36 to 1.40)

0.1 0.2 0.5 1 2 5 10 Favours arm 1 Favours arm 2

Studyor subcategory

Arm 1n/N

Arm 2n/N

RR (random)95% CI

RR (random)95% CI

17/47 7/53 47 53

54/209 102/20929/63 34/55 0/26 17/2551/111 42/11011/24 10/2623/44 32/4621/103 48/11810/271 9/29 607 618

16/50 14/4518/79 32/72 129 117

FIGURE 9 Results of individual trials involved in the direct and indirect comparisons in the meta-analysis by Soo et al.108

In a systematic review of antibiotic prophylaxis forpreventing surgical wound infection aftercolorectal surgery,3 147 randomised trials wereidentified in which more than 70 differentantibiotics or combinations of antibiotics wereassessed. Only a limited number of antibiotics wasdirectly compared within the trials. If all 70options had been directly compared, over 2400trials would have been required, withoutconsidering different dosages, routes and timingof administration of the same drug.

Using this systematic review as an example, thissection aims to explore the potential usefulnessand limitations of indirect comparison inevaluating the relative efficacy of competinginterventions.

MethodFrom the systematic review of antimicrobialprophylaxis in colorectal surgery 11 sets of trials

were identified in which different antibiotics couldbe compared both directly and indirectly.3 Eachset of trials contains at least three trials that testedthree different antibiotics (or combinations ofantibiotics). For example, suppose a set of trialsincludes a trial that compared antibiotic A with B,a trial that compared A with C, and a trial thatcompared B with C. Then the trials in this setcould be used for three different comparisons: A versus B, A versus C and B versus C.

For each comparison the relative efficacy ofantimicrobial prophylaxis (i.e. odds ratio forsurgical wound infection) was estimated usingthree different methods: direct comparison,adjusted indirect comparison and naive indirectcomparison. The method suggested by Bucherand colleagues was used to carry out the adjustedindirect comparison.14 Results from more than onetrial were weighted by the inverse ofcorresponding variances and then quantitatively


56

RR (95% CI)

0.1 1 10

1 (47/53)

8 (607/618) vs 2 (129/117)

8 (607) vs 2 (129)

(a)

(b)

H2RA vs Sucr H2RA vs placebo Sucr vs placebo

60

50

40

30

20

10

0

Patie

nts

with

mod

erat

e or

seve

re d

yspe

psia

(%)

36.2

13.2

32.8

49.2

26.4

39.3

FIGURE 10 Soo et al.:108 H2RA versus sucralfate (Sucr) for non-ulcer dyspepsia (global symptom assessment). (a) Solid circle, directestimate; square, adjusted indirect; cross, naive indirect. The numbers shown in the figure are the number of trials (patients). (b) Results of each intervention group from trials used in the direct and indirect comparison of H2RA and sucralfate.

pooled to obtain an overall estimate. The naiveindirect method compared the results of singlearms included in the trials that had been used inthe adjusted indirect comparison. In the naiveindirect comparison, when relevant the results ofmore than one trial were pooled by adding up thenumber of surgical wound infections and thenumber of patients.

As an example, Table 15 presents data from a set ofthree trials that could be used to conduct threedifferent comparisons: cefuroxime plusmetronidazole (Cefur-M) versus co-amoxiclav (Co-A),110 cefuroxime plus metronidazole (Cefur-M)versus cefotaxime plus metronidazole (Cefot-M)111

and co-amoxiclav versus cefotaxime plusmetronidazole.112 Table 16 shows the results ofdifferent analyses for each of the three

comparisons. For instance, Cefur-M and Co-Awere directly compared in the trial conducted byPalmer and colleagues.110 The indirectcomparison of Cefur-M and Co-A was based onthe other two trials that included the commonintervention Cefot-M.111,112 Likewise, the directcomparison of Cefur-M and Cefot-M was based onthe results obtained by Rowe-Jones andcolleagues,111 and the indirect comparison wasbased on the two trials with a commonintervention, Co-A.110,112

The results of the two indirect methods werecompared with the results from the gold standardof direct comparison. In this example, thediscrepancy was defined as the absolute value ofdifference in log odds ratio between the directmethod and the indirect method. To take into


57


0

10

20

30

40

50

60

70

80

Hea

rtbu

rn r

emiss

ion

(%)

PPI vs H2RA H2RA vs placeboPPI vs placebo

46.7

66.1

24.2

69.8

45.4

59.4

(b)

(a)

0.1 1 10

3 (1228/664)

1 (161/159) vs 2 (511/502)

1 (161) vs 2 (511)

RR (95% CI)

FIGURE 11 van Pinxteren et al.:109 PPI versus H2RA for reflux disease-like symptoms (heartburn remission). (a) Solid circle, directestimate; square, adjusted indirect; cross, naive indirect. The numbers shown in the figure are the number of trials (patients). (b) Results of each intervention group from trials used in the direct and indirect comparison of H2RA and PPI.

account the precision of estimated discrepancy, the corresponding z statistic was calculated bydividing the difference in log odds ratio by thesquare root of the sum of the variances. The cut-off point for statistical significance wasarbitrarily considered to be z = 1.64 (or two-sidedp-value = 0.10).

The difference in log odds ratio can be convertedinto the ratio of odds ratios (i.e. the antilogarithmof difference in log odds ratio). For example, whenthe difference in log odds ratio is 1.0, thecorresponding ratio of odds ratios is 2.72. It canbe mathematically shown that the absolute valueof discrepancy between the adjusted indirectmethod and the direct method will be the samefor three different comparisons using the same setof trials. For example, it was 1.636 for all threecomparisons in Table 16 (this example correspondsto trial set 5 in Figure 13).

ResultsThe results presented in Figure 13 indicate thatconsiderable discrepancies exist between the directand indirect comparisons. In each set of trials, thediscrepancies between the direct and the adjustedindirect method were the same in three differentcomparisons. In contrast, the discrepanciesbetween the direct and the naive indirectcomparisons were unpredictable and variedgreatly across the different comparisons.

The discrepancies between the direct and thenaive indirect comparisons may be either smalleror greater than those between the direct and theadjusted indirect comparisons (Figure 13),depending on which interventions have beencompared using a given set of trials. In nine of the11 sets of trials, there is a naive indirectcomparison with a greater discrepancy than theadjusted indirect comparison. The significant


58

22.63

9.6712.74

4.12

17.01

24.00

Para + Code vs Para

Para + Code vs Placebo

Para vs Placebo

40

35

30

25

20

25

21

5

0

Mea

n di

ffere

nce

(sum

of d

iffer

ence

in p

ain

inte

nsity

)

Mean difference (95% CI)

–15 –10 –5 0 5 10

13

12 vs 31

12 vs 31(a)

(b)

FIGURE 12 Zhang and Li Wan Po:27 paracetamol (Para) plus codeine (Code) versus paracetamol alone in surgical pain (analgesicefficacy). (a) Solid circle, direct estimate; square, adjusted indirect; cross, naïve indirect. The numbers shown in the figure are thenumber of trials (patients). (b) Results of each intervention group from used in the direct and indirect comparison of paracetamol pluscodeine and paracetamol alone.


59


TA

BLE

12

Char

acte

ristic

s of

stu

dies

invo

lved

in t

he d

irect

and

the

indi

rect

com

paris

ons

of P

PI a

nd H

2RA

for p

reve

ntin

g N

SAID

-indu

ced

uppe

r gas

troi

ntes

tinal

tox

icity

(to

tal e

ndos

copi

c ul

cers

)107

Stud

yIn

terv

enti

on (

dura

tion

)Pa

rtic

ipan

tsQ

ualit

ya

Yeom

ans,

199

8O

mep

razo

le 2

0 m

g pe

r da

yPa

tient

s w

ith R

A a

nd O

A w

ho n

eede

d m

aint

enan

ce t

reat

men

t af

ter

succ

essf

ul

Jada

d sc

ale:

3Ra

nitid

ine

2×

150

mg

trea

tmen

t of

ulc

ers

or e

rosio

ns in

eith

er t

he s

tom

ach

or d

uode

num

Allo

catio

n (6

mon

ths)

Mea

n ag

e 56

yea

rsco

ncea

lmen

t: D

Hel

icon

acte

r pyl

ori:

posit

ive:

50%

Cul

len,

199

8O

mep

razo

le 2

0 m

g pe

r da

yPa

tient

s ta

king

NSA

IDs

regu

larly

, chr

onic

ally

and

abo

ve d

efin

ed m

inim

um d

oses

Jada

d sc

ale:

2Pl

aceb

oA

lloca

tion

(6 m

onth

s)co

ncea

lmen

t: D

Ek

stro

m, 1

996

Om

epra

zole

20

mg

per

day

Patie

nts

with

dys

peps

ia o

r hi

stor

y of

pep

tic u

lcer

dise

ase

Jada

d sc

ale:

3Pl

aceb

oM

ean

age

58 y

ears

Allo

catio

n (3

mon

ths)

Prev

ious

pep

tic u

lcer

: 27%

conc

ealm

ent:

D

Hel

icob

acte

r pyl

orip

ositi

ve: 5

3%H

awke

y, 1

998

Om

epra

zole

20

mg

Patie

nt w

ith R

A a

nd O

A. M

aint

enan

ce t

hera

py fo

r pa

tient

s in

who

m t

reat

men

t w

as

Jada

d sc

ale:

3M

isopr

osto

l 400

�g

succ

essf

ulA

lloca

tion

Plac

ebo

Out

patie

nts

conc

ealm

ent:

D(6

mon

ths)

Mea

n ag

e 58

yea

rsLe

ngth

of N

SAID

s: >

6 m

onth

sPr

evio

us p

eptic

ulc

ers:

29%

Hel

icob

acte

r pyl

orip

ositi

ve: 4

2%H

udso

n, 1

997

Fam

otid

ine

40 m

g b.

d. (h

igh

dose

)Pa

tient

s w

ith R

A o

r O

A. N

SAID

use

rs w

ith h

eale

d ul

cers

Jada

d sc

ale:

3Pl

aceb

oH

elic

obac

ter p

ylor

ipos

itive

: 18/

39A

lloca

tion

(6 m

onth

s)co

ncea

lmen

t: D

Ta

ha, 1

996

Fam

otid

ine

20 m

g b.

d. (l

ow d

ose)

Patie

nts

with

out

pept

ic u

lcer

s w

ho w

ere

rece

ivin

g lo

ng-t

erm

NSA

ID t

hera

py fo

r Ja

dad

scal

e: 4

Fam

otid

ine

40 m

g b.

d. (h

igh

dose

)RA

(82%

) or

OA

Allo

catio

n Pl

aceb

oco

ncea

lmen

t: D

(6

mon

ths)

Ten

Wol

de, 1

996

Rani

tidin

e 30

0 b.

d. (h

igh

dose

)Pa

tient

s w

ith R

A a

nd a

hist

ory

of p

eptic

ulc

er d

iseas

eJa

dad

scal

e: 3

Plac

ebo

Allo

catio

n (T

he s

tudy

was

sto

pped

afte

r a

conc

ealm

ent:

D

blin

ded

inte

rim a

naly

sis)

Ehsa

nulla

h, 1

988

Rani

tidin

e 15

0 m

g b.

d. (l

ow d

ose)

Patie

nts

with

RA

or

OA

age

d >

18 y

ears

with

out

lesio

ns in

the

sto

mac

h an

d Ja

dad

scal

e: 5

Plac

ebo

duod

enum

at

base

line

endo

scop

y (a

fter

1 w

eek

with

out

taki

ng N

SAID

s).

Allo

catio

n (2

mon

ths)

Tho

se t

akin

g ot

her

antir

heum

atic

age

nts,

con

com

itant

ulc

erog

enic

dru

gs o

r tr

eatm

ent

conc

ealm

ent:

D

for

pept

ic u

lcer

s w

ithin

the

pre

viou

s 30

day

s w

ere

excl

uded

Mea

n ag

e 57

yea

rsLe

vine

, 199

3N

izat

idin

e 15

0 m

g b.

d.Pa

tient

s w

ith O

A w

ho w

ere

taki

ng N

SAID

s. E

ndos

copy

to

rule

out

the

pre

senc

e Ja

dad

scal

e: 3

Plac

ebo

of a

n ac

ute

ulce

r A

lloca

tion

(3 m

onth

s)co

ncea

lmen

t: D

aA

s co

ded

in t

he o

rigin

al r

evie

w.10

7

OA

, ost

eoar

thrit

is; R

A, r

heum

atoi

d ar

thrit

is.


60 TA

BLE

13

Char

acte

ristic

s of

stu

dies

invo

lved

in t

he d

irect

and

the

indi

rect

com

paris

ons

of H

2RA

and

sucr

alfa

te fo

r non

-ulc

er d

yspe

psia

108

Stud

yIn

terv

enti

ons

Part

icip

ants

Qua

litya

Misr

a, 1

992

Rani

tidin

e 15

0 m

gPa

tient

s w

ith 1

mon

th o

f abd

omin

al s

ympt

oms

refe

rabl

e to

the

upp

er G

I tra

ct.

Rand

omise

d, o

pen,

(In

dia)

Sucr

alfa

te 1

g q

.d.

Patie

nts

with

GO

RD o

r IB

S w

ere

excl

uded

. 87%

com

plet

ed t

rial

cont

rolle

d cl

inic

al t

rial

(4 w

eeks

) A

lloca

tion

conc

ealm

ent:

BD

elat

tre,

198

5C

imet

idin

e 20

0 m

g q.

d.Pa

tient

s w

ith n

on-u

lcer

dys

peps

ia

RCT,

dou

ble-

blin

d,

(USA

)Pl

aceb

o pl

aceb

o-co

ntro

lled,

(4

wee

ks)

mul

ticen

tre

tria

l A

lloca

tion

conc

ealm

ent:

B G

otth

ard,

198

8C

imet

idin

e 40

0 m

g b.

d.3/

12 o

f dys

peps

ia o

f unk

now

n or

igin

. Aci

d ou

tput

stu

dies

per

form

ed.

RCT,

dou

ble-

blin

d,

(Sw

eden

)A

ntac

id 1

0 m

l q.d

. 16

% d

uode

nitis

pl

aceb

o-co

ntro

lled

tria

l Pl

aceb

oA

lloca

tion

conc

ealm

ent:

B (6

wee

ks)

Had

i, 19

89Ra

nitid

ine

300

mg

daily

Dur

atio

n of

dys

peps

ia u

ncle

ar. G

astr

itis

on a

ll O

GD

. Dro

pout

rat

e fo

r pl

aceb

o RC

T, d

oubl

e-bl

ind,

pla

cebo

-(In

done

sia)

Plac

ebo

was

23%

and

for

rani

tidin

e w

as 4

%

cont

rolle

d tr

ial

(4 w

eeks

)A

lloca

tion

conc

ealm

ent:

B H

anse

n, 1

998

Cisa

prid

e 10

mg

t.d.

Prim

ary

care

rec

ruitm

ent.

Mea

n du

ratio

n of

dys

pept

ic s

ympt

om w

as 8

8 m

onth

s.

RCT,

dou

ble-

blin

d, p

lace

bo-

(Den

mar

k)N

izat

idin

e 30

0 m

g no

cte

Four

sub

grou

ps: u

lcer

-like

(13%

), re

flux-

like

(23%

), dy

smot

ility

-like

(46%

) and

co

ntro

lled

tria

l Pl

aceb

o un

clas

sifie

d (1

8%).

Incl

uded

sup

erfic

ial e

rosio

ns o

n O

GD

. 85%

com

plet

ed t

rial

Allo

catio

n co

ncea

lmen

t: A

(2

wee

ks)

Kelb

aek,

198

5C

imet

idin

e 20

0 m

g t.d

. and

Pr

imar

y ca

re r

ecru

itmen

t. Pa

tient

s w

ith 1

mon

th o

f epi

gast

ric p

ain.

Aci

d ou

tput

RC

T, d

oubl

e-bl

ind,

pla

cebo

-(D

enm

ark)

400

mg

noct

e st

udie

s pe

rfor

med

. Had

OG

D. 1

4 pa

tient

s w

ho w

ere

sym

ptom

free

at

end

of

cont

rolle

d tr

ial

Plac

ebo

trea

tmen

t ha

d 3

mon

ths

of fo

llow

-up.

96%

com

plet

ed t

rial

Allo

catio

n co

ncea

lmen

t: B

(3 w

eeks

)N

esla

nd, 1

985

Cim

etid

ine

400

mg

b.d.

Patie

nts

with

6 m

onth

s of

pre

dom

inan

tly u

lcer

-like

pai

n w

ith e

rosiv

e pr

epyl

oric

RC

T, d

oubl

e-bl

ind,

pla

cebo

-(N

orw

ay)

Plac

ebo

chan

ges.

90%

com

plet

ed t

rial

cont

rolle

d tr

ial

(4 w

eeks

)A

lloca

tion

conc

ealm

ent:

BSa

unde

rs, 1

986

Rani

tidin

e 15

0 m

g b.

d.

Prim

ary

care

rec

ruitm

ent.

88%

com

plet

ed t

rial.

One

-yea

r fo

llow

-up,

but

the

RC

T, d

oubl

e-bl

ind,

pla

cebo

-(U

K)

Plac

ebo

resu

lts in

clud

ed o

ther

pep

tic d

iseas

e co

ntro

lled

mul

ticen

tre

tria

ls (6

wee

ks)

Allo

catio

n co

ncea

lmen

t: A

Si

ngal

, 198

9C

imet

idin

e 40

0 m

g b.

d.Pa

tient

s w

ith 1

mon

th o

f prim

ary

sym

ptom

of u

pper

abd

omin

al d

iscom

fort

. RC

T, d

oubl

e-bl

ind,

pla

cebo

-(In

dia)

Plac

ebo

IBS

excl

uded

co

ntro

lled

tria

l (4

wee

ks)

Allo

catio

n co

ncea

lmen

t: B

Gud

jons

son,

199

3 Su

cral

fate

1 g

q.d

.Pr

ivat

e pr

actic

e re

crui

tmen

t. Pa

tient

s w

ith 1

mon

th o

f dys

peps

ia. H

ad O

GD

. RC

T, d

oubl

e-bl

ind,

pla

cebo

(Ic

elan

d)

Plac

ebo

91%

com

plet

ed t

rial

cont

rolle

d tr

ial

(3 w

eeks

)A

lloca

tion

conc

ealm

ent:

BKa

iralu

oma,

198

7Su

cral

fate

1 g

t.d

. Pa

tient

s w

ith 3

mon

ths

of d

yspe

psia

. Had

OG

D. 6

% h

ad d

uode

nitis

. 86%

RC

T, d

oubl

e-bl

ind,

pla

cebo

-(F

inla

nd)

Plac

ebo

com

plet

ed t

rial

cont

rolle

d tr

ial

(4 w

eeks

)A

lloca

tion

conc

ealm

ent:

D

aA

s co

ded

in t

he o

rigin

al r

evie

w10

8

GI,

gast

roin

test

inal

; IBS

, irr

itabl

e bo

wel

syn

drom

e; O

GD

, oes

opha

goga

stro

duod

enos

copy

.


61


TABLE 14 Direct and adjusted indirect estimates of efficacy of paracetamol plus codeine versus paracetamol alone for surgical pain27

Trials Mean difference in standardised score for the sum of pain Discrepancy intensity difference (95% CI) between the

direct and adjusted Direct comparison Adjusted indirect comparison indirect estimates

All trials 6.97 (3.56 to 10.37) –1.16 (–6.95 to 4.64) 8.13 (p = 0.018)n = 13 n1 = 12, n2 = 31

Dental surgery trials only 7.07 (3.37 to 10.78) –1.40 (–8.27 to 5.46) 8.47 (p = 0.033)n = 11 n1 = 7, n2 = 15

Trials that used paracetamol 7.28 (3.69 to 10.87) 5.72 (–5.37 to 16.81) 1.56 (p = 0.793)600–650 mg and codeine n = 10 n1 = 2, n2 = 1260 mg

n, n1 and n2 are the number of trials used in the direct comparison and indirect comparison, respectively. Trials included dental surgery, episiotomy, postpartum uterine cramp, orthopaedic and other surgery. Doses of paracetamolranged from 300 to 1000 mg and the dose of codeine was 30 or 60 mg.

TABLE 15 Example: a set of trials that could be used to compare different antibiotic regimens both directly and indirectly forpreventing surgical wound infection in colorectal surgery

Trial Cefur-M Co-A Cefot-M SWI/n SWI/n SWI/n

Palmer et al., 1994110 2/79 8/69 –

Rowe-Jones et al., 1990111 33/454 – 32/453

Kwok et al., 1993112 – 7/76 8/88

n, number of patients; SWI, surgical wound infection.

TABLE 16 Example: results of different methods for each comparison between antibiotics, using the trials presented in Table 15

Comparison Direct method Naive indirect method Adjusted indirect methodlnOR (95% CI) lnOR (95% CI) lnOR (95% CI)

Cefur-M vs Co-A –1.619 (–3.205 to –0.034) –0.258 (–1.112 to 0.596) 0.016 (–1.162 to 1.194)∆ = 1.361 ∆ = 1.636z = 1.481 z = 1.623

Cefur-M vs Cefot-M 0.031 (–0.474 to 0.535) –1.348 (–2.929 to 0.233) –1.605 (–3.514 to 0.305)∆ = –1.379 ∆ = –1.636z = –1.629 z = –1.623

Co-A vs Cefot-M 0.0144 (–1.050 to 1.079) 0.545 (–0.275 to 1.365) 1.650 (–0.014 to 3.314)∆ = 0.531 ∆ = 1.636z = 0.775 z = 1.623

lnOR, log odds ratio; ∆, difference in log odds ratio between the indirect method and the direct method.The z statistic was calculated by dividing the difference in log odds ratio by the square root of the sum of the variances.

discrepancies (|z| ≥ 1.64) occurred in five of the11 sets for the naive indirect comparison (45%)and two of the 11 for the adjusted indirectcomparison (18%).

The observed discrepancy between the direct andthe indirect comparison may be explained by

many possible factors, including chance error,study quality and other factors associated with theinternal and external validity of studies. Becauseof the relatively small sample size (number of trialsand patients) in many sets of trials, thediscrepancies between the direct and the indirectcomparisons may have been exacerbated in this


62

0.0(1.0)

1.0(2.7)

2.0(7.4)

3.0(20.1)

4.0(54.6)

5.0(148.4)

1 3 510

2 7 959

3 3 310

4 4 1173

5 3 1219

6 3 297

7 4 857

8 3 460

9 3 557

10 3 364

11 4 396

Tria

l set

No.

of

tria

ls

No.

of

Patie

nts

Absolute difference in log odds ratio(Ratio of odds ratios)

FIGURE 13 Absolute difference in log odds ratio between the direct and the adjusted indirect comparisons, and between the directand the naive indirect comparisons. Solid bars, difference between the direct and the adjusted indirect comparison; open bars,difference between the direct and the naive indirect comparison; dotted lines, standard error of difference in log odds ratio. Results arefrom 11 sets of trials of antimicrobial prophylaxis in colorectal surgery. For each set of trials, the absolute discrepancies by the adjustedindirect comparisons are the same and therefore only one solid bar is presented. Conversely, for each set of trials, absolutediscrepancies by three naive indirect comparisons often vary greatly.

example. In general, the discrepancy between thedirect and the adjusted indirect comparisonshould be investigated by examining the externaland internal validity of the trials involved.

SummaryFive examples from Chapter 6Although statistically significant discrepancy hasbeen observed between the direct and the adjustedindirect estimate, these examples indicate thatsuch discrepancy may not be clinically important.The direct and the adjusted indirect estimates maybe in the same direction and the difference isquantitative rather than qualitative inthree40,107,109 of the five examples. The remainingtwo examples suggest that the discrepancybetween the direct and the adjusted indirectestimate may be clinically important,27,108 in whichstatistically significant relative efficacy observed bythe direct comparison was not confirmed by theadjusted indirect comparison.

Study-level characteristics reported in thepublished meta-analyses are not sufficient fordetailed investigation of sources of significant

discrepancy between the direct and the indirectestimate of relative efficacy of competinginterventions.

Example of antimicrobial prophylaxis incolorectal surgeryFrom a systematic review of antimicrobialprophylaxis in colorectal surgery, 11 sets ofrandomised trials were identified that could beused to compare antibiotics both directly andindirectly. The discrepancy between the direct andthe indirect comparison is defined as the absolutevalue of difference in log odds ratio.

Considerable discrepancies exist between thedirect and the adjusted indirect comparisons. Theadjusted indirect comparison has the advantagesthat the prognostic factors of participants indifferent trials can be partially taken into accountand more uncertainty be incorporated into itsresult by providing a wider confidence interval.The findings of this section indicate that thediscrepancy between the direct and the naiveindirect estimate is more unpredictable and morelikely to be statistically significant than thatbetween the direct and the adjusted indirectestimate.


63


Indirect comparisons based onrandomised trials: currentpracticeIndirect comparisons are commonly used forevaluating the relative effectiveness of alternativeinterventions, with approximately 9.5% of meta-analyses of RCTs identified through DAREincluding some form of indirect comparison(Chapter 3). Even when only randomised trials areincluded, the strength of the randomisationprocedure is weakened when making indirectcomparisons because of the need to combineinformation across separate sets of trials.

Indirect comparisons are sometimes carried outimplicitly and the results interpreted as if fromdirect comparisons within randomised trials.Indeed, the findings of direct comparisons aresometimes ignored, even when data are available.The use of inferior indirect methods and theinappropriate interpretation of results of indirectcomparison may result in misleading estimates ofrelative efficacy of competing healthcareinterventions. Poor methods of analysis could evenyield results that are inferior to those obtainedthrough non-randomised studies (as discussedbelow).

The survey of published meta-analyses indicatesthat results obtained through indirect comparisonsare not always consistent with the findingsobtained by direct comparisons. Data available in28 identified meta-analyses allowed 44 analyses tobe undertaken comparing competinginterventions both directly and indirectly (Chapter6). Indirect comparisons can be broadly classifiedas either naive or adjusted. The naive approachrefers to the comparison of results of single armsbetween different trials. In the adjusted indirectcomparison, the comparison of the interventionsof interest is adjusted by the results of their directcomparison with a common control group (e.g.placebo). Empirical evidence from the 40 analysesundertaken in Chapter 6 shows the naiveapproach to be a highly unpredictable method formaking indirect comparisons, with a very highfrequency of statistically significant discrepanciesfrom the direct estimate. By contrast, for adjustedindirect comparisons there was no clear evidence

of significant differences from the direct estimatesbeyond expectation by chance. However, theamount of direct randomised evidence was ratherlimited in many of the systematic reviews in thesample.

Performance of different methodsof making indirect comparisonsMany systematic reviews were found to includeindirect comparisons, yet few reviewers performformal analyses. Perhaps this lack reflects the factthat there are few publications discussing theanalysis of such data (Chapter 4). While the validmethods make assumptions that cannot easily betested, it may be better to use these explicit formalapproaches than for reviewers (and readers) tomake such comparisons informally.

The recommended methods can all be describedas adjusted indirect comparisons, in which twotreatments are compared via their relative effectversus a common comparator. The main types ofanalysis are the simple weighted combination ofseparate estimates (e.g. as suggested by Bucherand colleagues14), meta-regression and generalisedlinear models (e.g. logistic regression). Althoughseveral examples were found of the use of a naivemethod in published reviews, no methodologicalpaper with a recommendation to use this strategywas found.

The simulation studies (Chapter 5) showed that allof these methods are unbiased and will give theright answer on average across many suchapplications. However, the correctness of thestandard error of the estimated treatment effectwill depend on strong but unverifiableassumptions. Little difference was found in theperformance of fixed effect methods for adjustedindirect comparisons, but they tended to giveconfidence intervals that were too narrow. Partlyfor this reason, and also in view of additionalstudy heterogeneity compared with the usual caseof direct evidence only, a random effects methodwill usually be appropriate if formal meta-analysisis applied (with the usual provisos regarding theapplication of such models, such as having enoughtrials).


65


Chapter 8

Discussion

The analyses described in Chapter 5 were allbased on the data from a single, large trial, with aunified protocol and hence consistent inclusioncriteria. Thus, the usual sources of heterogeneitywere not present, except for variation in location(country) and some associated case-mix variation.In general, one would expect more variationbetween independent component studiescontributing to an indirect comparison as a resultof variation in aspects of the study design. Inparticular, there may be differences between thetwo sets of trials being combined with respect tothe participants and perhaps also regarding theactual treatments given.

The naive approach, in which the numbers areadded as if there was just one trial making eachcomparison, is problematic even when consideringdirectly randomised trials, and may severelymislead owing to ‘Simpson’s paradox’.113 Forindirect comparisons, such inappropriatecombination is compounded by the discarding ofwithin-trial comparison groups, such that thewhole point of randomisation is lost. The resultsof such analysis are completely untrustworthy, andnaive comparisons should never be made.

Quality of evidence from RCTs(direct comparisons)Evidence from RCTs is, in general, considered tobe the best. However, such evidence may beimperfect for several reasons. First, the problem ofpatient comparability exists in RCTs.114 Thebaseline comparability of patients between groupsmay become problematic owing to a lack ofallocation concealment in randomised trials.115 Itis also possible that, purely by chance, the patientsrandomly allocated to different groups havedifferent prognostic characteristics, particularlywhen the sample size is small.

Second, after participants have been enrolledthere is considerable possibility that biases (such asperformance bias, attrition bias and detection bias)may be introduced into trials. For example, patientsmay drop out after randomisation for variousreasons or may be excluded from the analysis.Such withdrawal or exclusion may not be randomor balanced across groups. Therefore, even thoughthe patients in different treatment groups arecomparable at the time of randomisation, thatcomparability may disappear owing to non-random exclusion or withdrawal. In addition, lackof blinding in assessment of the outcome may leadto overestimation of treatment effects.90,115

Third, empirical evidence has confirmed thatpublished randomised trials may be a biasedsample of all trials that have been conducted,owing to publication and related biases.116 Trialswith non-significant or negative results are lesslikely to be published than trials with significant orpositive results. Recently, additional evidence hasdemonstrated the added problem of selectivereporting of outcomes.117 Therefore, evidence fromrandomised trials still needs to be interpreted withcaution and the possibility of publication biasshould be investigated and excluded if possible.

Finally, there may be considerable difference inresults among randomised trials, especially smalltrials. Empirical evidence indicates thatsubsequent large trials may sometimes overturnconclusions based on small published trials.118–120

It has been argued that “randomisation is notsufficient for comparability”.119,121 Here,comparability refers not only to baseline patientcharacteristics, but also to other studycharacteristics. It should not be assumed that theadjective ‘randomised’ is a guarantee of highquality. As a consequence, it is recommended thatmethodological quality is assessed in a systematicreview, although there is no consensus on how todo this or how to make use of the information.122

Nonetheless, the problems just described applyalso to non-RCTs, so while the methodology ofdirectly randomised trials may not always be ideal,such trials do offer the most reliable evidence oftreatment efficacy (especially if studies ofunacceptably poor quality are discounted).

Thus, when there is a substantial or modestamount of direct evidence the customary practiceof ignoring non-randomised studies and indirectrandomised evidence will generally be correct.Inevitably, however, in some circumstances theremay be few or no trials that have compareddirectly treatments of particular interest. Thus, thepossibility of using indirect evidence becomesimportant. This report has examined methods forperforming indirect comparisons, focusing onindirect evidence from RCTs. The next sectionsconsider issues relating to the use of indirectevidence when direct randomised evidence iseither insufficient or non-existent.

Similar problems may apply when performingsubgroup analyses within a meta-analysis of trialsall making the same direct comparison. If thesubgrouping factor relates to the intervention,such as the dose of active treatment, or choice ofcomparator treatment, then comparison of

Discussion

66

subgroups is an indirect comparison of exactly thesame type as discussed.

What to do when direct evidenceis available but insufficientEven when some direct randomised evidence isavailable it is often insufficient. In such cases theadjusted indirect comparison may providesupplementary information in evaluating relativeefficacy of competing interventions.53 Fourteen ofthe 40 direct comparisons in Chapter 3 are basedon one randomised trial, and a median of 18 trials(range 2–96) is incorporated in the correspondingadjusted indirect comparison (Appendix 3). Suchlarge amounts of data available for adjustedindirect comparisons could be used to attempt tostrengthen conclusions based on the directcomparisons. Results of the direct and theadjusted indirect comparison can be quantitativelycombined to increase statistical power or precision.The statistically non-significant relative effectestimated by the direct comparison may becomestatistically significant by combining the direct andthe adjusted indirect estimate, as happened inthree of the 40 comparisons (Table 11).

It is also possible that the significant relative effectestimated by the direct comparison becomes non-significant after it has been combined with theadjusted indirect estimate. H2RA versus sucralfatefor non-ulcer dyspepsia from a Cochranesystematic review is an example.108 Here, thedirect comparison based on one randomised trialfound that H2RA was less efficacious thansucralfate (RR 2.74, 95% CI 1.25 to 6.02), whilethe adjusted indirect comparison indicates thatH2RA was as effective as sucralfate (RR 0.99, 95%CI 0.47 to 2.08). The discrepancy between thedirect and the adjusted indirect estimates ismarginally statistically significant (z = 1.84). Thecombination of the direct and adjusted indirectestimate provided a statistically non-significantrelative risk of 1.56 (95% CI 0.93 to 2.75).

A general question raised by cases like this is:‘when is it appropriate to combine direct andindirect estimates?’ Bucher and colleagues14

concluded that only where direct comparisons areunavailable should indirect comparison meta-analysis be carried out. This is certainly thestandard approach, but it is not clear that this willalways be the best advice. Even when there isdirect randomised evidence, there may be farmore information available from indirectcomparisons. For example, Song and colleagues87

examined a case in which one head-to-headrandomised trial and six trials contributed to anindirect comparison (via two differentcomparators). In addition, the use of indirect datamay be particularly indicated when it is felt thatthe methodological quality of the trials makingdirect comparisons is low.

It will be a matter of judgement whether and howto take into account the indirect evidence.Although a combination of direct and indirectcomparisons may appear to strengthenconclusions (by increasing the quantity of data),the increase in precision must be balanced againsta loss of confidence in the certainty with whichbias is avoided. Few would argue that direct andindirect estimates should always be combined.Rather, many would feel that while presentingseparate estimates is necessary, combination willonly sometimes be suitable. Some criteria areneeded on which to base such a judgement. A verysimilar problem arises in other contexts withinsystematic reviews; for example, reviewers ponderwhether it is reasonable to combine parallel andcross-over trials, or in epidemiologicalinvestigations whether to combine case–controland cohort studies. It is not desirable to base suchdecisions on whether or not the differencebetween the two estimates is statisticallysignificant, although this is the easiest approach. Amore constructive approach would be to base thedecision on the similarity of the participants in thedifferent trials and the comparability of theinterventions. Such judgement applies to thedirectly randomised trials and each subset of trialscontributing to the indirect comparison.

What to do when there is nodirect evidenceIt is not unusual to find that different treatmentoptions have not been directly compared withinrandomised trials, and conclusions on relativeefficacy often end up based entirely on indirectevidence. The adjusted indirect method may beespecially useful to obtain some indirect evidenceabout the relative efficacy of competinginterventions. The validity of the indirect estimatewas discussed in the previous section. Althoughthe absence of any direct evidence avoids thequestion of whether to combine, it means that allof the available evidence is indirect. The reliabilityof that evidence is then of particular concern.

Evidence from an adjusted indirect comparisonshould be interpreted with caution. The internal


67


validity of the trials involved in the adjustedindirect comparison should be examined becausebias in trials will inevitably infect the results of theadjusted indirect comparison. In addition, for theadjusted indirect estimate to be valid, the keyassumption here is that the relative efficacy of anintervention is consistent in patients included indifferent trials. That is, the estimated relativeefficacy should be generalisable. However,generalisability (external validity) of trial results isoften questionable because of restricted inclusioncriteria, exclusion of patients and the higher levelof clinical settings where trials were carried out.123

Such assessments are more complex still whencomparing sets of trials evaluating differenttreatment comparisons. As in many othersituations, interpretation is a matter of judgementand there are no rules applicable across allcircumstances.

Are indirect comparisons of RCTspreferable to direct comparisonsfrom non-randomised trials?As noted above, in the absence of directrandomised evidence one could seek eitherindirect randomised evidence or non-randomisedstudies that examine directly the comparison ofinterest. The authors have not found or generatedany empirical evidence to investigate this issue, sothis discussion relies on knowledge of themechanisms that would render both types ofcomparison biased, and the comparativelikelihood that such mechanisms exist.

Non-randomised evaluations of healthcareinterventions involve making comparisonsbetween two groups of participants who receivedifferent interventions. If there are any otherdifferences between the groups that are themselveslinked to outcome, the comparison will beconfounded and potentially biased. For example,the case-mix in the participant groups could differin terms of age, gender, disease severity or co-morbidities, there could be other differences intreatments between the groups, or the way inwhich outcomes are defined and assessed couldvary. Judgement of the validity of the comparisondepends on the degree to which one is assuredthat ‘like is being compared with like’ such thatone knows that there are no differences betweenthe groups in all factors other than theintervention received. Although many devices areused in non-randomised studies to reduce thepotential for such confounding (such asmeasurement of change scores, matching,

stratification and statistical risk adjustment ofresults), the degree to which these are successful inany individual study is largely inestimable.Empirical studies have found, however, thatadjustment may fail to remove the bulk ofselection bias.15 In addition, as there is always alikelihood of prognostic factors that are unknownor unmeasurable, adjustment methods cannotcope with all eventualities. Indirect comparisonsalso feature comparisons between non-randomisedgroups; however, in this instance the groups arenot cohorts receiving one or other treatment, butrandomised trials within which differentcomparisons are made. Now the same differencesin case-mix, concomitant therapy and follow-upcould exist between the trials in an indirectcomparison in the same way as they do betweenthe groups within a non-randomised study.However, even if the magnitudes of thesedifferences are similar (and there are reasons toargue that they are most likely to be less inindirect comparisons), there are reasons to believethat the bias they introduce would be less than in anon-randomised study.

For binary data the mechanism by which thisprediction is reached is as follows. Variations inpotentially confounding factors will change theaverage event rate observed in the trial as they doin a non-randomised study, as they relate to thefrequency of outcome. However, it is likely thattheir impact on outcome would affect both groupsin each trial in a proportionate manner. If thishappens, and if the treatment effect is calculatedusing an appropriate metric (probably a risk ratioor an odds ratio), very little variation in thetreatment effect will be observed between trials ofthe same comparison among participants withvarying baseline risks. When this is the case theindirect comparison will be unbiased. Thisobservation is fundamental to the practice ofmeta-analysis.57

The case of continuous outcomes seems not tohave been considered in the same light. Althoughthe broad concerns are the same, the impact ofeffect modifiers, including case-mix variation, isunpredictable; it may work in additive ormultiplicative manners, affecting either or both ofthe mean and standard deviation.

An indirect comparison will, however, be biased ifthe differences in baseline risk between trials arelinked to differences in the observed treatmenteffect. This would occur if any of the factorsvarying between the trials are known effectmodifiers (or, in meta-analysis terminology, sources

Discussion

68

of heterogeneity). Thus, it would be wrong to claim,based on the argument of constancy of treatmenteffects, that indirect comparisons are alwaysunbiased. There are examples where such effectmodifiers exist such that indirect comparisons willbe biased. Therefore, it will always be desirable toshow similar distributions of confounding factors inthe trials included in an indirect comparison,rather than rely on the assumption of constancy ofeffect across varying baseline risk.

The resampled results (Chapter 5), despitedemonstrating heterogeneity between trials,probably underestimate the potential for such bias,as all the trials in the reconstructed analysis usedexactly the same protocol. Such uniformity ofprotocol is unlikely in reality.

In conclusion, bias is less likely in indirectcomparisons than within a non-randomised study,as an indirect comparison between treatmenteffects estimated in trials with different baselinerisks is not necessarily biased, whereas the samedifference in baseline risks between groups in anon-randomised comparison is sufficient to renderit biased. The present authors thus agree with therelative placing of these levels of evidence byMcAlister and colleagues,5 as discussed inChapter 1.

Comparing multiple interventionssimultaneouslyThe focus of this review and empirical researchhas been the use of indirect comparisons tocompare two prespecified healthcareinterventions. As noted in Chapter 1, reviewerssometimes consider simultaneously severalinterventions with the natural desire to saysomething about their relative efficacy. The resultsof such a review may be presented in the form of aleague table of efficacy.

This situation was not studied in the presentempirical work, although it did consider indirectcomparison of two specific treatments from trialsinvolving four different treatments (Chapter 5).Here, some different situations within this broadframework are briefly summarised, and anindication is given of how analysis might proceed.Distinction is made between two rather differentcontexts in which multiple treatments areconsidered simultaneously.

First, there may be several competinginterventions, each of which has been compared

against the same comparator in one or moreRCTs. The most obvious example is where each ofseveral drugs has been evaluated in placebo-controlled trials. There are several such examplesin pain research.124,125 The common comparatormakes the comparison of the various drugs seemrelatively simple. In the particular case of trials ofpain relief there is also much greater consistencyof trial methodology (notably standardisedoutcome measure) than is seen in most medicalareas. However, the quoted results are often simplyseparate meta-analyses of each active drug versusplacebo, ranked by treatment effect (perhapsNNT); this display does not allow an easyassessment of the difference between any twoparticular treatments, and so may give a falseimpression of their relative merits. Also, when datasets are presented in this way, there may be asuspicion that any direct comparisons have beenomitted to simplify the presentation.

Sometimes researchers making such comparisonsuse the naive approach by summarising treatmentarms across trials,6 or ignore the control groupsaltogether.46 The strong warnings given againstthe naive approach apply even more strongly inthe multiple treatment case.

This data structure is similar to that where thereare several trials of the same intervention versuscontrol, but where that intervention was deliveredin different ways in different trials; for example,dose, regimen or route may vary across trials.However, the aim of meta-analysis in such cases isusually to evaluate the treatment against thecomparator rather than to study explicitly theimportance of the mode of delivery.

The second case is where there are multiplestudies that have each compared two or more of aset of treatments, but without a commoncomparator. This more general situation isexemplified by the trial of antithrombotic therapyconsidered in Chapter 4 and also a set of trials ofthrombolytic therapy discussed by Hasselblad andKong,68 which is shown in Table 17.

The aim here may be to estimate the effect of aparticular drug versus placebo when no such trialhas been done, but such data may also be the basisof an attempt to rank the treatments. Data such asthese can arise when new treatments areintroduced over a period of several years, an effectseen more clearly in the subset of trials shown inTable 18. This subset perhaps makes it clearer thatestimation is based on a chain of inference. Forexample, adjusted indirect comparisons or logistic


69


regression could be used to estimate the effect ofaccelerated t-PA versus placebo. Such an analysiswould link the results of five trials, three of whichdid not investigate either of the treatments ofinterest. Like all chains, its strength depends onthe weakest link. Here TEAM-3 was a very smalltrial of 325 patients, with an imprecise estimate oftreatment effect: it would therefore beunsatisfactory to rely on this chain of inference.ISIS-3 enrolled over 17,000 patients to make thesame comparison, so there would be no need torely on TEAM-3. The amount of information ineach component of such an inferential chainwould certainly be a concern, even though the

uncertainty of the final estimate would takesample size into account. Other concerns wouldinclude the methodological quality and the degreeof comparability of the treatments, participantsand protocols of the component trials. Hasselbladand Kong rather underplay the assumptionsbehind such an analysis. They suggest that themethod is acceptable as long as there is a commonsubpopulation of participants in the differenttrials.68 In fact, the more links there are thestronger will be the assumptions of adjustedincorrect comparisons. Recently, other authorshave proposed new models for analysing suchnetworks of comparisons.76,88

Discussion

70

TABLE 17 Treatments studied in 17 randomised trials reporting 30-day mortality examining thrombolytic therapy for patients withacute coronary syndrome (treatment within 6 hours of onset) (see Hasselblad and Kong68 for further details)

Trial Placebo t-PA APSAC SK RPA Accelerated t-PA

ASSENT � �ECGS � �AIMS � �Bassand-1 � �GISSI � �ISIS-2 � �ISAM � �TEAM-3 � �Bassand-2 � �GISSI-2 � �ISG � �ISIS-3 � � �TIMI-4 � �TAPS � �INJECT � �GUSTO I � �GUSTO III � �

APSAC, anisoylated plasminogen streptokinase activator complex; RPA, reteplase; SK, streptokinase; tPA, tissue plasminogenactivator.

TABLE 18 Subset of trials from Table 17, showing sequential evaluation of new treatments

Trial Placebo t-PA APSAC SK RPA Accelerated t-PA

ASSENT � �TEAM-3 � �ISIS-3 � �INJECT � �GUSTO III � �

APSAC, anisoylated plasminogen streptokinase activator complex; RPA, reteplase; SK, streptokinase; tPA, tissue plasminogenactivator.

Implications for practice ofsystematic reviewsWhen conducting systematic reviews to evaluatethe effectiveness of interventions, direct evidencefrom good-quality RCTs should be used whereverpossible. If no such evidence exists a call forfurther trials may be necessary (if they are deemedethical). If further research is not feasible, it maybe necessary to look for direct comparisons innon-randomised studies and/or indirectcomparisons from RCTs, which would requireadditional searches of the literature. The reviewerneeds, however, to be aware that both aresusceptible to bias, although bias is less likely inindirect comparisons. The use of non-randomisedstudies within a systematic review is, in addition,perhaps more problematic in terms of identifyingthe studies. The development of better indexingand sensitive search strategies is required to aidthe identification of study designs other thanRCTs. It should also be noted, however, that thedevelopment of search strategies for theidentification of RCTs may actually exclude datathat could be used when making indirectcomparisons, particularly if a search is drug ortreatment specific. Ideally, indirect comparisonsshould be prespecified in a review’s protocol andthe search strategy developed so as to include allrelevant drug/treatment comparisons.

When making indirect comparisons in a systematicreview, the reviewer needs to be clear that that iswhat they are doing, and interpret the resultsappropriately. The naive approach, comparing theresults of single arms between trials, should beavoided as empirical evidence shows it to have ahigh frequency of statistically significantdiscrepancies from the direct estimate. Ideally, anadjusted indirect comparison method should beused, using the random effects model (given theincrease in study heterogeneity). The main typesof adjusted indirect comparison, showing littledifference in performance, are the adjustedindirect comparison, meta-regression and logisticregression.

If both direct and indirect comparisons arepossible within a review, these should be doneseparately before considering whether to pool

data. Whether or not the two sets of informationare combined, or when only indirect comparisonsare available, interpretations should be made evenmore cautiously than in a standard meta-analysisof just head-to-head randomised trials, in view ofthe partially observational nature of thecomparisons.

Implications for clinical researchIndirect comparisons can be used to overcomecertain problems that may arise in the design andconduct of an RCT. An example can be seen in aproposed European multicentre trial of surgicaltechniques for repairing cleft lip and palate.126

Each centre approached used a different surgicaltechnique and was only willing to participate in thetrial if their ‘preferred’ surgical technique was oneof the treatment arms to which patients would berandomised. To overcome this, a common protocolwas devised that allowed each centre to randomiseto one of two surgical techniques: their usual,preferred technique, and an alternative techniquethat was common to all participating centres. Thealternative technique was planned to be used as thecommon comparator, allowing adjusted indirectcomparisons to be made between the centres’preferred surgical techniques. A head-to-headcomparison of the different surgical techniquesmay have been preferable, but the indirectcomparison was able to provide an acceptabledesign for the surgeons, who otherwise may nothave been willing to participate in the trial.

When considering the use of an indirectcomparison when designing such a multicentreRCT, consideration should be given to devising aunified protocol with consistent inclusion criteriaacross all centres.

Recommendations formethodological researchThe majority of the indirect comparisonsidentified were made within meta-analyses ofbinary data. There is a need for evaluation ofalternative methods for analysis of indirectcomparisons for continuous data.


71


Chapter 9

Conclusions

In addition, there is a need for empirical researchinto how different methods of indirect comparisonperform in cases where there is a large treatmenteffect.

Further research is required to consider how todetermine when it is appropriate to look at indirectcomparisons and how to judge when to combineboth direct and indirect comparisons. Researchinto how evidence from indirect comparisonscompares to evidence from non-randomisedstudies may also be warranted. (This question hasnot been considered in the current project.)

The empirical investigations (Chapter 5) werebased on one large, multicentre trial101 with acommon protocol across each centre. It would beuseful to repeat the investigations using individualpatient data from a meta-analysis of several RCTsusing different protocols.

The odds ratio was used as the measure of effectwithin the simulation study. Although logisticregression calls for the effect measure to be theodds ratio, it would be interesting to evaluate theimpact of choosing different binary effectmeasures for the inverse variance method.

Conclusions

72

The authors would like to thank Kath Wright(CRD, York) for her assistance with the

development of the electronic search strategies,and members of the International Stroke TrialCollaborative Group for allowing data from theirmulticentre RCT to be used for the empiricalinvestigations within this report. We would alsolike to thank the referees who provided valuablecomments on the draft report.

Contribution of authorsDG Altman (Professor of Statistics in Medicine), JJ Deeks (Senior Medical Statistician), AJ Eastwood (Senior Research Fellow), AM Glenny(Lecturer in Evidence Based Oral Health Care)and F Song (Reader in Research Synthesis)developed the structure of the report.

AJE and AMG coordinated the project.

In developing the search strategy, the initialsearches were developed and undertaken by

Kath Wright (Centre for Reviews and Dissemination,University of York) and the update searches wereundertaken by AMG.

Handsearching was carried by DGA (Chapter 4),AJE (Chapter 3), AMG (Chapter 3), and FS(Chapters 3 and 6).

DGA, JJD, AJE, AMG and FS carried out thescreening of search results and retrieved papersagainst inclusion criteria.

The analysis of data was carried out by DGA(Chapters 4 and 5), M Bradburn (Statistician)(Chapter 5), R D’Amico (Research Fellow)(Chapter 5), JJD (Chapters 4 and 5), AJE (Chapter 3), AMG (Chapter 3), C Sakarovitch(Biostatistician) (Chapter 5) and FS (Chapters 3, 6 and 7).

Production of the full report was carried out byDGA, JJD, AJE, AMG and FS.


73


Acknowledgements

1. Pocock SJ. Clinical trials: a practical approach. NewYork: John Wiley; 1996.

2. Moher D, Schulz KF, Altman D. CONSORT Group(Consolidated Standards of Reporting Trials). TheCONSORT statement: revised recommendationsfor improving the quality of reports of parallel-group randomized trials. JAMA 2001;285:1987–91.

3. Song F, Glenny AM. Antimicrobial prophylaxis incolorectal surgery: a systematic review ofrandomised controlled trials. Health Technol Assess1998;2(7).

4. de Craen AJM, Tijssen JGP, de Gans J, Kleijnen J.Placebo effect in the acute treatment of migraine:subcutaneous placebos are better than oralplacebos. J Neurol 2000;247:183–8.

5. McAlister FA, Laupacis A, Wells GA, Sackett DL.Users’ Guides to the Medical Literature: XIX.Applying clinical trial results B. Guidelines fordetermining whether a drug is exerting (morethan) a class effect. JAMA 1999;282:1371–7.

6. Felson DT, Anderson JJ, Meenan RF. Thecomparative efficacy and toxicity of second-linedrugs in rheumatoid arthritis. Results of twometaanalyses. Arthritis Rheum 1990;33:1449–61.

7. Garg R, Yusuf S, for the Collaborative Group onACE Inhibitor Trials. Overview of randomizedtrials of angiotensin-converting enzyme inhibitorson mortality and morbidity in patients with heartfailure. JAMA 1995;273:1450–6.

8. Homocysteine Lowering Trialists’ Collaboration.Lowering blood homocysteine with folic acid basedsupplements: meta-analysis of randomised trials.BMJ 1998;316:894–8.

9. Boersma E, Maas AC, Deckers JW, Simoons ML.Early thrombolytic treatment in acute myocardialinfarction: reappraisal of the golden hour. Lancet1996;348:771–5.

10. Moore A, McQuay H, Gray M, editors. Bandolier.Oxford: Pain Relief Unit. URL:http://www.jr2.ox.ac.uk/bandolier/

11. Tramer MR, Moore RA, McQuay HJ. Preventionof vomiting after paediatric strabismus surgery: asystematic review using the numbers-needed-to-treat method. Br J Anaesth 1995;75:556–61.

12. de Craen AJM, Vickers AJ, Tijssen JGP, Kleijnen J.Number-needed-to treat and placebo-controlledtrials. Lancet 1998;351:310.

13. Murray DW, Britton A, Bulstrode CJ.Thromboprophylaxis and death after total hip

replacement. J Bone Joint Surg Am 1996;78-B:863–70.

14. Bucher HC, Guyatt GH, Griffith LE, Walter SD.The results of direct and indirect treatmentcomparisons in meta-analysis of randomizedcontrolled trials. J Clin Epidemiol 1997;50:683–91.

15. Deeks JJ, Dinnes J, D’Amico R, Sowden AJ,Sakarovitch C, Song F, et al. Evaluating non-randomised intervention studies. Health TechnolAssess 2003;7(27).

16. Antiplatelet Trialists’ Collaboration. Collaborativeoverview of randomised trials of antiplatelettherapy. II. Maintenance of vascular graft orarterial patency by antiplatelet therapy. BMJ 1994;308:159–68.

17. Antiplatelet Trialists’ Collaboration. Collaborativeoverview of randomised trials of antiplatelettherapy. I. Prevention of death, myocardialinfarction, and stroke by prolonged antiplatelettherapy in various categories of patients. BMJ1994;308:81–106.

18. Antiplatelet Trialists’ Collaboration. Collaborativeoverview of randomised trials of antiplatelettherapy. III. Reduction in venous thrombosis andpulmonary embolism by antiplatelet prophylaxisamong surgical and medical patients. BMJ 1994;308:235–46.

19. Lowenthal A, Buyse M. Secondary prevention ofstroke: does dipyridamole add to aspirin? ActaNeurol Belg 1994;94:24–34.

20. Li Wan Po A, Zhang WY. Systematic overview of co-proxamol to assess analgesic effects of additionof dextropropoxyphene to paracetamol. BMJ1997;315:1565–71.

21. Marshall JK, Irvine EJ. Rectal corticosteroidsversus alternative treatments in ulcerative colitis: a meta-analysis. Gut 1997;40:775–81.

22. Matchar DB, McCrory DC, Barnett HJ, Feussner MD. Medical treatment for strokeprevention. Ann Intern Med 1994;121:41–53.

23. Milne RJ, Olney RW, Gamble GD, Turnidge J.Tolerability of roxithromycin vs erythromycin incomparative clinical trials in patients with lowerrespiratory tract infections. Clinical DrugInvestigation 1997;14:405–17.

24. Moore A, Collins S, Carroll D, McQuay H.Paracetamol with and without codeine in acutepain: a quantitative systematic review. Pain1997;70:193–201.


75


References

25. Pope JE, Anderson JJ, Felson DT. A meta-analysisof the effects of nonsteroidal anti-inflammatorydrugs on blood pressure. Arch Intern Med 1993;153:477–84.

26. Poynard T, Leroy V, Cohard M, Thevenot T,Mathurin P, Opolon P, et al. Meta-analysis ofinterferon randomized trials in the treatment ofviral hepatitis C: effects of dose and duration.Hepatology 1996;24:778–89.

27. Zhang WY, Li Wan Po A. Analgesic efficacy ofparacetamol and its combination with codeine andcaffeine in surgical pain – a meta-analysis. J ClinPharm Ther 1996;21:261–82.

28. Piccinelli M, Pini S, Bellantuono C, Wilkinson G.Efficacy of drug treatment in obsessive–compulsivedisorder: a meta-analytic review. Br J Psychiatry1995;166:424–43.

29. Zalcberg JR, Siderov J, Simes J. The role of 5-fluorouracil dose in the adjuvant therapy ofcolorectal cancer. Ann Oncol 1996;7:42–6.

30. Leizorovicz A. Comparison of the efficacy andsafety of low molecular weight heparins andunfractionated heparin in the initial treatment ofdeep venous thrombosis. An updated meta-analysis. Drugs 1996;7:30–7.

31. Pignon JP, Arriagada R, Ihde DC, Johnson DH,Perry MC, Souhami RL, et al. A meta-analysis ofthoracic radiotherapy for small-cell lung cancer. N Engl J Med 1992;327:1618–24.

32. Non-small Cell Lung Cancer Collaborative Group.Chemotherapy in non-small cell lung cancer; ameta-analysis using updated data on individualpatients from 52 randomised clinical trials. BMJ1995;311:899–909.

33. Rossouw JE. Lipid-lowering interventions inangiographic trials. Am J Cardiol 1995;76:86–92C.

34. Koch M, Dezi A, Ferrario F, Capurso L. Preventionof nonsteroidal anti-inflammatory drug-inducedgastrointestinal mucosal injury. Arch Intern Med1996;156:2321–32.

35. Holme I. Cholesterol reduction and its impact oncoronary artery disease and total mortality. Am JCardiol 1995;76:10–17C.

36. Moore RA, McQuay HJ. Single-patient data meta-analysis of 3453 postoperative patients: oraltramadol versus placebo, codeine and combinationanalgesics. Pain 1997;69:287–94.

37. Aro J. Cardiovascular and all-cause mortality inprostatic cancer patients treated with estrogens ororchiectomy as compared to the standardpopulation. Prostate 1991;18:131–7.

38. Poynard T, Lemaire M, Agostini H. Meta-analysisof randomized clinical trials comparinglansoprazole with ranitidine or famotidine in thetreatment of acute duodenal ulcer. Eur JGastroenterol Hepatol 1995;7:661–5.

39. Lefering R, Neugebauer EAM. Steroid controversyin sepsis and septic shock: a meta-analysis. CritCare Med 1995;23:1294–303.

40. Chiba N, de-Gara CJ, Wilkinson FM, Hunt RH.Speed of healing and symptom relief in grade IIto IV gastroesophageal reflux disease: a meta-analysis. Gastroenterology 1997;112:1798–810.

41. Coulter A, Kelland J, Peto V, Rees MC. Treatingmenorrhagia in primary care: an overview of drugtrials and a survey of prescribing practice. Int JTechnol Assess Health Care 1995;11:456–71.

42. Bansal VK, Beto JA. Treatment of lupus nephritis:a meta-analysis of clinical trials. Am J Kidney Dis1997;29:193–9.

43. Schmieder RE, Martus P, Klingbeil A. Reversal ofleft ventricular hypertrophy in essentialhypertension. A meta-analysis of randomizeddouble-blind studies. JAMA 1996;275:1507–13.

44. Unge P, Berstad A. Pooled analysis of anti-Helicobacter pylori treatment regimens. Scand JGastroenterol Suppl 1996;220:27–40.

45. Imperiale TF, Speroff TS. A meta-analysis ofmethods to prevent venous thromboembolismfollowing total hip replacement. JAMA 1994;271:1780–5.

46. Conlin PR, Spence JD, Williams B, Ribeiro AB,Saito I, Benedict C, et al. Angiotensin IIantagonists for hypertension: are there differencesin efficacy? Am J Hypertens 2000;13:418–26.

47. Fibrinolytic Therapy Trialists’ (FTT) CollaborativeGroup. Indications for fibrinolytic therapy insuspected acute myocardial infarction:collaborative overview of early mortality and majormorbidity results from all randomised trials ofmore than 1000 patients. Lancet 1994;343:311–22.

48. Ekbom A, Taube A. Methodological flaws in studyon prostatic cancer. Prostate 1994;25:53–4.

49. NHS Centre for Reviews and Dissemination.Undertaking systematic reviews of research oneffectiveness. CRD guidelines for those carrying out orcommissioning reviews. Report No. 4. York: NHSCentre for Reviews and Dissemination; 1996.

50. Berkey CS, Anderson JJ, Hoaglin DC. Multiple-outcome meta-analysis of clinical trials. Stat Med1996;15:537–57.

51. Dominici F, Parmigiani G, Wolpert RL, Hasselblad V. Meta-analysis of migraine headachetreatments: combining information fromheterogeneous designs. Journal of the AmericanStatistical Association 1999;94:16–28.

52. Hasselblad V. Meta-analysis of multitreatmentstudies. Med Decis Making 1998;18:37–43.

53. Higgins JP, Whitehead A. Borrowing strength fromexternal trials in a meta-analysis. Stat Med1996;15:2733–49.

References

76

54. Hirotsu C, Yamada Y. Estimating odds ratiosthrough the connected comparative experiments.Communications in Statistics – Theory and Methods1999;28:905–29.

55. Gleser LJ, Olkin I. Meta-analysis for 2 × 2 tableswith multiple treatment groups. In Stangl DK,Berry DA, editors. Meta-analysis in medicine andhealth policy. New York: Marcel Dekker; 2001. pp. 179–89.

56. Gleser LJ, Olkin I. Stochastically dependent effect sizes. In Cooper H, Hedges LV, editors. The handbook of research synthesis. New York: RussellSage; 1994. pp. 340–55.

57. Deeks JJ. Issues in the selection of a summarystatistic for meta-analysis of clinical trials withbinary outcomes. Stat Med 2002;21:1575–600.

58. Davey Smith G, Egger M. Going beyond the grandmean: subgroup analysis in meta-analysis ofrandomised trials. In Egger M, Davey Smith G,Altman DG, editors. Systematic reviews in healthcare.Meta-analysis in context. 2nd ed. London: BMJBooks; 2001. pp. 143–56.

59. Thompson SG. Why and how sources ofheterogeneity should be investigated. In Egger M,Davey Smith G, Altman DG, editors. Systematicreviews in healthcare. Meta-analysis in context. 2nd ed.London: BMJ Books; 2001. pp. 157–75.

60. Deeks JJ, Altman DG, Bradburn MJ. Statisticalmethods for examining heterogeneity andcombining results from several studies in meta-analysis. In Egger M, Davey Smith G, Altman DG,editors. Systematic reviews in health care: Meta-analysisin context. London: BMJ Books; 2001. pp. 285–312.

61. Altman DG, Bland JM. Interaction revisited: thedifference between two estimates. BMJ 2003;326:219.

62. Thompson SG, Higgins JPT. How should meta-regression analyses be undertaken andinterpreted? Stat Med 2002;21:1559–73.

63. Anderson S, Hauck WW. The transitivity ofbioequivalence testing: potential for drift. Int JClin Pharmacol Ther 1996;34:369–74.

64. Hauck WW, Anderson S. Some issues in the designand analysis of equivalence trials. Drug InformationJournal 1999;33:109–18.

65. Mainland D. The treatment of clinical and laboratorydata. Edinburgh: Oliver and Boyd; 1938. pp. 151–3.

66. Armitage P, Berry G. The sampling error of adifference. In Statistical methods in medical research.2nd ed. Oxford: Blackwell; 1987. pp. 88–90.

67. Eddy DM, Hasselblad V, Schachter R. Meta-analysisby the confidence profile method. Boston, MA:Academic Press; 1992.

68. Hasselblad V, Kong DF. Statistical methods forcomparison to placebo in active–control trials.Drug Information Journal 2001;35:435–49.

69. Packer M, Antonopoulos GV, Berlin JA, Chittams J,Konstam MA, Udelson JE. Comparative effects ofcarvedilol and metoprolol on left ventricularejection fraction in heart failure: results of a meta-analysis. Am Heart J 2001;141:899–907.

70. Fisher LD, Gent M, Butler HR. Active–controltrials: how would a new agent compare withplacebo? A method illustrated with clopidogrel,aspirin, and placebo. Am Heart J 2001;141:26–32.

71. Baker SG, Kramer BS. The transitive fallacy forrandomized trials: if A bests B and B bests C inseparate trials, is A better than C? BMC Med ResMethodol 2002;2:13.

72. Clarke MJ, Stewart LA. Obtaining data fromrandomised controlled trials: how much do weneed for reliable and informative meta-analysis?BMJ 1994;309:1007–10.

73. Higgins JP, Whitehead A, Turner RM, Omar RZ,Thompson SG. Meta-analysis of continuousoutcome data from individual patients. Stat Med2001;20:2219–41.

74. Smith CT, Williamson PR, Marson AG.Investigating heterogeneity in an individualpatient data meta-analysis of time to eventoutcomes Stat Med 2005;24:1307–19.

75. Brown H, Prescott R. Applied mixed models inmedicine. Chichester: John Wiley; 1999.

76. Ades AE. A chain of evidence with mixedcomparisons: models for multi-parametersynthesis and consistency of evidence. Stat Med2003;22:2995–3016.

77. Sutton AJ, Abrams KR, Jones DR, Sheldon TA,Song F. Methods for meta-analysis in medical research.Chichester: John Wiley; 2000.

78. Thompson SG, Sharp SJ. Explainingheterogeneity in meta-analysis: a comparison ofmethods. Stat Med 1999;18:2693–708.

79. Lim E, Ali A, Routledge T, Edmonds L, AltmanDG, Large S. Indirect comparison meta-analysis ofaspirin therapy after coronary surgery. BMJ 2003;327:1309–11.

80. Lim E. First principles or evidence based critique?URL: http://bmj.bmjjournals.com/cgi/eletters/327/7427/1309#46233 (accessed September 2004).

81. Spiegelhalter DJ, Abrams KR, Myles JP. Bayesianapproaches to clinical trials and health-care evaluation.Chichester: John Wiley; 2004.

82. Phillips A. Trial and error: cross-trial comparisonsof antiretroviral regimens. AIDS 2003;17:619–23.

83. Senn S. Active control equivalence trials. Statisticalissues in drug development. Chichester: John Wiley;2002. pp. 207–17.


77


84. Siegel JP. Equivalence and non-inferiority trials.Am Heart J 2000;139:S166–70.

85. ICH Harmonised Tripartite Guideline. Choice ofcontrol group and related issues in clinical trials.URL: http://www.nihs.go.jp/dig/ich/efficacy/e10/e10step4.pdf (accessed March 2005).

86. Engels EA, Schmid CH, Terrin N, Olkin I, Lau J.Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses.Stat Med 2000;19:1707–28.

87. Song F, Glenny AM, Altman DG. Indirectcomparison in evaluating relative efficacy –illustrated by antimicrobial prophylaxis in colorectalsurgery. Control Clin Trials 2000;21:488–97.

88. Lumley T. Network meta-analysis for indirecttreatment comparisons. Stat Med 2002;21:2313–24.

89. Chan A-W, Altman DG. Epidemiology andreporting of randomised trials published inPubMed journals. Lancet 2005;365:1159–62.

90. Lambert PC, Sutton AJ, Jones DR. A comparisonof summary patient-level covariates in meta-regression with individual patient data meta-analysis. J Clin Epidemiol 2002;55:86–94.

91. Berlin JA, Santanna J, Schmid CH, Szczech LA,Feldman HI. Individual patient- versus group-leveldata meta-regression for the investigation oftreatment effect modifiers: ecological bias rears itsugly head. Stat Med 2002;21:371–87.

92. Turner RM, Omar RZ, Yang M, Goldstein H,Thompson SG. A multilevel model framework formeta-analysis of clinical trials with binaryoutcomes. Stat Med 2000;19:3417–32.

93. Begg CB, Pilote L. A model for incorporatinghistorical controls into a meta-analysis. Biometrics1991;47:899–906.

94. Li Z, Begg CB. Random effects models forcombining results from controlled anduncontrolled studies in a meta-analysis. Journal ofthe American Statistical Association 1994;89:1523–7.

95. Raghunathan TE. Pooling controls from differentstudies. Stat Med 1991;10:1417–26.

96. Büchner T, Dohner H, Ehninger G, Ganser A,Hasford J. Up-front randomization and commonstandard arm: a proposal for comparing AMLtreatment strategies between different studies.Leuk Res 2002;26:1073–5.

97. Büchner T, Dohner H, Ehninger G, Ganser A,Niederwieser D, Hasford J. Cross-trial networkingin AML: a step forward rather than corner cutting.Leuk Res 2004;28:649–50.

98. Hills RK, Richards SM, Wheatley K. Cornercutting compromises clinical trials: the inherentproblems with up-front randomisation and acommon standard arm. Leuk Res 2003;27:1071–3.

99. Tsong Y, Wang SJ, Hung HM, Cui L. Statisticalissues on objective, design, and analysis ofnoninferiority active-controlled clinical trial. J Biopharm Stat 2003;13:29–41.

100. Wang SJ, Hung HM, Tsong Y. Utility and pitfallsof some statistical methods in active controlledclinical trials. Control Clin Trials 2002;23:15–28.

101. Sandercock PAG, Collins R, Counsell C, Farrell B,Peto R, Slattery J, et al. The International StrokeTrial (IST): a randomised trial of aspirin,subcutaneous heparin, both, or neither among 19 435 patients with acute ischaemic stroke. Lancet1997;349:1569–81.

102. Sterne JAC, Bradburn MJ, Egger M. Meta-analysisin StataTM. In Egger M, Davey Smith G, AltmanDG, editors. Systematic reviews in health care. Meta-analysis in context. London: BMJ Books; 2001. pp. 347–85.

103. Bradburn MJ, Deeks JJ, Altman DG. Metan-analternative meta-analysis command (sbe24)[correction appears in Stata Technical Bulletin, No. 45, 21 (sbe24.1)]. Stata Technical Bulletin 1998;44:4–15.

104. Sharp S. sbe23: Meta-analysis regression. StataTechnical Bulletin 1998;42:16–24.

105. Robinson LD, Jewell NP. Some surprising resultsabout covariate adjustment in logistic-regressionmodels. International Statistical Review 1991;59:227–40.

106. Song F, Altman DG, Glenny AM, Deeks JJ. Validityof indirect comparison for estimating efficacy ofcompeting interventions: empirical evidence frompublished meta-analyses. BMJ 2003;326:472–5.

107. Rostom A, Wells G, Tugwell P, Welch V, Dube C,McGowan J. Prevention of chronic NSAID inducedupper gastrointestinal toxicity. In The CochraneLibrary (Issue 3). Oxford: Update Software; 2000.

108. Soo S, Moayyedi P, Deeks J, Delaney B, Innes M,Forman D. Pharmacological interventions for non-ulcer dyspepsia. In The Cochrane Library (Issue 3).Oxford: Update Software; 2000.

109. van Pinxteren B, Numans ME, Bonis PA, Lau J.Short-term treatment with proton pumpinhibitors, H2-receptor antagonists and prokineticsfor gastro-oesophageal reflux disease-likesymptoms and endoscopy negative reflux disease.In The Cochrane Library (Issue 3). Oxford: UpdateSoftware; 2000.

110. Palmer BV, Mannur KR, Ross WB. An observerblind trial of co-amoxiclav versus cefuroxime plusmetronidazole in the prevention of postoperativewound infection after general surgery. J Hosp Infect1994;26:287–92.

111. Rowe-Jones DC, Peel ALG, Kingston RD, Shaw JFL,Teasdale C, Cole DS. Single dose cefotaxime plusmetronidazole versus three dose cefuroxime plus

References

78

metronidazole as prophylaxis against woundinfection in colorectal surgery: multicentreprospective randomised study. BMJ 1990;300:18–22.

112. Kwok SPY, Lau WY, Leung KL, Ku KW, Ho WS, Li AKC. Amoxycillin and clavulanic acid versuscefotaxime and metronidazole as antibioticprophylaxis in elective colorectal resectionalsurgery. Chemotherapy 1993;39:135–9.

113. Altman DG, Deeks JJ. Meta-analysis, Simpson’sparadox, and the number needed to treat. BMCMed Res Methodol 2002;2:3.

114. Altman DG, Dore CJ. Randomisation and baselinecomparisons in clinical trials. Lancet 1990;335:149–53.

115. Schulz KF, Chalmers I, Hayes RJ, Altman DG.Empirical evidence of bias: dimensions ofmethodological quality associated with estimates oftreatment effects in controlled trials. JAMA 1995;273:408–12.

116. Song F, Eastwood A, Gilbody S, Duley L, Sutton A.Publication and related biases. Health Technol Assess2000;4(10).

117. Chan A-W, Hròbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence forselective reporting of outcomes in randomizedtrials: comparison of protocols to publishedarticles. JAMA 2004;291:2457–65.

118. Villar J, Carroli G, Belizan JM. Predictive ability ofmeta-analyses of randomised controlled. Lancet1995;345:772–6.

119. Cappelleri JC, Ioannidis JPA, Schmid CH, Ferranti SD de, Aubert M, Chalmers TC, et al.Large trials vs meta-analysis of smaller trials. Howdo their results compare? JAMA 1996;276:1332–8.

120. LeLorier J, Gregoire G, Benhaddad A, Lapierre J,Derderian F. Discrepancies between meta-analysesand subsequent large randomized, controlledtrials. N Engl J Med 1997;337:536–42.

121. Abel U, Koch A. The role of randomization inclinical studies: myths and beliefs. J Clin Epidemiol1999;52:487–97.

122. Jüni P, Altman DG, Egger M. Assessing the qualityof randomised controlled trials. In Egger M,Davey Smith G, Altman DG, editors. Systematicreviews in health care. Meta-analysis in context.London: BMJ Books; 2001. pp. 87–108.

123. Black N. Why we need observational studies toevaluate the effectiveness of health care. BMJ1996;312:1215–18.

124. McQuay HJ, Moore RA. An evidence-based resource forpain relief. Oxford: Oxford University Press; 1998.

125. Bandolier. The Oxford league table of analgesicefficacy. URL: http:// www.jr2.ox.ac.uk/bandolier/booth/painpag/acutrev/analgesics/lftab.html(accessed September 2004).

126. European Collaboration on CraniofacialAnomalies (Eurocran). URL:http://www.eurocran.org/content.asp?contentID=1345(accessed September 2004).

127. Sackett DL. Rules of evidence and clinicalrecommendations on the use of asymptomaticatherosclerosis. Stroke 1989;20:844–9.

128. Poynard T. Evaluation de la qualite methodolgiquedes essais therapeutiques randomises. Presse Med1988;17:315–88.

129 Marshall JK, Irvine EJ. Rectal aminosalicylatetherapy for distal ulcerative colitis: a meta-analysis.Aliment Pharmacol Ther 1995;9:293–300.

130. Chalmers TC, Smith H Jr, Blackburn B, Silverman B, Schroeder B, Reitman D et al. A method for assessing the quality of a randomizedcontrol trial. Control Clin Trials 1981;2:31–49.

131. Liberati A, Himel HN, Chalmers TC. A qualityassessment of randomized control trials of primarytreatment of breast cancer. J Clin Oncol 1986;4:942–51.

132. Evans M, Pollock AV. A score system for evaluatingrandom control clinical trials of prophylaxis ofabdominal surgical wound infection. Br J Surg1985;72:265–70.

133. Jadad AR, Moore RA, Carrol D, Jenkinson C,Reynolds DJM, Gavaghan DJ, et al. Assessing thequality of reports of randomised clinical trials: isblinding necessary? Control Clin Trials 1996;17:1–12.

134. Poynard T, Pignon JP. Duodenal ulcer. Analysis of 293randomized clinical trials. London: Montrouge, JohnLibbey Eurotext; 1989.

135. DerSimonian R, Charette LJ, McPeek B, Mosteller F. Reporting on methods in clinicaltrials. N Engl J Med 1982;306:1332–7.

136. Schmieder RE, Schlaich MP, Klingbeil AU, Martus P. Update on reversal of left ventricularhypertrophy in essential hypertension (a meta-analysis of all randomized double-blind studiesuntil December 1996). Nephrol Dial Transplant1998;13:564–9.

137. Cheng L, Gulmezoglu AM, Ezcurra E, van-Look PFA. Interventions for emergencycontraception (Cochrane Review). In The CochraneLibrary (Issue 3). Oxford: Update Software; 2000.

138. Collins SL, Moore RA, McQuay HJ, Wiffen PJ,Edwards JE. Single dose oral ibuprofen anddiclofenac for postoperative pain (CochraneReview). In The Cochrane Library (Issue 3). Oxford:Update Software; 2000.

139. Delaney BC, Innes MA, Deeks JJ, Wilson S, OakesR, Moayyedi P, et al. Initial management strategiesfor dyspepsia (Cochrane Review). In The CochraneLibrary (Issue 3). Oxford: Update Software; 2001


79


140. Di Mario F, Battaglia G, Leandro G, Grasso G,Vianello F, Vigneri S. Short-term treatment ofgastric ulcer. A meta-analytical evaluation of blindtrials. Dig Dis Sci 1996;41:1108–31.

141. Handoll HHG, Farrar MJ, McBirnie J,Tytherleigh-Strong G, Awal KA, Milne AA, et al.Heparin, low molecular weight heparin andphysical methods for preventing deep veinthrombosis and pulmonary embolism followingsurgery for hip fractures (Cochrane Review). InThe Cochrane Library (Issue 4). Oxford: UpdateSoftware; 2002.

142. Horn J, Limburg M. Calcium antagonists for acuteischemic stroke (Cochrane Review). In TheCochrane Library (Issue 3). Oxford: UpdateSoftware; 2001.

143. McIntosh HM, Olliaro P. Artemisinin derivativesfor treating uncomplicated malaria (CochraneReview). In The Cochrane Library (Issue 3). Oxford:Update Software; 2001.

144. Silagy C. Physician advice for smoking cessation(Cochrane Review). In The Cochrane Library (Issue3). Oxford: Update Software; 2001.

145. Silagy C, Mant D, Fowler G, Lancaster T. Nicotinereplacement therapy for smoking cessation(Cochrane Review). In The Cochrane Library (Issue3). Oxford: Update Software; 2001.

146. Trindade E, Menon D. Selective serotoninreuptake inhibitors (SSRIs) for major depression.

Part I. Evaluation of the clinical literature. ReportNo. 3E. Ottawa, Ontario: Canadian CoordinatingOffice for Health Technology Assessment; 1997August 1997.

147. Zhang WY, Li Wan Po A. Efficacy of minoranalgesics in primary dysmenorrhoea: a systematicreview. Br J Obstet Gynaecol 1998;105:780–9.

148. Ausejo M, Saenz A, Pham B, Kellner JD, Johnson DW, Moher D, et al. Glucocorticoids forcroup (Cochrane Review). In The Cochrane Library(Issue 3). Oxford: Update Software; 2001.

149. Sauriol L, Laporta M, Edwardes MD, Deslandes M,Ricard N, Suissa S. Meta-analysis comparing newerantipsychotic drugs for the treatment ofschizophrenia: evaluating the indirect approach.Clin Ther 2001;23:942–56.

150. Hart RG, Benavente O, McBride R, Pearce LA.Antithrombotic therapy to prevent stroke inpatients with atrial fibrillation: a meta-analysis.Ann Intern Med 1999;131:492–501.

151. Singer DE. Antithrombotic therapy to preventstroke in patients with atrial fibrillation. Ann InternMed 2000;132:841–2.

152. EAFT (European Atrial Fibrillation Trial) StudyGroup. Secondary prevention in non-rheumaticatrial fibrillation after transient ischaemic attack orminor stroke. Lancet 1993;342:1255–62.

References

80


81


Appendix 1

Reviews of effectiveness prescreening form

Appendix 1

82 Aut

hor/

year

R

elev

ant

Use

ful

Type

of s

tudy

des

ign

M-A

ENR

ISIC

Com

men

tsM

etho

dolo

gy

DA

RE

Acc

essi

on

proj

ect

(�/?

/✕)

cita

tion

snu

mbe

r(E

/I/N

)

EI

BR

CTs

Com

pN

on-

VA

CF

ICD

C/I

CSu

ff.co

mp

done

done

data


83


Key

:1

.A

uth

or/

year

an

d D

AR

E a

cces

sio

n n

um

ber

if

rele

van

t.2

.R

elev

ant

pro

ject

E

: E

NR

IS;

I: i

nd

irec

t co

mp

aris

on

N:

nei

ther

No

te:

can

be

bo

th E

an

d I

.3

.U

sefu

l�

: if

ad

equ

ate

dat

a ar

e p

rese

nte

d f

or

use

in

eit

her

rev

iew

?:

if

un

sure

or

insu

ffic

ien

t d

ata

are

pre

sen

ted

, bu

t th

ere

is a

po

ssib

ilit

y o

f o

bta

inin

g f

urt

her

dat

a, o

r if

met

a-an

alys

is o

fn

on

-RC

Ts

bu

t n

o v

alid

ity

asse

ssm

ent

or

con

tro

l fo

r co

nfo

un

din

g v

aria

ble

s h

as b

een

un

der

take

n (

spec

ific

ally

fo

rE

NR

IS)

X:

if t

her

e ar

e in

suff

icie

nt

dat

a fo

r u

se i

n e

ith

er r

evie

w (

mar

k r

elev

ant

colu

mn

)B

: m

ark i

f th

e p

aper

is

use

ful

bu

t o

nly

fo

r th

e bac

kg

rou

nd

/dis

cuss

ion

.4

. In

clu

ded

stu

dy

des

ign

En

ter

nu

mber

of

RC

Ts,

co

mp

arat

ive

and

no

n-c

om

par

ativ

e st

ud

ies.

5.

M-A

�

: if

met

a-an

alys

is h

as b

een

un

der

take

nX

: if

no

met

a-an

alys

is h

as b

een

un

der

take

n (

nar

rati

ve r

evie

w).

6.

EN

RIS

VA

: n

eed

to

kn

ow

wh

eth

er s

om

e fo

rm o

f va

lid

ity

asse

ssm

ent

has

bee

n u

nd

erta

ken

CF:

nee

d t

o k

no

w w

het

her

th

ey h

ave

con

tro

lled

fo

r co

nfo

un

din

g v

aria

ble

s7

.IC

IC d

on

e: h

as a

n i

nd

irec

t co

mp

aris

on

bee

n c

on

du

cted

in

th

e re

view

?IC

/DC

do

ne:

do

es t

he

revi

ew p

rese

nt

bo

th d

irec

t an

d i

nd

irec

t co

mp

aris

on

s o

f d

ata?

Su

ff.

dat

a: a

re t

her

e su

ffic

ien

t p

rim

ary

dat

a (i

.e.

nu

mber

of

even

ts a

nd

sam

ple

siz

e fo

r ea

ch a

rm o

f th

e in

clu

ded

tria

ls)

or

sum

mar

y st

atis

tics

an

d S

E/C

Is,

in o

rder

to

fo

rce

an i

nd

irec

t co

mp

aris

on

, w

her

e a

dir

ect

com

par

iso

n h

as b

een

do

ne?

If

so,

stat

e co

mp

aris

on

s fo

r w

hic

h a

n I

C i

s p

oss

ible

in

th

e co

mm

ents

fie

ld.

8.

Co

mm

ents

9.

Met

ho

do

log

y ci

tati

on

sL

ist

cita

tio

ns

of

po

ten

tial

ly r

elev

ant

met

ho

do

log

y p

aper

s ci

ted

in

th

e re

view

(re

fere

nce

lis

ts o

f al

l re

view

s sh

ou

ld b

eex

amin

ed).

NB

. A

ll c

olu

mn

s sh

ou

ld b

e co

mp

lete

d.

Author (year)

Title

Journal

Objectives

COMPARISONS BEING MADE FOR RCTs

INDIRECT Done Not Done*�

Intervention A / ctrl Intervention B / ctrl No. of trialsA / B

DIRECT (only those that are relevant to ICs) Done Not Done�

Intervention A Intervention B No. of trials

METHODS USED FOR CONDUCTING IC


85


Appendix 2

Data extraction form

RESULTS OF IC

Intervention A / ctrl Intervention B /ctrl Summary statistic

RESULTS OF DC

Intervention A Intervention B Summary statistic

INTERPRETATION OF RESULTSIC interpreted appropriately? Yes � No � Unsure �How were ICs interpreted? Details:

Other COMMENTS

Appendix 2

86 *But sufficient data presented in paper to force IC.


87


Appendix 3

Table of systematic reviews making indirect comparisons

Appendix 3

88 TA

BLE

19

Syst

emat

ic re

view

s re

port

ing

adju

sted

indi

rect

com

paris

ons

and

dire

ct c

ompa

rison

s

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

ATC

17

Val

idit

y as

sess

men

t un

dert

aken

wit

hin

the

revi

ew:

Onl

y un

conf

ound

ed t

rials

dem

onst

ratin

g co

ncea

led

trea

tmen

tal

loca

tion

wer

e in

clud

ed

Ass

essm

ent

of h

eter

ogen

eity

:A

naly

ses

wer

e pe

rfor

med

sep

arat

ely

acco

rdin

g to

pat

ient

gro

ups

and

outc

ome

mea

sure

s. A

naly

ses

wer

eal

so p

erfo

rmed

sep

arat

ely

acco

rdin

gto

age

, gen

der,

dias

tolic

blo

odpr

essu

re a

nd d

iabe

tes

Met

hod

used

:A

djus

ted

ICC

omm

on c

ontr

ol.

Resu

lts p

rese

nted

ina

grap

h

Com

pari

sons

mad

e:a.

Asp

irin+

Dip

/ctr

l (n

= 3

4)b.

Asp

irin/

ctrl

(n=

46)

c.H

igh-

dose

asp

irin/

ctrl

(n=

30)

d.M

ediu

m-d

ose

aspi

rin/c

trl

(n=

19)

Res

ults

:%

odd

s re

duct

ion

(SD

)a.

28%

(5)

b.25

% (2

) c.

21%

(4)

d. 2

8% (3

) (1

60–3

25 m

g pe

r da

y)26

% (1

1) (<

160

mg

per

day)

Com

pari

sons

mad

e:e.

Asp

irin+

Dip

/asp

irin

(n=

14)

f.H

igh-

dose

aspi

rin/m

ediu

m-d

ose

aspi

rin (n

=3)

Res

ults

:%

odd

s re

duct

ion

(SD

)e.

–1%

(9)

f.5%

(11)

Indi

rect

(adj

uste

d) e

vide

nce

used

to

enha

nce

dire

ct e

vide

nce

“Suc

h in

dire

ct c

ompa

rison

s ne

ed t

obe

inte

rpre

ted

mor

e ca

utio

usly,

for

alth

ough

man

y of

the

bia

ses

inhe

rent

in n

on-r

ando

m m

etho

ds (s

uch

asth

ose

invo

lvin

g hi

stor

ic c

ontr

ols)

are

avoi

ded,

som

e po

tent

ial f

or b

ias

rem

ains

”

No

stat

istic

ally

sig

nific

ant

diffe

renc

e in

the

resu

lts b

etw

een

the

indi

rect

and

dire

ct c

ompa

rison

s w

as n

oted

App

ropr

iate

inte

rpre

tati

on?

Yes

Add

ition

alco

mpa

rison

sw

ere

mad

e.T

hose

with

grea

test

sam

ple

size

pres

ente

dhe

re

ATC

16

Val

idit

y as

sess

men

t un

dert

aken

wit

hin

the

revi

ew:

Onl

y un

conf

ound

ed t

rials

dem

onst

ratin

g co

ncea

led

trea

tmen

tal

loca

tion

wer

e in

clud

ed

Ass

essm

ent

of h

eter

ogen

eity

:Se

para

te a

naly

ses

wer

e un

dert

aken

acco

rdin

g to

pat

ient

gro

ups

and

whe

ther

the

ant

ipla

tele

t ag

ent

was

star

ted

befo

re, w

ithin

24

hour

s, o

r>

24 h

ours

afte

r th

e pr

oced

ure

Met

hod

used

:A

djus

ted

ICC

omm

on c

ontr

ol,

grap

hic

plot

Com

pari

sons

mad

e:a.

Asp

irin+

Dip

/ctr

l (n

=20

)b.

Asp

irin/

ctrl

(n=

13)

Res

ults

:%

odd

s re

duct

ion

(SD

)a.

40%

(6)

b.48

% (7

)

Com

pari

sons

mad

e:c.

Asp

irin+

Dip

/asp

irin

(n=

9)R

esul

ts:

% o

dds

redu

ctio

n (S

D)

c.–1

% (1

1)

Indi

rect

(adj

uste

d) e

vide

nce

used

to

enha

nce

dire

ct e

vide

nce

“Thi

s ov

ervi

ew p

rovi

des

som

e di

rect

and

indi

rect

ran

dom

ised

com

paris

ons

of t

he e

ffect

s of

diff

eren

t dr

ugre

gim

ens

on t

he p

reve

ntio

n of

occl

usio

n bu

t fin

ds n

o ev

iden

ce o

f any

diffe

renc

es in

effi

cacy

. The

num

bers

of

patie

nts

stud

ied

and

the

num

bers

of

even

ts t

hat

occu

rred

wer

e, h

owev

er,

not

larg

e en

ough

to

excl

ude

som

esm

all b

ut r

eal d

iffer

ence

s in

effi

cacy

betw

een

diffe

rent

dru

g re

gim

ens”

App

ropr

iate

inte

rpre

tati

on?

Yes

Add

ition

alco

mpa

rison

sw

ere

mad

e.T

hose

with

grea

test

sam

ple

size

pres

ente

dhe

re


89


TA

BLE

19

Syst

emat

ic re

view

s re

port

ing

adju

sted

indi

rect

com

paris

ons

and

dire

ct c

ompa

rison

s (c

ont’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

ATC

18

Val

idit

y as

sess

men

t un

dert

aken

wit

hin

the

revi

ew:

Onl

y un

conf

ound

ed t

rials

dem

onst

ratin

g co

ncea

led

trea

tmen

tal

loca

tion

wer

e in

clud

ed

Ass

essm

ent

of h

eter

ogen

eity

:D

ata

from

pat

ient

s un

derg

oing

gene

ral,

trau

mat

ic o

rtho

paed

ic a

ndel

ectiv

e or

thop

aedi

c su

rger

y, a

ndhi

gh-r

isk m

edic

al p

atie

nts

wer

ean

alys

ed s

epar

atel

y. S

epar

ate

anal

yses

wer

e al

so u

nder

take

n fo

r tr

ials

inw

hich

pat

ient

s di

d or

did

not

rec

eive

hepa

rin

Met

hod

used

:A

djus

ted

IC u

sing

ano

-tre

atm

ent

com

paris

on g

roup

.Ill

ustr

ated

with

plo

t

Com

pari

sons

mad

e:O

n D

VT:

a.A

spiri

n+D

ip/c

trl (

n =

18)

b.A

spiri

n/ct

rl (n

= 1

6)

On

PE:

c.A

spiri

n+D

ip/c

trl (

n =

18)

d.A

spiri

n/ct

rl (n

= 2

1)

Res

ults

:O

dds

redu

ctio

na.

56%

b.

23%

c.43

%d.

67%

Com

pari

sons

mad

e:O

n D

VT:

e.A

spiri

n+D

ip/a

spiri

n(n

= 9

)

On

PE:

f.A

spiri

n+D

ip/a

spiri

n(n

= 1

1)

Res

ults

:O

dds

redu

ctio

ne.

52%

The

disc

ussio

n of

asp

irin

plus

Dip

vers

us a

spiri

n al

one

was

form

ed o

nev

iden

ce fr

om d

irect

com

paris

ons.

The

con

clus

ions

wer

e ca

utio

us

Resu

lts fr

om t

he IC

sup

port

tho

sefr

om d

irect

com

paris

ons

App

ropr

iate

inte

rpre

tati

on?

Yes

Low

enth

al a

nd B

uyse

19

Val

idit

y as

sess

men

t un

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

RCTs

incl

uded

Ass

essm

ent

of h

eter

ogen

eity

:�

2te

st fo

r he

tero

gene

ity w

asca

lcul

ated

for

each

end

-poi

nt a

nd fo

rea

ch g

roup

of t

rials

(asp

irin

vspl

aceb

o, a

spiri

n pl

us d

ipyr

imad

ole

vspl

aceb

o). N

o sig

nific

ant

hete

roge

neity

show

n

Met

hod

used

:A

djus

ted

IC, u

sing

plac

ebo

asco

mpa

rato

r.D

ispla

yed

in g

raph

Com

pari

sons

mad

e:a.

Asp

irin+

Dip

/pla

cebo

(n=

2)b.

Asp

irin/

plac

ebo

(n=

9)

Res

ults

:Ri

sk r

educ

tion

(SD

)A

ll de

aths

: a.

30%

(11)

b.10

% (8

)(n

s)Va

scul

ar d

eath

s:a.

24%

(13)

b.–4

% (1

0)(n

s)A

ll st

roke

sa.

42%

(9)

b.17

% (7

)(p

=0.

007)

Fata

l str

okes

:a.

43%

(18)

b.–1

0% (2

1)(p

= 0

.03)

IVE:

a.40

% (8

)b.

18%

(6)

(p=

0.00

7)

Com

pari

sons

mad

e:c.

Asp

irin+

Dip

/asp

irin

(n=

2)

Res

ults

:Ri

sk r

educ

tion

(95%

CI)

All

deat

hs: –

19%

(–77

to 2

0%)

Vasc

ular

dea

ths:

–11

%(–

78 t

o 31

%)

All

stro

kes:

13%

(–22

to

38%

)Fa

tal s

trok

es: 2

% (–

129

to 8

%)

IVE:

4%

(–28

to

28%

)

Indi

rect

(adj

uste

d) e

vide

nce

used

to

enha

nce

dire

ct e

vide

nce.

“It

mus

t be

str

esse

d th

at t

hese

res

ults

are

base

d on

an

indi

rect

com

paris

onbe

twee

n tw

o gr

oups

of t

rials,

and

may

the

refo

re r

efle

ct d

iffer

ence

s in

sele

ctio

n cr

iteria

or

othe

rco

nfou

ndin

g fa

ctor

s ra

ther

tha

n a

trul

y gr

eate

r tr

eatm

ent

effe

ct o

fco

mbi

natio

n th

erap

y”

“The

se r

esul

ts s

ugge

stth

at t

heco

mbi

natio

n th

erap

y of

asp

irin

with

dipy

ridam

ole

may

be

supe

rior

toas

pirin

alo

ne”

App

ropr

iate

inte

rpre

tati

on?

Yes

Resu

ltssu

bseq

uent

lyco

nfirm

ed b

yla

rger

RC

T

Appendix 3

90 TA

BLE

19

Syst

emat

ic re

view

s re

port

ing

adju

sted

indi

rect

com

paris

ons

and

dire

ct c

ompa

rison

s (c

ont’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Li W

an P

o an

d Z

hang

20

Val

idit

y as

sess

men

t un

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

RCTs

incl

uded

Ass

essm

ent

of h

eter

ogen

eity

:St

atist

ical

het

erog

enei

ty a

cros

sin

divi

dual

stu

dies

tes

ted

usin

g th

e Q

stat

istic

. If t

rials

wer

e st

atist

ical

lyhe

tero

gene

ous

a ra

ndom

effe

cts

mod

el w

as u

sed

Met

hod

used

:A

djus

ted

IC u

sing

plac

ebo

asco

mpa

rato

r

Com

pari

sons

mad

e:a.

Para

ceta

mol

+de

xtro

poro

pxyp

hene

/pl

aceb

o (n

=9)

b.Pa

race

tam

ol/p

lace

bo(n

=17

)

Res

ults

:M

ean

diffe

renc

e in

per

cent

age

sum

of d

iffer

ence

s in

pai

nin

tens

ity (9

5% C

I)

Fixe

d ef

fect

: a.

12.7

% (9

.2 t

o 16

.2%

)b.

9.4%

(6.9

to

11.9

%)

Rand

om e

ffect

: a.

13.5

% (8

.8 t

o 18

.3%

)b.

9.4%

(6.6

to

12.2

%)

Com

pari

sons

mad

e:c.

Para

ceta

mol

/pla

cebo

(n=

6, o

nly

thre

eus

ed in

met

a-an

alys

is)

Res

ults

:M

ean

diffe

renc

e in

perc

enta

ge s

um o

fdi

ffere

nces

in p

ain

inte

nsity

(95%

CI)

c.Fi

xed

effe

ct: 7

.3%

(–0.

2 to

14.

9%)

Rand

om e

ffect

s:7.

4% (–

0.4

to15

.1%

)

Indi

rect

(adj

uste

d) e

vide

nce

used

to

enha

nce

dire

ct e

vide

nce

Aut

hors

con

clud

ed t

hat

“On

the

basis

of d

ata

on a

nalg

esic

effi

cacy

and

acu

tesa

fety

in b

oth

head

to

head

and

indi

rect

com

paris

ons,

the

re is

litt

leob

ject

ive

evid

ence

to

supp

ort

pres

crib

ing

a co

mbi

natio

n of

para

ceta

mol

and

dext

ropr

opox

yphe

ne in

pre

fere

nce

topa

race

tam

ol a

lone

in m

oder

ate

pain

such

as

that

afte

r su

rger

y”

App

ropr

iate

inte

rpre

tati

on?

Yes

Oth

er o

utco

me

mea

sure

s al

sopr

esen

ted

in t

here

view

Mat

char

et

al.22

Val

idit

y as

sess

men

t un

dert

aken

wit

hin

the

revi

ew:

Onl

y RC

Ts in

clud

ed a

ccor

ding

to

prev

ious

ly p

ublis

hed

crite

ria12

7

Ass

essm

ent

of h

eter

ogen

eity

:N

o fo

rmal

tes

t of

het

erog

enei

tyre

port

ed, a

lthou

gh it

is s

tate

d th

at a

rand

om e

ffect

s m

etho

d w

as u

sed

asap

prop

riate

Met

hod

used

:A

djus

ted

IC u

sing

plac

ebo/

ctrl

grou

p

Com

pari

sons

mad

e:N

on-v

alvu

lar

atria

l fib

rilla

tion;

a.W

arfa

rin/c

trl (

n =

5)

b.A

spiri

n/pl

aceb

o (n

= 2

)

Res

ults

:RR

for

stro

ke (9

5% C

I) a.

0.33

(0.2

2 to

0.5

0)b.

0.67

(0.4

5 to

0.9

9)

Com

pari

sons

mad

e:N

on-v

alvu

lar

atria

lfib

rilla

tion;

c.W

arfa

rin/a

spiri

n(n

=1)

Res

ults

:RR

for

stro

ke (9

5% C

I) c.

0.34

(0.1

1 to

0.8

7)

The

res

ults

of t

he IC

wer

e us

ed t

oen

hanc

e th

e fin

ding

s of

the

dire

ctco

mpa

rison

App

ropr

iate

inte

rpre

tati

on?

Yes

Det

ails

on o

ther

com

paris

ons

and

outc

ome

mea

sure

s ar

epr

esen

ted

in t

here

view


91


TA

BLE

19

Syst

emat

ic re

view

s re

port

ing

adju

sted

indi

rect

com

paris

ons

and

dire

ct c

ompa

rison

s (c

ont’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Moo

re e

t al

.24

Val

idit

y as

sess

men

t un

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

RCTs

with

blin

ded

desig

n in

clud

ed

Ass

essm

ent

of h

eter

ogen

eity

:Te

sts

used

not

rep

orte

d. A

ll bu

t tw

oco

mbi

natio

ns o

f tria

ls w

ere

hom

ogen

eous

Met

hod

used

:A

djus

ted

IC u

sing

plac

ebo

com

para

tor

Com

pari

sons

mad

e:a.

Para

ceta

mol

+co

dein

e/pl

aceb

o (n

=18

)b.

Para

ceta

mol

/pla

cebo

(n=

13)

Res

ults

:>

50%

max

TO

TPA

R. R

iskra

tio (9

5% C

I)a.

2.6

(2.1

to

3.2)

b.1.

7 (1

.3 t

o 2.

2)

Sum

mar

y st

atist

ic (R

R): 1

.53

Com

pari

sons

mad

e:c.

Para

ceta

mol

+co

dein

e/ p

arac

etam

ol(n

= 1

1)

Res

ults

:>

50%

max

TO

TPA

R.Ri

sk r

atio

(95%

CI)

c.1.

19 (0

.98

to 1

.44)

Resu

lts o

f adj

uste

d IC

and

dire

ctco

mpa

rison

are

disc

usse

d se

para

tely,

with

no

atte

mpt

to

com

bine

the

resu

lts

The

IC g

ave

a gr

eate

r es

timat

e th

anth

e di

rect

com

paris

on in

ter

ms

ofef

ficac

y of

par

acet

amol

plu

s co

dein

eco

mpa

red

with

par

acet

amol

alo

ne

App

ropr

iate

inte

rpre

tati

on?

Yes

Non

-sig

nific

ant

resu

lts o

f dire

ctco

mpa

rison

cou

ldbe

com

e sig

nific

ant

if th

e re

sults

of

the

adju

sted

ICar

e in

corp

orat

ed

Picc

inel

li et

al.28

Val

idit

y as

sess

men

t un

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

doub

le-b

lind

RCTs

of

4 w

eeks

’ dur

atio

n in

clud

ed

Ass

essm

ent

of h

eter

ogen

eity

:�

2te

st o

f het

erog

enei

ty w

asun

dert

aken

. Fix

ed e

ffect

mod

el u

sed

Met

hod

used

:A

djus

ted

ICC

ompa

riso

ns m

ade:

Clo

mip

ram

ine/

plac

ebo

(n=

9)SS

RI/p

lace

bo (n

= 8

)

Res

ults

:O

bses

sive/

com

pulsi

vesy

mpt

oms

(effe

ct s

ize

=g)

(95%

CI)

a.g

= 1

.31

(1.1

5 to

1.4

7)b.

g=

0.4

7 (0

.33

to 0

.61)

Com

pari

sons

mad

e:c.

Clo

mip

ram

ine/

SSRI

s(n

=3)

Res

ults

:c.

–0.0

4 (–

0.43

to

0.35

)

The

aut

hors

con

clud

ed t

hat

“alth

ough

the

incr

ease

in im

prov

emen

t ra

teov

er p

lace

bo w

as g

reat

er fo

rcl

omip

ram

ine

than

for

SSRI

s, d

irect

com

paris

on b

etw

een

thes

e dr

ugs

show

ed t

hat

they

had

sim

ilar

ther

apeu

tic e

ffica

cy o

nob

sess

ive–

com

pulsi

ve s

ympt

oms”

App

ropr

iate

inte

rpre

tati

on?

Yes

Con

sider

able

diffe

renc

eob

serv

ed in

indi

rect

com

paris

on b

utno

t in

dire

ctco

mpa

rison

Appendix 3

92 TA

BLE

19

Syst

emat

ic re

view

s re

port

ing

adju

sted

indi

rect

com

paris

ons

and

dire

ct c

ompa

rison

s (c

ont’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Poyn

ard

et a

l.26

Val

idit

y as

sess

men

t un

dert

aken

wit

hin

the

revi

ew:

Use

of p

revi

ously

val

idat

edqu

estio

nnai

re12

8

Ass

essm

ent

of h

eter

ogen

eity

:�

2te

st o

f het

erog

enei

ty w

asun

dert

aken

. Bot

h fix

ed a

nd r

ando

mef

fect

mod

els

wer

e us

ed

Met

hod

used

:A

djus

ted

ICC

ompa

riso

ns m

ade:

Com

plet

e A

LTa.

3 M

U 1

2 m

onth

s/ct

rl(n

=7)

b.3

MU

6 m

onth

s/ct

rl(n

=7)

Sust

aine

d A

LTc.

3 M

U 1

2 m

onth

s/ct

rl(n

=5)

d.3

MU

6 m

onth

s/ct

rl(n

=6)

Res

ults

:Re

spon

se r

ate

(95%

CI)

a.48

%b.

45%

(35

to 5

5%)

Diff

eren

ce in

res

pons

e ra

te:

3% c.35

% (2

8 to

43%

)d.

21%

(13

to 2

8%)

Diff

eren

ce in

res

pons

e ra

te:

14%

Com

pari

sons

mad

e:C

ompl

ete

ALT

e.3

MU

12

mon

ths/

3MU

6m

onth

s (n

=4)

Sust

aine

d A

LTf.

3 M

U 1

2 m

onth

s/3

MU

6 m

onth

s(n

=4)

Res

ults

:e.

11%

f.16

% (9

to

23%

)

IC u

sed

to e

nhan

ce t

he r

esul

ts o

f the

dire

ct c

ompa

rison

s

App

ropr

iate

inte

rpre

tati

on?

Yes

Dos

e–ef

fect

dat

aav

aila

ble

in p

aper


93


TA

BLE

19

Syst

emat

ic re

view

s re

port

ing

adju

sted

indi

rect

com

paris

ons

and

dire

ct c

ompa

rison

s (c

ont’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

ATC

, Ant

ipla

tele

t Tr

ialis

ts’ C

olla

bora

tion;

ctr

l, co

ntro

l; D

ip, d

ipyr

idam

ole;

DVT

, dee

p ve

in t

hrom

bosis

; IVE

, im

port

ant

vasc

ular

eve

nts;

ns,

not

sig

nific

ant;

PE, p

ulm

onar

y em

bolis

m;

Rem

RR, r

emed

icat

ion

rate

rat

io; R

esRR

, res

pons

e ra

te r

atio

; SPI

D, s

um o

f pai

n in

tens

ity d

iffer

ence

.

Zha

ng a

nd L

i Wan

Po27

Val

idit

y as

sess

men

t un

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

doub

le-b

lind

RCTs

incl

uded

Ass

essm

ent

of h

eter

ogen

eity

:�

2te

st o

f het

erog

enei

ty w

asun

dert

aken

. A r

ando

m e

ffect

s m

odel

was

use

d w

hen

hete

roge

neity

was

pres

ent

Met

hod

used

:A

djus

ted

IC u

sing

plac

ebo

com

para

tor

Com

pari

sons

mad

e:a.

Para

ceta

mol

+co

dein

e/pl

aceb

o (n

=37

)b.

Para

ceta

mol

/pla

cebo

(n=

13)

c.Pa

race

tam

ol+

caffe

ine/

plac

ebo

(n=

228)

d.Pa

race

tam

ol/p

lace

bo(n

=10

)

Res

ults

:D

iffer

ence

in T

OT

PAR%

a.d 2

= 2

3.18

(SE

2.67

)b.

d 1=

15.

06 (S

E 0.

90)

d 2–

d 1=

8.1

2 (S

E 2.

82)

c.d 2

= 1

7.36

(SE

1.89

)d.

d 1=

13.

91 (S

E 1.

52)

d 2–

d 1=

3.4

5 (S

E 2.

43)

Com

pari

sons

mad

e:e.

Para

ceta

mol

+co

dein

e/ p

arac

etam

ol(n

=13

)f.

Para

ceta

mol

+ca

ffein

e/ p

arac

etam

ol(n

=10

)

Res

ults

:D

iffer

ence

in T

OT

PAR%

e.d

= 7

.39

(SE

2.43

)f.

d=

3.9

7 (S

E 1.

73)

IC u

sed

to s

uppo

rt r

esul

ts fr

om d

irect

com

paris

ons

“The

ana

lges

ic e

ffica

cy o

f par

acet

amol

600

mg

was

enh

ance

d w

ith t

head

ditio

n of

cod

eine

60

mg

(usin

gT

OT

PAR%

as

outc

ome)

in b

oth

indi

rect

and

hea

d-to

-hea

dco

mpa

rison

s”

App

ropr

iate

inte

rpre

tati

on?

Yes

Resu

lts fo

rSP

ID%

, Res

RRan

d Re

mRR

also

avai

labl

e in

the

revi

ew.

The

obs

erve

ddi

ffere

nce

inT

OT

PAR

was

not

conf

irmed

usin

gth

e m

ore

clin

ical

lym

eani

ngfu

lRe

mRR

Appendix 3

94 TA

BLE

20

Syst

emat

ic re

view

s re

port

ing

naive

indi

rect

com

paris

ons

and

dire

ct c

ompa

rison

s

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Mar

shal

l and

Irvi

ne21

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

30-p

oint

sco

ring

syst

emus

ed.12

9Va

lidity

sco

re fo

r ea

chin

clud

ed t

rial p

rese

nted

. RC

Tson

ly in

clud

ed

Ass

essm

ent

ofhe

tero

gene

ity:

Hom

ogen

eity

with

in g

roup

s of

tria

ls co

nfirm

ed u

sing

Bres

low

–Day

tes

t

Met

hod

used

:N

aive

pre

sent

atio

n of

pool

ed r

espo

nse

rate

sac

ross

all

tria

ls fo

r ea

chtr

eatm

ent

Com

pari

sons

mad

e:Re

ctal

cor

ticos

tero

ids/

plac

ebo

or o

ther

tre

atm

ent

(n=

16)

5-A

SA/o

ther

tre

atm

ent

(n=

9)

Res

ults

:Po

oled

impr

ovem

ent

rate

s by

sym

ptom

atic

, end

osco

pic

and

hist

olog

ical

crit

eria

:Re

ctal

cor

ticos

tero

ids

77%

,66

% a

nd 5

2%, r

espe

ctiv

ely;

5-A

SA 8

2%, 7

3% a

nd 6

6%,

resp

ectiv

ely

Com

pari

sons

mad

e:Re

ctal

cor

ticos

tero

ids/

5-A

SA (n

=7)

Res

ults

:Po

oled

OR

(95%

CI)

Sym

ptom

atic

impr

ovem

ent:

1.36

(0.8

8to

2.0

9)

Endo

scop

icim

prov

emen

t: 1.

06 (0

.61

to 1

.85)

Hist

olog

ical

impr

ovem

ent:

2.27

(1.2

2 to

4.2

7)

Emph

asis

give

n to

res

ults

from

dire

ct c

ompa

rison

App

ropr

iate

inte

rpre

tati

on?

Yes

Miln

e et

al.23

a

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. C

ompa

rativ

ecl

inic

al s

tudi

es in

clud

ed

Ass

essm

ent

ofhe

tero

gene

ity:

�2

test

of h

eter

ogen

eity

was

unde

rtak

en. N

o sig

nific

ant

hete

roge

neity

was

sho

wn

Met

hod

used

:N

aive

ICSu

mm

ary

stat

istic

of

repo

rted

adv

erse

eve

nts

and

with

draw

als

base

d on

data

from

arm

s of

all

tria

ls

Com

pari

sons

mad

e:a.

Roxi

thro

myc

in/o

ther

mac

rolid

e or

age

ntco

mm

only

use

d as

firs

t lin

eth

erap

y (n

=13

)b.

Eryt

hrom

ycin

/oth

erm

acro

lide

or a

gent

com

mon

ly u

sed

as fi

rst

line

ther

apy

(n=

15)

Res

ults

:A

dver

se e

vent

s, r

ate

% (9

5%C

I):a.

10%

(8 t

o 12

%)

b.24

.8%

(22

to 2

7%)

Diff

eren

ce (S

E) 1

4% (1

.5%

)

a.2.

0% (1

to

3%)

b.7.

1% (6

to

9%)

Diff

eren

ce (S

E) 5

% (0

.045

%)

Com

pari

sons

mad

e:c.

Ro

xith

rom

ycin

/er

ythr

omyc

in (n

=3)

Res

ults

:N

o su

mm

ary

stat

istic

for

DC

alo

ne p

rese

nted

.A

dver

se e

vent

s:Ro

xith

rom

ycin

12.

8%(1

0.5

to 1

7.5%

)Er

ythr

omyc

in 2

7.15

% (8

to 5

1.3%

)

With

draw

als:

Ro

xith

rom

ycin

: 1.8

%Er

ythr

omyc

in: 2

.65%

Aut

hors

disc

usse

d th

epo

ssib

ility

of s

yste

mat

ic b

ias

due

to d

issim

ilar

patie

ntgr

oups

, but

con

clud

ed t

hat

this

is un

likel

y in

thi

s ca

se

App

ropr

iate

inte

rpre

tati

on?

Unc

lear

Pote

ntia

l con

foun

ding

fact

ors

disc

usse

d an

dco

nclu

ded

no s

igni

fican

tdi

ffere

nce

betw

een

grou

ps in

ter

ms

ofcl

inic

al e

ffica

cy, a

ge,

gend

er, s

ettin

gs,

dura

tion

of t

reat

men

t,in

dica

tions

or

year

of

publ

icat

ion


95


TA

BLE

20

Syst

emat

ic re

view

s re

port

ing

naive

indi

rect

com

paris

ons

and

dire

ct c

ompa

rison

s (c

ont’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

Pope

et

al.25

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Stud

ies

wer

e as

sess

ed fo

rbl

indi

ng, r

ando

misa

tion

and

cont

rol g

roup

. Ind

ivid

ual

qual

ity a

sses

smen

t fo

r ea

chst

udy

not

repo

rted

Ass

essm

ent

ofhe

tero

gene

ity:

Het

erog

enei

ty w

as e

xam

ined

and

a ra

ndom

effe

cts

mod

elus

ed

Met

hod

used

:N

aive

IC. A

djus

ted

for

diet

ary

salt

inta

ke

Com

pari

sons

mad

e:a.

Nap

roxe

n (n

=4)

b.In

dom

etha

cin

(n=

57)

c.Pi

roxi

cam

(n=

4)d.

Sulin

dac

(n=

23)

e.A

spiri

n (n

=4)

f.Ib

upro

fen

(n=

6)g.

Plac

ebo

(n=

10)

(n =

num

ber

of t

reat

men

tgr

oups

for

hype

rten

sive

patie

nts)

Res

ults

:M

ean

MA

P ±

SEM

(adj

uste

dfo

r sa

lt in

take

):b.

3.59

±1.

12

d.–0

.16

±1.

45M

ean

diffe

renc

e in

MA

P 3.

75 ±

1.83

Mea

n M

AP

±SE

M (a

djus

ted

for

salt

inta

ke):

b.3.

59 ±

1.12

g.

–2.5

9 ±

1.78

Mea

n di

ffere

nce

in M

AP

6.18

±2.

10

Com

pari

sons

mad

e:h.

Indo

met

haci

n/su

linda

c(n

=16

)i.

Indo

met

haci

n/pl

aceb

o(n

=11

)

Res

ults

:M

ean±

SEM

diff

eren

ce in

MA

P:a.

4.1

5±1.

00b.

3.9

3±1.

42

The

res

ults

of t

he n

aive

ICw

ere

com

pare

d w

ith t

hose

of

dire

ct c

ompa

rison

s w

hen

avai

labl

e

App

ropr

iate

inte

rpre

tati

on?

Yes

Resu

lts fo

r ot

her

NSA

IDs

are

avai

labl

e in

the

revi

ew

aN

ot id

entif

ied

thro

ugh

DA

RE.

Appendix 3

96 TA

BLE

21

Syst

emat

ic re

view

s re

port

ing

adju

sted

indi

rect

com

paris

ons

only

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Aro

37a

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

RCTs

incl

uded

Ass

essm

ent

ofhe

tero

gene

ity:

No

form

al t

est

ofhe

tero

gene

ity r

epor

ted

Met

hod

used

:A

djus

ted

ICA

ge-s

tand

ardi

sed

Com

pari

sons

mad

e:a.

PEP+

EE/o

rchi

dect

omy

(n=

1)b.

PEP/

orch

idec

tom

y (n

=1)

Res

ults

:A

ge-s

tand

ardi

sed

deat

h ra

tera

tio (9

5% C

I)A

ll-ca

use

mor

talit

y:a.

2.31

(1.9

2 to

2.7

9)b.

1.50

(1.0

6 to

2.1

1)

CH

D d

eath

s:a.

1.51

(1.1

0 to

2.0

8)b.

0.17

(0.0

5 to

0.5

7)

Com

pari

sons

mad

e:N

ot d

one

No

tria

l inc

lude

d in

the

revi

ew d

irect

lyco

mpa

red

PEP

+EE

with

PEP

alo

ne

Res

ults

:N

ot a

pplic

able

Aut

hors

con

clud

ed t

hat

“int

ram

uscu

lar

PEP

mon

othe

rapy

is a

ssoc

iate

dw

ith lo

w c

ardi

ovas

cula

rm

orta

lity

and

with

an

all-c

ause

and

pros

tatic

can

cer

mor

talit

yeq

ual t

o or

chid

ecto

my”

App

ropr

iate

inte

rpre

tati

on?

No

A s

ubse

quen

tpu

blic

atio

n48hi

ghlig

hts

the

maj

or d

iffer

ence

s in

incl

usio

n cr

iteria

betw

een

the

two

stud

ies

incl

uded

in t

here

view

. The

obs

erve

dlo

w m

orta

lity

in P

EPal

one

may

be

due

tofla

ws

in t

hem

etho

dolo

gy

Boer

sma

et a

l.9

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

RCTs

incl

udin

g

100

patie

nts

wer

ein

clud

ed

Ass

essm

ent

ofhe

tero

gene

ity:

Het

erog

enei

ty w

as e

xam

ined

usin

g th

e Br

eslo

w–D

ay t

est.

Sens

itivi

ty a

naly

sis w

asun

dert

aken

whe

nhe

tero

gene

ity e

xist

ed

Met

hod

used

:A

djus

ted

IC, p

rese

nted

info

rest

plo

tLi

near

and

non

-line

arre

gres

sion

Com

pari

sons

mad

e:T

ime

to fi

brin

olyt

ic t

hera

py(h

ours

:)a.

0–1/

ctrl

b.≥

1–2/

ctrl

c.≥

2–3/

ctrl

d.≥

3–6/

ctrl

e.≥

6–12

/ctr

lf.

≥12

–24/

ctrl

Tota

l n =

22

Res

ults

:Su

bgro

up r

esul

ts. A

bsol

ute

redu

ctio

n (S

D) i

n m

orta

lity

per

1000

a.

65 (1

4)b.

37 (9

)c.

26 (6

)d.

29 (5

)e.

18 (6

)f.

9 (7

)

Not

don

eT

he a

utho

rs c

oncl

ude

that

“The

ben

efic

ial e

ffect

of

fibrin

olyt

ic t

hera

py is

subs

tant

ially

hig

her

in p

atie

nts

pres

entin

g w

ithin

2 h

afte

rsy

mpt

om o

nset

com

pare

d to

thos

e pr

esen

ting

late

r”

The

aut

hors

do

not

men

tion

the

pote

ntia

l pro

blem

sas

soci

ated

with

ICs.

How

ever

,in

an

earli

er m

eta-

anal

ysis

(upo

n w

hich

thi

s re

view

isba

sed)

it is

arg

ued

that

“If

patie

nt c

ateg

orie

s ca

n be

arra

nged

in s

ome

mea

ning

ful

orde

r th

en …

may

be

reas

onab

ly r

elia

bly

info

rmat

ive

…”

App

ropr

iate

inte

rpre

tati

on?

No

Alte

rnat

ive

anal

ysis

ofpr

evio

us m

eta-

anal

ysis47


97


TA

BLE

21

Syst

emat

ic re

view

s re

port

ing

adju

sted

indi

rect

com

paris

ons

only

(co

nt’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Gar

g an

d Yu

suf7

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

RCTs

incl

uded

Ass

essm

ent

ofhe

tero

gene

ity:

�2

test

for

hete

roge

neity

was

unde

rtak

en. N

o sig

nific

ant

hete

roge

neity

sho

wn

for

tota

lm

orta

lity

Met

hod

used

:A

djus

ted

ICC

ompa

riso

ns m

ade:

Diff

eren

t A

CE

inhi

bito

rs:

a.Be

naze

pril

hydr

ochl

orid

e/ct

rl (n

=2)

b.C

apto

pril/

ctrl

(n=

6)c.

Cila

zapr

il/ct

rl (n

=1)

d.En

alap

ril m

alea

te/c

trl

(n=

7)e.

Lisin

opril

/ctr

l (n

=4)

f.Pe

rindo

pril/

ctrl

(n=

1)g.

Qui

napr

ilhy

droc

hlor

ide/

ctrl

(n=

5)h.

Ram

ipril

/ctr

l (n

=6)

Diff

eren

t du

ratio

n of

follo

w-

up:

�90

day

s/ct

rl (n

=32

)>

90 d

ays/

ctrl

(n=

12)

Res

ults

:To

tal m

orta

lity

(OR)

Diff

eren

t A

CE

inhi

bito

rs:

a.0.

36b.

0.79

c.0.

12d.

0.78

e.0.

62f.

0.14

g.0.

79h.

0.67

Dur

atio

n of

follo

w-u

p:�

90 d

ays/

ctrl

0.56

(0.4

4, 0

.70)

> 9

0 da

ys/c

trl 0

.87

(0.7

5,1.

01)

Not

don

e: n

o tr

ials

mak

ing

dire

ctco

mpa

rison

bet

wee

ndi

ffere

nt A

CE

inhi

bito

rsre

port

ed in

the

rev

iew

Aut

hors

con

clud

ed t

hat

“sim

ilar

bene

fits

wer

eob

serv

ed w

ith s

ever

al d

iffer

ent

AC

E in

hibi

tors

, alth

ough

dat

aw

ere

larg

ely

base

d on

ena

lapr

ilm

alea

te, c

apto

pril,

ram

ipril

,qu

inap

ril h

ydro

chlo

ride

and

lisin

opril

”

“The

gre

ates

t ef

fect

was

see

ndu

ring

the

first

3 m

onth

s, b

utad

ditio

nal b

enef

it w

asob

serv

ed d

urin

g fu

rthe

rtr

eatm

ent”

App

ropr

iate

inte

rpre

tati

on?

Unc

lear

It is

poss

ible

to

mak

ew

ithin

-tria

l com

paris

onfo

r du

ratio

n of

follo

w-

up a

nd e

ffect

, alth

ough

the

conc

lusio

n is

unlik

ely

to b

e di

ffere

nt

Appendix 3

98 TA

BLE

21

Syst

emat

ic re

view

s re

port

ing

adju

sted

indi

rect

com

paris

ons

only

(co

nt’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Hol

me35

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

RCTs

with

6-m

onth

follo

w-u

pin

clud

ed

Ass

essm

ent

ofhe

tero

gene

ity:

Met

a-re

gres

sion

unde

rtak

en

Met

hod

used

:M

eta-

regr

essio

n (m

ultip

le)

adju

sted

by

chol

este

rol

chan

ges

and

base

line

risk

ofC

AD

Com

pari

sons

mad

e:D

iet/

plac

ebo

(n=

12)

Hor

mon

es/p

lace

bo (n

=3)

Fibr

ates

/pla

cebo

(n=

7)St

atin

s/pl

aceb

o (n

=3)

Oth

er (o

ther

drug

s/su

rger

y)/p

lace

bo (n

=6)

Res

ults

:To

tal m

orta

lity:

Die

t/fib

rate

s O

R 0.

975

(p>

0.05

), z

=0.

84

Stat

ins/

fibra

tes

OR

0.83

3(p

<0.

05),

z=

3.37

Not

don

e: n

o tr

ials

mak

ing

dire

ctco

mpa

rison

bet

wee

ndi

ffere

nt in

terv

entio

nsre

port

ed in

the

rev

iew

App

ropr

iate

inte

rpre

tati

on?

Unc

lear

Hom

ocys

tein

e Lo

wer

ing

Tria

lists

’ Col

labo

ratio

n8

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

RCTs

incl

uded

Ass

essm

ent

ofhe

tero

gene

ity:

Het

erog

enei

ty a

cros

s st

udie

sas

sess

ed b

y m

ultiv

aria

tere

gres

sion

anal

ysis

Met

hod

used

:A

djus

ted

IC u

sing

a no

-tr

eatm

ent

cont

rol a

sco

mm

on c

ompa

rato

r.Ill

ustr

ated

in g

raph

Com

pari

sons

mad

e:D

iffer

ent

folic

aci

d re

gim

en:

a.1

mg/

ctrl

(n=

5)b.

1–3

mg/

ctrl

(n=

5)

c.>

3 m

g/ct

rl (n

=4)

Res

ults

:%

red

uctio

n in

blo

odho

moc

yste

ine

(95%

CI):

a.26

% (2

35 t

o 29

%)

b.25

% (2

0 to

29%

)c.

255

(21

to 2

8%)

Not

don

eA

utho

rs s

ugge

st a

wid

e ra

nge

of d

oses

(0.5

–5 m

g) t

o be

simila

rly e

ffect

ive

App

ropr

iate

inte

rpre

tati

on?

Yes


99


TA

BLE

21

Syst

emat

ic re

view

s re

port

ing

adju

sted

indi

rect

com

paris

ons

only

(co

nt’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Koch

et

al.34

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Valid

ity a

sses

sed

usin

gpr

evio

usly

pub

lishe

dcr

iteria

.130,

131

Med

ian

qual

itysc

ore

for

all t

rials

pres

ente

d.O

nly

RCTs

incl

uded

Ass

essm

ent

ofhe

tero

gene

ity:

A L

’Abb

e pl

ot a

nd t

he Q

stat

istic

wer

e us

ed t

o ex

amin

ehe

tero

gene

ity

Met

hod

used

:A

djus

ted

ICC

ompa

riso

ns m

ade:

H2

bloc

kers

/pla

cebo

(nin

etr

ials)

Miso

pros

tol/p

lace

bo (t

entr

ials)

Res

ults

:Ra

te d

iffer

ence

(95%

CI)

Shor

t-te

rm r

isk fo

r ga

stric

ulce

r:

H2

bloc

kers

/pla

cebo

: –0.

9%(–

4.0

to 2

.2%

)M

isopr

osto

l/pla

cebo

: –13

.3%

(–25

.7 t

o –0

.9%

)

Long

-ter

m r

isk fo

r ga

stric

ulce

r:H

2bl

ocke

rs/p

lace

bo: –

0.3%

(–2.

9 to

2.2

%)

Miso

pros

tol/p

lace

bo: –

8.4%

(–17

.7 t

o –1

.0%

)

Not

don

e: n

o tr

ials

mak

ing

dire

ctco

mpa

rison

bet

wee

n H

2bl

ocke

rs a

nd m

isopr

otol

repo

rted

in t

he r

evie

w

Aut

hor’

s cl

aim

tha

t “g

astr

icul

cer

was

foun

d to

be

signi

fican

tly r

educ

ed b

ym

isopr

osto

l, bo

th in

sho

rt-

term

and

long

-ter

m N

SAID

trea

tmen

t, bu

t no

t by

H2

bloc

kers

”

“We

foun

d di

scre

panc

ies

betw

een

the

resu

lts o

f H2

bloc

ker

and

miso

pros

tol t

rials.

Stud

ies

on m

isopr

osto

lad

mitt

ed s

ubje

cts

with

ahi

gher

risk

for

the

deve

lopm

ent

of g

astr

icda

mag

e”

App

ropr

iate

inte

rpre

tati

on?

Unc

lear

Resu

lts a

lso p

rese

nted

for

gast

ric le

sions

,du

oden

al u

lcer

s an

ddu

oden

al le

sions

Lefe

ring

and

Neu

geba

uer39

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

A p

revi

ously

pub

lishe

d qu

ality

chec

klist

was

use

d.13

2A

qua

lity

scor

e w

as p

rese

nted

for

each

tria

l. O

nly

RCTs

incl

uded

Ass

essm

ent

ofhe

tero

gene

ity:

A t

est

of h

eter

ogen

eity

was

cond

ucte

d ac

cord

ing

toC

ochr

an a

nd a

ran

dom

effe

cts

mod

el u

sed

Met

hod

used

:A

djus

ted

ICC

ompa

riso

ns m

ade:

a.Lo

w-d

ose

cort

icos

tero

id/c

trl (

n=

5)b.

Hig

h-do

seco

rtic

oste

roid

/ctr

l (n

=5)

Res

ults

:M

orta

lity

rate

% (9

5% C

I)a.

–1.9

% (–

20.0

to

16.2

%)

b.3.

6% (2

.5 t

o 9.

8%)

Not

don

e. N

o he

ad-t

o-he

ad t

rials

of h

igh

vs lo

wdo

se a

vaila

ble

“Nei

ther

the

typ

e of

ste

roid

used

nor

the

sep

arat

ion

into

low

-dos

e or

hig

h-do

sere

gim

en in

dica

ted

are

mar

kabl

e di

ffere

nce

betw

een

the

ster

oid

grou

p an

dco

ntro

l gro

up”

App

ropr

iate

inte

rpre

tati

on?

Unc

lear

Appendix 3

100 TA

BLE

21

Syst

emat

ic re

view

s re

port

ing

adju

sted

indi

rect

com

paris

ons

only

(co

nt’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Leiz

orov

icz30

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

RCTs

incl

uded

Ass

essm

ent

ofhe

tero

gene

ity:

No

signi

fican

t he

tero

gene

ityw

as o

bser

ved

Met

hod

used

:N

o fo

rmal

met

hod

used

.C

ompa

rison

of O

R,pr

esen

ted

on fo

rest

plo

t

Com

pari

sons

mad

e:a.

LMW

H h

ospi

tal/U

FHho

spita

l (n

=18

)b.

LMW

H h

ome/

UFH

hos

pita

l(n

=2)

Res

ults

:Re

curr

ent

thro

mbo

embo

licev

ents

(OR)

:a.

0.76

b.0.

79

Mor

talit

y:a.

0.66

b.0.

75

Not

don

e. T

rials

ongo

ing

“Alth

ough

thi

s ap

proa

chre

duce

d th

e st

atist

ical

pow

erfo

r ea

ch s

ubgr

oup,

effi

cacy

resu

lts w

ere

simila

r fo

rre

curr

ence

of t

hrom

boem

bolic

even

ts o

r de

ath”

App

ropr

iate

inte

rpre

tati

on?

Yes

IC a

lso m

ade

for

rout

eof

adm

inist

ratio

n

Moo

re a

nd M

cQua

y36

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Valid

ity a

sses

sed

usin

g th

eJa

dad

scor

e133

All

tria

ls sc

ored

max

imum

5 p

oint

s

Ass

essm

ent

ofhe

tero

gene

ity:

No

form

al e

xam

inat

ion

ofhe

tero

gene

ity p

rese

nted

Met

hod

used

:A

djus

ted

IC, u

sing

plac

ebo

as c

ompa

rato

r. T

he s

ame

plac

ebo

grou

p m

ay h

ave

been

use

d to

com

pare

with

diffe

rent

act

ive

drug

s w

ithin

tria

ls

Com

pari

sons

mad

e:a.

Cod

eine

(60

mg)

/pla

cebo

b.Tr

amad

ol (5

0 m

g)/p

lace

boc.

Tram

adol

(75

mg)

/pla

cebo

d.Tr

amad

ol (1

00 m

g)/p

lace

boe.

Tram

adol

(150

mg)

/pla

cebo

f.A

ceta

min

ophe

n (6

50 m

g)pl

us p

ropo

xyph

ene

(100

mg)

/pla

cebo

g.A

spiri

n (6

50 m

g) p

lus

code

ine

(60

mg)

/pla

cebo

Res

ults

:RR

(95%

CI)

for

patie

nts

achi

evin

g

50%

of %

max

-T

OT

PAR

Den

tal p

ain:

c.1.

3 (0

.8 t

o 2.

1)d.

2.9

(1.6

to

5.2)

e.2.

7 (1

.1 t

o 6.

5)f.

3.8

(2.4

to

5.8)

g.4.

8 (2

.1 t

o 11

.1)

h.4.

0 (1

.7 t

o 9.

4)i.

3.8

(2.2

to

6.8)

Not

don

e, a

lthou

ghpo

ssib

le fr

om d

ata

pres

ente

d

Inte

rpre

ted

as if

dire

ctco

mpa

rison

s w

ere

mad

e

App

ropr

iate

inte

rpre

tati

on?

Unc

lear

Dat

a on

pos

tsur

gica

lpa

in a

lso p

rese

nted

inpa

per


101


TA

BLE

21

Syst

emat

ic re

view

s re

port

ing

adju

sted

indi

rect

com

paris

ons

only

(co

nt’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Non

-sm

all C

ell L

ong

Can

cer

Col

labo

rativ

e G

roup

32

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

RCTs

incl

uded

Ass

essm

ent

ofhe

tero

gene

ity:

�2

test

for

hete

roge

neity

was

unde

rtak

en fo

r bo

th b

etw

een

and

with

in c

hem

othe

rapy

cate

gorie

s. W

hen

gros

she

tero

gene

ity w

as d

etec

ted,

the

ratio

nale

for

com

bini

ngda

ta w

as q

uest

ione

d an

dso

urce

s of

het

erog

enei

tyex

amin

ed, r

athe

r th

an u

sing

ara

ndom

effe

cts

mod

el

Met

hod

used

:A

djus

ted

ICC

ompa

riso

ns m

ade:

a.Su

rger

y pl

us lo

ng-t

erm

alky

latin

g ag

ent/

surg

ery

alon

eb.

Surg

ery

plus

cisp

latin

/sur

gery

alo

nec.

Radi

cal r

adio

ther

apy

(RT

)pl

us a

lkyl

atin

gag

ents

/rad

ical

RT

alo

ned.

Radi

cal R

T p

lus

cisp

latin

/rad

ical

RT

alo

ne

Res

ults

:O

–E d

eath

s (v

aria

nce)

a.55

.53

(394

.74)

b.–2

1.58

(151

.83)

c.–2

.83

(140

.23)

d.–5

7.08

(411

.18)

Not

don

e: n

o tr

ials

mak

ing

dire

ctco

mpa

rison

bet

wee

nch

emot

hera

py d

rugs

repo

rted

in t

he r

evie

w

“Fur

ther

ran

dom

ised

tria

ls ar

ene

eded

to

dete

rmin

e w

hich

regi

men

s ar

e th

e m

ost

effe

ctiv

e of

the

mod

ern

chem

othe

rapi

es s

tudi

ed”

App

ropr

iate

inte

rpre

tati

on?

Yes

Oth

er c

ompa

rison

spr

esen

ted

in a

rtic

le.

Onl

y m

ain

resu

ltsex

trac

ted

Pign

on e

t al

.31

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

RCTs

incl

uded

Ass

essm

ent

ofhe

tero

gene

ity:

Test

s of

het

erog

enei

ty w

ere

perf

orm

ed; h

owev

er, t

heau

thor

s st

ate

that

“su

bsta

ntia

lhe

tero

gene

ity d

oes

not

inva

lidat

e th

e re

sults

of a

met

a-an

alys

is”

Met

hod

used

:A

djus

ted

ICC

ompa

riso

ns m

ade:

a.Ea

rly R

T/c

trl (

n=

7)b.

Late

RT

/ctr

l (n

=6)

c.W

ith s

eque

ntia

l RT

/ctr

l(n

=8)

d.W

ithou

t se

quen

tial R

T/c

trl

(n=

5)

Res

ults

:O

R (9

5% C

I)a.

0.88

(0.7

8 to

0.9

8)b.

0.81

(0.6

9 to

0.9

4)c.

0.86

(0.7

5 to

1.0

0)d.

0.85

(0.7

5 to

0.9

6)

Not

don

e. N

o tr

ials

dire

ctly

com

parin

gea

rly/la

te, w

ith/w

ithou

tse

quen

tial R

T a

vaila

ble

“Ind

irect

com

paris

on o

f ear

lyw

ith la

te r

adio

ther

apy

and

ofse

quen

tial w

ith n

on-s

eque

ntia

lra

diot

hera

py d

id n

ot r

evea

l any

optim

al ti

me

for

trea

tmen

t”

“The

sel

ectio

n of

an

optim

alsc

hedu

le o

f CT

com

bine

d w

ithRT

tha

t w

ould

lead

to

a m

ajor

incr

ease

in s

urvi

val w

ithm

inim

al t

oxic

ity is

the

prin

cipa

lch

alle

nge

raise

d by

our

stud

y….W

e ho

pe t

hat

the

resu

lts o

f fut

ure

tria

ls w

illse

ttle

thi

s qu

estio

n”

App

ropr

iate

inte

rpre

tati

on?

Yes

Appendix 3

102 TA

BLE

21

Syst

emat

ic re

view

s re

port

ing

adju

sted

indi

rect

com

paris

ons

only

(co

nt’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Poyn

ard

et a

l.38

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

The

met

hodo

logi

cal q

ualit

y of

each

tria

l was

ass

esse

d us

ing

a14

-item

que

stio

nnai

re a

ndsc

ored

bet

wee

n –2

and

26.

Abr

eakd

own

of t

he s

corin

g fo

rea

ch t

rial i

s pr

esen

ted

Ass

essm

ent

ofhe

tero

gene

ity:

Sens

itivi

ty a

naly

sis w

aspe

rfor

med

by

stra

tific

atio

nac

cord

ing

to H

2bl

ocke

r us

ing

both

ran

dom

and

fixe

d ef

fect

s

Met

hod

used

:A

djus

ted

ICC

ompa

riso

ns m

ade:

Lans

opra

zole

/ran

itidi

ne o

rfa

mot

idin

e (n

=5)

Oth

er d

rugs

/ran

iditi

ne o

rfa

mot

idin

e (n

=?)

Res

ults

:O

R (9

5% C

I) fo

r 4-

wee

khe

alin

g ra

te in

com

paris

onw

ith r

aniti

dine

or

fam

otid

ine

Lans

opra

zole

: 2.5

(1.7

to

3.7)

Om

epra

zole

: 2.9

(1.5

to

5.7)

Rani

tidin

e or

fam

otid

ine:

0.9

(0.7

to

1.2)

Niz

atid

ine:

1.0

(0.8

to

1.4)

Cim

etid

ine:

1.8

Sucr

alfa

te: 1

.0 (0

.7 t

o 1.

4)

Lans

opra

zole

dire

ctly

com

pare

d w

ithra

nitid

ine

or fa

mot

idin

e.A

utho

rs m

entio

n an

RCT

dire

ctly

com

parin

gla

nsop

razo

le v

ersu

som

epra

zole

in a

cute

duod

enal

ulc

erat

ion;

how

ever

, thi

s tr

ial i

s no

tin

clud

ed in

the

met

a-an

alys

is

The

indi

rect

com

paris

on w

asus

ed t

o ra

nk e

ffica

cy. I

t w

asco

nclu

ded

that

“th

ere

was

asig

nific

ant

diffe

renc

e be

twee

nth

e ef

ficac

y of

om

epra

zole

and

lans

opra

zole

on

the

one

hand

and

all t

he o

ther

gro

ups

on t

heot

her”

App

ropr

iate

inte

rpre

tati

on?

No

A p

revi

ous

met

a-an

alys

is us

ed t

he s

ame

indi

rect

met

hod.

134

Ross

ouw

33

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

RCTs

incl

uded

Ass

essm

ent

ofhe

tero

gene

ity:

The

aut

hors

ack

now

ledg

e th

atcl

inic

al h

eter

ogen

eity

exi

sts

betw

een

the

incl

uded

stu

dies

.C

orre

latio

n an

alys

es w

ere

perf

orm

ed t

o as

sess

whe

ther

the

ORs

for

dise

ase

chan

gew

ere

rela

ted

to b

asel

ine

or in

-tr

ial l

ow-d

ensit

y lip

opro

tein

leve

ls or

to

rela

tive

orab

solu

te d

iffer

ence

s be

twee

ntr

eatm

ent

and

cont

rol g

roup

sdu

ring

the

tria

l

Met

hod

used

:A

djus

ted

ICC

ompa

riso

ns m

ade:

Vario

us in

terv

entio

ns/c

trl:

a.Li

fest

yle/

ctrl

(n=

3)b.

Resin

s/ct

rl (n

=2)

c.St

atin

s/ct

rl (n

=4)

d.C

ombi

natio

n/ct

rl (n

=5)

e.Su

rger

y/ct

rl (n

=1)

Res

ults

:O

R (9

5% C

I) fo

rca

rdio

vasc

ular

eve

nts

pres

ente

d in

fore

st p

lot.

Onl

y th

e de

tails

of t

he o

vera

llO

R fo

r al

l int

erve

ntio

nspr

esen

ted

in t

he t

ext

0.53

(0.4

5 to

0.6

3)

Not

don

eA

utho

rs c

oncl

uded

tha

t “t

here

is no

con

clus

ive

evid

ence

of a

clas

s ef

fect

”

App

ropr

iate

inte

rpre

tati

on?

Yes


103


TA

BLE

21

Syst

emat

ic re

view

s re

port

ing

adju

sted

indi

rect

com

paris

ons

only

(co

nt’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

aN

ot id

entif

ied

thro

ugh

DA

RE.

CH

D, c

oron

ary

hear

t di

seas

e; C

T, c

hem

othe

rapy

; RT,

rad

ioth

erap

y.

Tram

er e

t al

.11

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

The

Jada

d sc

ore

was

use

d133

Resu

lts n

ot p

rese

nted

Ass

essm

ent

ofhe

tero

gene

ity:

No

form

al e

xam

inat

ion

ofhe

tero

gene

ity p

rese

nted

Met

hod

used

:A

djus

ted

ICC

ompa

riso

ns m

ade:

Diff

eren

t do

ses

of d

rope

ridol

: a.

10 �

g kg

–1(n

= 1

)b.

20 �

g kg

–1(n

= 1

)c.

50 �

g kg

–1(n

= 2

)d.

75 �

g kg

–1(n

= 1

0)

Res

ults

:A

bsen

ce o

f ear

ly v

omiti

ng:

OR

(95%

CI)

a.1.

6 (0

.4 t

o 7.

2)b.

1.9

(0.7

to

5)c.

1.5

(0.7

to

3.2)

d.3.

3 (2

.4 t

o 4.

7)

Not

don

eT

he a

utho

rs’ c

autio

n ab

out

the

adju

sted

IC w

as m

ainl

y du

e to

smal

l sam

ple

size.

Pot

entia

lbi

as w

as n

ot m

entio

ned

“The

num

ber

of c

hild

ren

stud

ied

at t

he lo

wes

t do

ses

was

sm

all a

nd t

he c

onfid

ence

inte

rval

s w

ide;

nev

erth

eles

sth

ere

wou

ld s

eem

to

besu

ffici

ent

info

rmat

ion

tosu

gges

t th

at t

he u

se o

fsu

bmax

imal

dos

es is

not

wor

thw

hile

”

App

ropr

iate

inte

rpre

tati

on?

Unc

lear

Det

ails

of o

ther

com

paris

ons

and

outc

ome

mea

sure

spr

esen

ted

in t

he r

evie

w

Zal

cber

g et

al.29

a

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

RCTs

incl

uded

Ass

essm

ent

ofhe

tero

gene

ity:

No

form

al e

xam

inat

ion

ofhe

tero

gene

ity p

rese

nted

Met

hod

used

:A

djus

ted

ICRe

gres

sion

anal

ysis

was

also

used

Com

pari

sons

mad

e:a.

≥10

g 5

-FU

/ctr

l (n

=3)

b.8–

9 g

5-FU

/ctr

l (n

=7)

c.<

8 g

5-FU

/ctr

l (n

=5)

d.O

ral C

T/c

trl (

n=

2)e.

5-FU

+le

vam

isole

/ctr

l(n

=2)

f.5-

FU/c

trl (

n=

15)

Res

ults

:O

R (9

5% C

I)a.

0.71

b.0.

79c.

0.93

d.1.

04e.

0.64

(0.4

9 to

0.8

5)f.

0.86

Not

don

e. N

o he

ad-t

o-he

ad t

rials

pres

ente

d in

the

revi

ew

“It

shou

ld b

e po

inte

d ou

t th

atth

e re

lativ

e ef

fect

s of

5-F

Udo

se a

nd o

f lev

amiso

le a

reba

sed

on in

dire

ct, n

on-

rand

omise

d co

mpa

rison

s in

this

anal

ysis,

so

that

conf

ound

ing

by t

he t

ype

ofpa

tient

s be

ing

stud

ied

in e

ach

tria

l is

a po

ssib

ility

”

The

aut

hors

also

con

clud

e th

atan

RC

T is

req

uire

d to

com

pare

5-F

U+

leva

miso

lew

ith 5

-FU

alo

ne

App

ropr

iate

inte

rpre

tati

on?

Yes

The

dos

e–re

spon

sere

latio

nshi

p is

quite

conv

inci

ng. H

owev

er,

the

bene

fit o

f add

ition

alle

vam

isole

to

5-FU

isun

clea

r as

the

tria

ls th

atin

clud

ed a

dditi

onal

leva

miso

le a

lso u

sed

high

er d

oses

of 5

-FU

Appendix 3

104 TA

BLE

22

Syst

emat

ic re

view

s re

port

ing

naive

indi

rect

com

paris

ons

only

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Bans

al a

nd B

eto42

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. P

rosp

ectiv

eco

ntro

lled

tria

ls w

ithtr

eatm

ent

allo

catio

n by

rand

om a

ssig

nmen

t or

cons

ecut

ive

enro

lmen

t

Ass

essm

ent

ofhe

tero

gene

ity:

Het

erog

enei

ty w

as a

sses

sed

and

a ra

ndom

effe

cts

mod

elus

ed. T

he a

ppro

pria

tene

ss o

fpo

olin

g w

as a

lso a

sses

sed

byco

mpa

ring

the

resu

lts b

etw

een

thos

e tr

ials

with

mat

ched

cont

rol a

nd a

ll tr

ials

Met

hod

used

:N

aive

IC o

f res

ults

of

diffe

rent

arm

s ac

ross

stud

ies

The

app

ropr

iate

ness

of

pool

ing

was

exa

min

ed b

ytw

o m

etho

ds (Z

-sco

re a

ndhe

tero

gene

ity t

est)

, but

the

resu

lts o

f the

tes

ts w

ere

not

pres

ente

d

Com

pari

sons

mad

e:Tr

eatm

ent

arm

s ac

ross

stu

dies

wer

e po

oled

to

com

pare

vario

us im

mun

osup

pres

sive

agen

ts p

lus

oral

pre

dniso

new

ith p

redn

isone

alo

ne (n

=?)

Res

ults

:A

bsol

ute

risk

diffe

renc

e (9

5%C

I) fo

r al

l im

mun

osup

pres

sive

agen

ts w

ithpr

edni

sone

/pre

dniso

ne a

lone

To

tal m

orta

lity:

13.

2% (2

.5 t

o23

.9%

)ES

RD: 1

2.9%

(2.2

to

23.6

%)

Not

don

eT

he c

oncl

usio

n w

as b

ased

on

ana

ive

IC w

ithou

t an

y ef

fort

to

adju

st fo

r po

tent

ial b

ias

and

conf

ound

ing.

Res

ults

wer

ein

terp

rete

d as

tho

ugh

DC

unde

rtak

en w

ith n

o di

scus

sion

of p

oten

tial b

iase

s

App

ropr

iate

inte

rpre

tati

on?

No

The

rev

iew

incl

uded

RCTs

or

quas

i-RC

Ts,

but

the

resu

lts o

f the

stud

ies

wer

e us

ed t

om

ake

betw

een-

stud

yco

mpa

rison

s, lo

sing

the

pow

er/r

igou

r of

rand

omisa

tion

Chi

ba e

t al

.40

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Blin

ding

and

met

hod

ofra

ndom

isatio

n w

ere

the

mai

nqu

ality

item

s as

sess

ed. O

nly

singl

e- o

r do

uble

-blin

d RC

Tsw

ere

incl

uded

Ass

essm

ent

ofhe

tero

gene

ity:

No

exam

inat

ion

ofhe

tero

gene

ity r

epor

ted

Met

hod

used

:N

aive

IC

For

each

dru

g cl

ass

linea

rre

gres

sion

anal

ysis

estim

ated

the

ave

rage

perc

enta

ge o

f pat

ient

s w

how

ere

heal

ed a

nd h

eart

burn

free

per

wee

k

Com

pari

sons

mad

e:43

stu

dies

use

d to

com

pare

PPIs

(om

epra

zole

,la

nsop

razo

le a

ndpa

ntop

razo

le),

H2R

As

(cim

etid

ine,

niz

atid

ine,

rant

idin

e an

d fa

mot

idin

e),

sucr

alfa

te, p

roki

netic

s, p

lace

boan

d ot

her

Res

ults

:M

ean

over

all h

ealin

gpr

opor

tion:

PPIs

: 83.

6 ±

11.4

%

H2R

A: 5

1 ±

17.1

%Su

cral

fate

: 39.

2 ±

22.4

%Pl

aceb

o: 2

8.2

±15

.6%

Not

don

e bu

t po

ssib

leT

he a

utho

rs c

oncl

ude

that

“mor

e co

mpl

ete

esop

hagi

tishe

alin

g an

d he

artb

urn

relie

f is

obse

rved

with

PPI

s vs

H2-

RAs”

App

ropr

iate

inte

rpre

tati

on?

No

The

rev

iew

incl

udes

only

sin

gle-

or

doub

le-

blin

d RC

Ts, b

ut n

ow

ithin

-stu

dyco

mpa

rison

was

mad

e

It is

not

clea

r w

heth

er it

is a

stan

dard

err

or o

rst

anda

rd d

evia

tion

follo

win

g th

e es

timat

edov

eral

l pro

port

ion


105


TA

BLE

22

Syst

emat

ic re

view

s re

port

ing

naive

indi

rect

com

paris

ons

only

(co

nt’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Con

lin e

t al

.46

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

doub

le-

blin

d RC

Ts w

ere

incl

uded

Ass

essm

ent

ofhe

tero

gene

ity:

No

exam

inat

ion

ofhe

tero

gene

ity r

epor

ted

Met

hod

used

:N

aive

IC

Ana

lysis

bas

ed o

n tr

eatm

ent

arm

s. T

he a

bsol

ute

(non

-pl

aceb

o co

rrec

ted)

wei

ghte

d av

erag

e bl

ood

pres

sure

red

uctio

n w

asca

lcul

ated

for

each

AIIA

Com

pari

sons

mad

e:43

tria

ls w

ere

used

to

com

pare

the

effi

cacy

of

losa

rtan

, val

sart

an, i

rbes

arta

nan

d ca

ndes

arta

n

Res

ults

:T

he a

bsol

ute

wei

ghte

dav

erag

e re

duct

ions

in d

iast

olic

and

syst

olic

blo

od p

ress

ure

wer

e co

mpa

rabl

e fo

r al

l AIIA

s

Not

don

e, a

lthou

ghpo

ssib

le t

o do

so

Aut

hors

con

clud

e th

at “

this

anal

ysis

sugg

ests

tha

t A

IIAlo

wer

blo

od p

ress

ure

with

simila

r ef

ficac

y w

hen

adm

inist

ered

at

thei

r us

ual

dose

s fo

r th

e tr

eatm

ent

ofhy

pert

ensio

n”

App

ropr

iate

inte

rpre

tati

on?

No

The

rev

iew

incl

udes

only

RC

Ts, b

ut n

ow

ithin

-stu

dyco

mpa

rison

was

mad

e.O

vere

stim

atio

n of

bloo

d pr

essu

rere

duct

ion

beca

use

ofre

gres

sion

to m

ean

effe

ct, o

win

g to

the

igno

ring

of t

he p

lace

bogr

oups

Cou

lter

et a

l.41

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

RCTs

wer

e in

clud

ed

Ass

essm

ent

ofhe

tero

gene

ity:

Clin

ical

het

erog

enei

tybe

twee

n tr

ials

was

not

ed. N

ote

sts

of h

eter

ogen

eity

wer

ere

port

ed

Met

hod

used

:N

aive

ICC

ompa

riso

ns m

ade:

31 s

tudi

es u

sed

to c

ompa

reth

e ef

ficac

y of

a v

arie

ty o

fdr

ugs

used

to

trea

tm

enor

rhag

ia

Res

ults

:D

rugs

list

ed a

ccor

ding

to

perc

enta

ge r

educ

tion

inm

enst

rual

blo

od lo

ss

Not

don

e, a

lthou

ghpo

ssib

le t

o do

so

Aut

hors

men

tion

limita

tions

such

as

lack

of p

lace

boco

ntro

ls, b

ut c

laim

the

rev

iew

of R

CTs

is s

atisf

acto

ry fo

rco

mpa

ring

rela

tive

effic

acy

ofdr

ugs

App

ropr

iate

inte

rpre

tati

on?

Unc

lear

The

rev

iew

incl

udes

only

RC

Ts, b

ut n

ow

ithin

-stu

dyco

mpa

rison

was

mad

e.H

alf o

f the

tria

lsin

clud

ed a

pla

cebo

grou

p, b

ut r

esul

ts o

fth

e pl

aceb

o co

ntro

lsw

ere

not

repo

rted

Appendix 3

106 TA

BLE

22

Syst

emat

ic re

view

s re

port

ing

naive

indi

rect

com

paris

ons

only

(co

nt’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Felso

n et

al.6

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Mod

ifica

tion

of a

pre

viou

slypu

blish

ed c

heck

list.13

5O

nly

RCTs

wer

e in

clud

ed

Ass

essm

ent

ofhe

tero

gene

ity:

�2

test

of h

eter

ogen

eity

was

unde

rtak

en a

nd a

ran

dom

effe

cts

mod

el u

sed

whe

nap

prop

riate

Met

hod

used

:N

aive

ICA

djus

tmen

t w

as m

ade

for

som

e co

varia

tes,

but

not

clea

r ho

w a

djus

tmen

t w

asm

ade

Com

pari

sons

mad

e:Tr

ials

of s

econ

d lin

e dr

ugs

totr

eat

rheu

mat

oid

arth

ritis

wer

e in

clud

ed:

a.Pl

aceb

o (n

=22

)b.

Ant

imal

aria

l dru

gs (n

=11

)c.

Aur

anof

in (n

=23

)d.

Inje

ctab

le g

old

(n=

29)

e.M

etho

trex

ate

(n=

7)f.

DP

(n=

19)

g.SS

Z (n

=6)

Res

ults

:Ef

ficac

y:A

uran

ofin

was

foun

d to

be

signi

fican

tly w

eake

r th

anm

etho

trex

ate,

inje

ctab

le g

old,

DP

and

SSZ

, and

slig

htly,

but

not

signi

fican

tly w

eake

r th

anan

timal

aria

l age

nts

Toxi

city

:In

ject

able

gol

d ha

d hi

gher

toxi

city

rat

es a

nd h

ighe

r to

tal

drop

out

than

any

oth

er d

rug

Com

pari

sons

mad

e:N

ot d

one,

alth

ough

poss

ible

to

do s

o

With

in-s

tudy

com

paris

ons

igno

red

Info

rmat

ion

from

all

outc

ome

mea

sure

s an

d fr

om e

ach

tria

lco

mbi

ned

into

a c

ompo

site

mea

sure

of t

reat

men

t ef

fect

App

ropr

iate

inte

rpre

tati

on?

No

Thi

s re

view

was

bee

nqu

oted

by

Impe

riale

and

Sper

off a

s pr

oof o

fus

ing

naiv

e IC

s of

RC

Tda

ta

Dire

ct c

ompa

rison

stud

ies

avai

labl

e


107


TA

BLE

22

Syst

emat

ic re

view

s re

port

ing

naive

indi

rect

com

paris

ons

only

(co

nt’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

cont

inue

d

Met

hod

used

:N

aive

IC b

ased

on

prem

iseth

at t

he t

reat

men

t gr

oups

wer

e cl

inic

ally

hom

ogen

eous

inco

mpo

sitio

n

Com

pari

sons

mad

e:a.

LMW

H/c

trl (

n=

20)

b.W

arfa

rin/c

trl (

n=

10)

c.C

ompr

essio

n st

ocki

ngs/

ctrl

(n=

6)

Res

ults

:N

NT

(95%

CI)

for

all D

VTa.

3.2

(2.9

to

3.7)

b.4.

3 (3

.0 t

o 8.

0)c.

3.9

(3.1

to

5.1)

Not

don

eA

ppro

pria

te in

terp

reta

tion

?N

oA

utho

rs g

ive

the

impr

essio

n th

at t

heir

conc

lusio

ns w

ere

base

don

evi

denc

e fr

om R

CTs

,ev

en t

houg

h on

lybe

twee

n-st

udy

com

paris

ons

wer

em

ade

Quo

tes

revi

ew b

yFe

lson

et a

l.6to

sup

port

use

of IC

met

hod

Mor

e ou

tcom

es a

ndco

mpa

rison

s av

aila

ble

inth

e re

view

Schm

iede

r et

al.43

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. O

nly

doub

le-

blin

d RC

Ts in

clud

ed

Ass

essm

ent

ofhe

tero

gene

ity:

Hom

ogen

eity

of d

ata

coul

dno

t be

con

firm

ed fo

r al

ltr

eatm

ent

arm

s, s

o se

nsiti

vity

anal

ysis

was

und

erta

ken

Met

hod

used

:N

aive

IC

All

trea

tmen

t ar

ms

of t

hesa

me

drug

wer

e co

mbi

ned

Com

pari

sons

mad

e:a.

Diu

retic

s (n

= 1

3)b.

�-B

lock

ers

(n =

21)

c.C

alci

um-c

hann

el b

lock

ers

(n=

19)

d.A

CE

inhi

bito

rs (n

=18

)e.

Plac

ebo

(n=

13)

Res

ults

:M

ean

(SD

) % d

ecre

ase

insy

stol

ic/d

iast

olic

blo

odpr

essu

rea.

10.7

(1.8

)/13.

1 (3

.5)

b.12

.8 (3

.8)/

15.4

(2.8

)c.

10.3

(3.6

)/13.

4 (2

.3)

d.11

.9 (4

.6)/

13.2

(5.4

)

Not

don

e, a

lthou

ghpo

ssib

le t

o do

so

The

aut

hors

disc

uss

the

impo

rtan

ce o

f ran

dom

isatio

nan

d sc

ient

ific

qual

ity o

f stu

dies

.H

owev

er, t

hey

igno

re w

ithin

-st

udy

com

paris

ons

and

mak

eno

att

empt

to

adju

st t

he IC

acco

rdin

g to

a c

omm

onpl

aceb

o gr

oup

App

ropr

iate

inte

rpre

tati

on?

No

See

upda

ted

revi

ew b

yth

e sa

me

auth

ors13

6

Impe

riale

and

Spe

roff45

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Onl

y RC

Ts in

clud

ed. T

rials

asse

ssed

on

incl

usio

n/ex

clus

ion

crite

ria; b

asel

ine

simila

rity;

blin

d ad

min

istra

tion

ofin

terv

entio

n; d

escr

iptio

n of

co-in

terv

entio

ns; d

rop-

outs

and

with

draw

als.

Sum

mar

ysc

ore

(max

. 10)

ass

igne

d to

each

tria

l

Ass

essm

ent

ofhe

tero

gene

ity:

Ana

lysis

bas

ed o

n pr

emise

tha

ttr

eatm

ent

grou

ps w

ere

clin

ical

ly h

omog

enou

s.Ra

ndom

effe

cts

mod

el u

sed

Appendix 3

108 TA

BLE

22

Syst

emat

ic re

view

s re

port

ing

naive

indi

rect

com

paris

ons

only

(co

nt’d

)

Stud

yM

etho

d of

IC

Indi

rect

com

pari

son

Dir

ect

com

pari

son

Des

crip

tion

of i

nter

pret

atio

nC

omm

ents

DP,

D-p

enic

illam

ine;

SSZ

, sul

fasa

lazi

ne.

Ung

e an

d Be

rsta

d44

Val

idit

y as

sess

men

tun

dert

aken

wit

hin

the

revi

ew:

Non

e re

port

ed. S

tudy

des

igns

uncl

ear

Ass

essm

ent

ofhe

tero

gene

ity:

Sens

itivi

ty a

naly

sis o

n m

ajor

diffe

renc

es in

dos

e, d

osag

ean

d du

ratio

n w

as p

erfo

rmed

Met

hod

used

:N

aive

ICC

ompa

riso

ns m

ade:

17 d

iffer

ent

trea

tmen

t gr

oups

For

exam

ple:

a.

Om

epra

zole

+ a

mox

ycill

in+

cla

rithr

omyc

in (n

=59

)b.

Bism

uth

+ n

itroi

mid

azol

e+

tet

racy

clin

e (n

=87

)

Res

ults

:Er

adic

atio

n ra

te:

a.87

% (r

ange

72–

100%

b.82

% (r

ange

43–

100%

)

Not

don

e, a

lthou

ghpo

ssib

le t

o do

so

The

aut

hors

con

clud

e th

at“o

mep

razo

le/c

larit

hrom

ycin

base

d tr

iple

reg

imen

s ar

e th

em

ost

effe

ctiv

e an

ti-H

. pyl

ori

ther

apeu

tic s

trat

egy,

slig

htly

supe

rior

to b

ismut

h tr

iple

regi

men

s”

App

ropr

iate

inte

rpre

tati

on?

No

Obs

erva

tiona

l stu

dies

and

RCTs

are

incl

uded

in t

he r

evie

w. D

etai

ls of

othe

r co

mpa

rison

s ar

epr

esen

ted

in t

he r

evie

w

The following reviews were identified from thesearches as potentially including indirect

comparisons or both direct and indirectcomparison of competing interventions. However,after assessment they did not include suitable dataand so were excluded.

Arriagada R, Pignon JP, Ihde DC, Johnson DH,Perry MC, Souhami RL, et al. Effect of thoracicradiotherapy on mortality in limited small celllung cancer. A meta-analysis of 13 randomizedtrials among 2,140 patients. Anticancer Res 1994;14(1B):333–5.

van Balkom AJ, Nauta MC, Bakker A. Meta-analysis on the treatment of panic disorder withagoraphobia; review and re-examination. ClinPsychol Psychother 1995;2:1–14.

Bhansali M, Vaidya J, Bhatt R, Patil P, Badwe R,Desai P. Chemotherapy for carcinoma of theesophagus: a comparison of evidence from meta-analyses of randomized trials and of historicalcontrol studies. Ann Oncol 1996;7:355–9.

Boyer W. Serotonin uptake inhibitors are superiorto imipramine and alprazolam in alleviating panicattacks: a meta-analysis. Int Clin Psychopharmacol1995;10:45–9.

Chang D, Wilson S. Meta-analysis of the clinicaloutcome of carbapenem monotherapy in theadjunctive treatment of intra-abdominalinfections. Am J Surg 1997;174:284–90.

Childhood ALL Collaborative Group. Durationand intensity of maintenance chemotherapy inacute lymphoblastic leukaemia: overview of 42trials involving 12 000 randomised children.Lancet 1996;347:1783–8.

de Craen A, Di Giulio G, Lampe-SchoenmaechersA, Kessels A, Kleijnen J. Analgesic efficacy andsafety of paracetamol-codeine combinations versusparacetamol alone: a systematic review. BMJ1996;313:321–5.

Droitcour J, Silberman G, Chelimsky E. A newform of meta-analysis for combining results fromrandomized clinical trials and medical-practice

databases. Int J Technol Assess Health Care 1993;9:440–9.

Eriksson S, Langstrom G, Rikner L, Carlsson R,Naesdal J. Omeprazole and H2 receptorantagonists in the acute treatment of duodenalulcer, gastric ulcer and reflux oesophagitis: ameta-analysis. Eur J Gastroenterol Hepatol 1995;7:467–75.

Golzari H, Cebul R, Bahler R. Atrial fibrillation:restoration and maintenance of sinus rhythm andindications for anticoagulation therapy. Ann InternMed 1996;125:311–23.

Halliday H. Overview of clinical trials comparingnatural and synthetic surfactants. Biol Neonate1995;67(Suppl):32–47.

Held P, Yusuf S. Calcium anatagonists in thetreatment of ischemic heart disease: myocardialinfarction. Coron Artery Dis 1994;5:21–6.

Hoes A, Grobbee D, Lubsen J. Does drugtreatment improve survival? Reconciling the trialsin mild-to-moderate hypertension. J Hypertens1995;13:805–11.

Hoes A, Grobbee D, Peet T, Lubsen J. Do non-potassium-sparing diuretics increase the risk ofsudden cardiac death in hypertensive patients?Drugs 1994;47:711–33.

Hooks M. Tacrolimus, a new immunosuppressant– a review of the literature. Ann Pharmacother1994;28:501–11.

Koes B, Assendelft W, van der Heijden G, Bouter L. Spinal manipulation for low back pain.An updated systematic review of randomizedclinical trials. Spine 1996;21:2860–71.

Linde K, Ramirez G, Mulrow C, Pauls A,Weidenhammer W, Melchart D. St John's wort fordepression – an overview and meta-analysis ofrandomised clinical trials. BMJ 1996;313:253–8.

MacRae H, McLeod R. Comparison ofhemorroidal treatment modalities: a meta-analysis.Dis Colon Rectum 1995;38:687–94.


109


Appendix 4

List of excluded reviews

McQuay H, Carroll D, Jadad AR, Wiffen P, Moore A. Anticonvulsant drugs for managementof pain: a systematic review. BMJ 1995;311:1047–52.

McQuay H, Tramer M, Nye B, Carroll D, Wiffen P,Moore R. A systematic review of antidepressants inneuropathic pain. Pain 1996;68:217–27.

Meunier F, Paesmans M, Autier P. Value ofantifungal prophylaxis with antifungal drugsagainst oropharyngeal candidiasis in cancerpatients. Eur J Cancer 1994;30B:196–9.

Ofman J, Koretz R. Clinical economics review:nutritional support. Aliment Pharmacol Ther 1997;11:453–71.

Patrono C, Roth GJ. Aspirin in ischemiccerebrovascular disease. How strong is the case fora different dosing regimen? Stroke 1996;27:756–60.

Piccinelli M, Pini S, Bellantuono C, Wilkinson G.Efficacy of drug treatment in obsessive–compulsivedisorder: a meta-analytic review. Br J Psychiatry1995;166:424–43.

Rahlfs V, Macciocchi A, Monti T. Brodimoprim inupper respiratory tract infections. Two meta-analyses of randomised, controlled clinical trials inacute sinusitis and otitis media. Clinical DrugInvestigation 1996;11:65–76.

Rains C, Noble S, Faulds D. Sulfasalazine. A reviewof its pharmacological properties and therapeutic

efficacy in the treatment of rheumatoid arthritis.Drugs 1995;50:137–56.

Riedemann P, Bersinic S, Cuddy L, Torrance G,Tugwell P. A study to determine the efficacy andsafety of tenoxicam versus piroxicam, diclofenacand indomethacin in patients with osteoarthritis: ameta-analysis. J Rheumatol 1993;20:2095–103.

Tramonte S, Brand M, Mulrow C, Amato M,O'Keefe M, Ramirez G. The treatment of chronicconstipation in adults. A systematic review. J GenIntern Med 1997;12:15–24.

Vaitkus PT, Berlin JA, Schwartz JS, Barnathan ES.Stroke complicating acute myocardial infarction. A meta-analysis of risk modification byanticoagulation and thrombolytic therapy. ArchIntern Med 1992;152:2020–4.

Voogel A, van der Meulen J, van Montfrans G.Effects of antihypertensive drugs on the circadianblood pressure profile. J Cardiovasc Pharmacol1996;28:463–9.

Wade C, Kramer G, Grady J, Fabian T, Younes R.Efficacy of hypertonic 7.5% saline and 6%dextran-70 in treating trauma: a meta-analysis ofcontrolled clinical studies. Surgery 1997;122:609–16.

Wood M. The comparative efficacy and safety ofteicoplanin and vancomycin. J Antimicrob Chemother1996;37:209–22.

Appendix 4

110

Search strategy 1The process involved an initial ‘One Search’ onDialog to gauge the amount of literature onindirect comparisons and to identify suitabledatabases for future searching (search strategy 1)

Search strategy 1:

S1 randomized controlled trials/ab,ti,deS2 trial?/ab,ti,deS3 s1 or s2S4 (indirect(2w)comparison?)/ab,ti,deS5 (direct(2w)comparison?)/ab,ti,deS6 (indirect(2w)evaluat?)/ab,ti,deS7 (direct(2w)evaluat?)/ab,ti,deS8 (treatment(2w)arm?)/ab,ti,deS9 (compet?(2w)technolog?)/ab,ti,de

S10 (compet?(2w)intervention?)/ab,ti,deS11 s4:s10S12 s3 and s11

Search strategy 1 was run across all the databaseslisted below up to October 1997 and retrievedrecords as follows:

Database Records retrievedMEDLINE 723ERIC 1PsycINFO 29EMBASE 901Dissertation Abstracts 33MathSci 8

Search strategy 2The next step was to run a strategy on MEDLINEvia Ovid (Search strategy 2) to retrieve the full textof each record to help in the process of identifyingsuitable terms for inclusion in the final searchstrategy. This strategy identified 759 records onMEDLINE when run for the period 1966 toOctober 1997.

Search strategy 2:

1 exp RANDOMIZED CONTROLLED TRIALS/2 trial$.tw.3 1 or 2

4 (indirect adj2 comparison$).tw.5 (direct adj2 comparison$).tw.6 (indirect adj2 evaluat$).tw.7 (direct adj2 evaluat$).tw.8 (treatment adj2 arm$).tw.9 (compet$ adj2 technolog$).tw.

10 (compet$ adj2 intervention$).tw.11 4 or 5 or 6 or 7 or 8 or 9 or 1012 3 and 11

Search strategy 3After further consideration of how ‘competinginterventions’ or indirect comparisons weredescribed in individual studies and which MeSHheadings had been used to index records, thestrategy was then further developed (searchstrategy 3).

Search strategy 3 was run for the year 1997 andretrieved 1690 records. When these records wereexamined by the reviewers it was found that manyof them were reports of single RCTs, rather thandiscussion of the methodology of RCTs.

Search strategy 3:

1 RANDOMIZED CONTROLLED TRIALS/2 controlled clinical trials.sh.3 CLINICAL TRIALS/4 clinical trials.tw.5 trial$.tw.6 meta-analysis.sh.7 meta-analysis.tw.8 metaanalys$.tw.9 (meta adj analys$).tw.

10 RESEARCH DESIGN/11 data interpretation,statistical.sh.12 models,statistical.sh.13 (indirect adj2 comparison$).tw.14 (direct adj2 comparison$).tw.15 (indirect adj2 evaluat$).tw.16 (direct adj2 evaluat$).tw.17 (compet$ adj2 technolog$).tw.18 (compet$ adj2 intervention$).tw.19 (treatment adj2 arm$).tw.20 (treatment adj2 group$).tw.21 (randomi$ adj2 group$).tw.22 (randomi$ adj2 comparison$).tw.


111


Appendix 5

Search strategy development

23 (therapeutic adj2 arm).tw.24 (therapeutic adj2 arms).tw.25 (study adj2 arm).tw.26 (study adj2 arms).tw.27 4 limb study.tw.28 four limb study.tw.29 (trial adj2 arm).tw.30 (trial adj2 design).tw.31 (placebo adj2 arm).tw.32 (preventive adj2 arm).tw.33 (preventative adj2 arm).tw.34 or/1-935 or/10-3336 34 and 35

Search strategy 4In an attempt to remove records describing singlestudies from the search results the strategy wasamended. The desired outcome was to retrieverecords in which there was reference to trials andmeta-analysis and one of the possible free textterms used for describing competinginterventions/indirect comparisons (searchstrategy 4).

Search strategy 4:

1 RANDOMIZED CONTROLLED TRIALS/2 controlled clinical trials.sh.3 CLINICAL TRIALS/4 clinical trials.tw.5 trial$.tw.6 meta-analysis.sh.7 meta-analysis.tw.8 metaanalys$.tw.9 (meta adj analys$).tw.

10 RESEARCH DESIGN/11 data interpretation,statistical.sh.12 models,statistical.sh.13 (indirect adj2 comparison$).tw.14 (direct adj2 comparison$).tw.15 (indirect adj2 evaluat$).tw.16 (direct adj2 evaluat$).tw.17 (compet$ adj2 technolog$).tw.18 (compet$ adj2 intervention$).tw.19 (treatment adj2 arm$).tw.20 (treatment adj2 group$).tw.21 (randomi$ adj2 group$).tw.22 (randomi$ adj2 comparison$).tw.23 (therapeutic adj2 arm).tw.24 (therapeutic adj2 arms).tw.25 (study adj2 arm).tw.26 (study adj2 arms).tw.27 4 limb study.tw.28 four limb study.tw.

29 (trial adj2 arm).tw.30 (trial adj2 design).tw.31 (placebo adj2 arm).tw.32 (preventive adj2 arm).tw.33 (preventative adj2 arm).tw.34 or/1-535 or/6-936 or/10-3337 34 and 35 and 36

Search strategy 5Search strategy 4 was run on MEDLINE for theperiod 1995 to December 1998 and retrieved 238records. There was still some uncertainty, however,about the recall of the search strategy, and in anattempt to improve this additional free text termswere included in the third facet of the strategy(search strategy 5). The inclusion of the additionalterms resulted in the retrieval of some additional131 records.

Search strategy 5:

1 RANDOMIZED CONTROLLED TRIALS/2 controlled clinical trials3 CLINICAL TRIALS/4 clinical trials.tw.5 trial$.tw.6 1 or 2 or 3 or 4 or 57 meta-analysis.sh.8 meta-analysis.tw.9 metaanalys$.tw.

10 (meta adj analys$).tw.11 7 or 8 or 9 or 1012 RESEARCH DESIGN/13 data interpretation, statistical.sh.14 models, statistical.sh.15 (indirect adj2 comparison$).tw.16 (direct adj2 comparison$).tw.17 (indirect adj2 evaluat$).tw.18 (direct adj2 evaluat$).tw.19 (compet$ adj2 technolog$).tw.20 (compet$ adj2intervention$).tw.21 (treatment adj2 arm$).tw.22 (treatment adj2group$).tw.23 (randomi$ adj2group$).tw.24 (randomi$ adj2comparison$).tw.25 (therapeutic adj2 arm).tw.26 (therapeutic adj2 arms).tw.27 (study adj2 arm).tw.28 (study adj2 arms).tw.29 4 limb study.tw.30 four limb study31 (trial adj2 arm).tw.32 (trial adj2 design).tw.

Appendix 5

112

33 (placebo adj2 arm).tw.34 (preventive adj2arm).tw.35 (preventative adj2 arm).tw.36 multiple arm study.tw.37 multiple arms study.tw.38 multiple arm studies.tw.39 multiple arms studies.tw.40 multiple arm.tw.41 multiple arms.tw.42 multi arm.tw.43 multi arms.tw.44 (multi adj2 arm).tw.45 (multi adj2 arms).tw.46 (multiple adj2 arm).tw.47 (multiple adj2 arms).tw.48 ((three arm or three arms or 3 arm or 3 arms

or three limb or three limbs or 3 limb or3limbs) adj5 (trial$ or stud$ or random$))

49 ((four arm or four arms or 4 arm or 4 arms orfour limb or four limbs or 4 limb or 4 limbs)adj5 (trial$ or stud$ or random$))

50 (competing adj2 therap$).tw.51 (multi$ adj2 (study or studies)).tw.52 or/12-5153 6 and 11 and 52

Before 1989, records in MEDLINE referring tometa-analysis were indexed using the terms:Outcome and Process Assessment (1977–1979),Follow-Up Studies (1977–1979), Research(1980–1982), Research Design (1980–1988) andStatistics (1980–1988). It was not clear what effectthis would have on running the existing strategyon these earlier years, so various trial strategieswere undertaken. These included (1) omitting themeta-analysis facet from the strategy entirely; (2) replacing the meta-analysis terms with the termsresearch, research design, follow-up studies,statistics, outcome and process assessment using theexplosion facility; and (3) replacing the meta-analysis terms with the terms research, researchdesign, follow-up studies, statistics, outcome andprocess assessment not using the explosion facility.None of these search strategies resulted in anyadditional studies being identified. Strategy 5 wastherefore used as a basis from which to developstrategies to use in other databases withamendments as appropriate regarding thesaurusterms and subject indexing. The MEDLINE searchwas run from 1966 to March 1999. It was updatedto include records published by February 2001.


113


PsycLITThe PsycLIT database was searched viaSilverPlatter (1887 to March 1999) and 287records were retrieved. The search was updated toinclude records published up to February 2001,identifying a further 40 records. The searchstrategy used was:

1 randomized controlled trials2 randomi* control* trial*3 control* clinical trial*4 clinical trial*5 trial*6 exact{EMPIRICAL-STUDY} in PT7 1 or 2 or 3 or 4 or 5 or 68 multiple arm study9 multiple arms study

10 multiple arms studies11 multiple arms studies12 multiple arm13 multiple arms14 multi arm15 multi arms16 multi near2 arm17 multi near2 arms18 multiple near2 arm19 multiple near2 arms20 three arm or three arms or 3 arm or 3 arms or

three limb or three limbs or 3 limb or 3 limbs21 four arm or four arms or 4 arm or 4 arms or

four limb or four limbs or 4 limb or 4 limbs22 multi* near3 (study or studies)23 8 or 9 or 10 or 11or 12 or 13 or 14 or 15 or

16 or 17 or 18 or 19 or 20 or 21 or 2224 7 or 2325 explode META-ANALYSIS26 meta-analy*27 metaanaly*28 meta near analy*29 25 or 26 or 27 or 2830 24 and 2931 explode EXPERIMENTAL-DESIGN32 explode STATISTICAL-ANALYSIS33 indirect near2 comparison*34 direct near2 comparison*35 indirect near2 evaluat*36 direct near2 evaluat*37 compet* near2 technolog*38 compet* near2 intervention*

39 treatment near2 arm*40 treatment near2 group*41 randomi* near2 group*42 randomi* near2 comparison*43 therapeutic near2 arm*44 study near2 arm*45 trial near2 arm*46 trial near2 design*47 placebo near2 arm*48 preventive near2 arm*49 preventative near2 arm*50 competing near2 therap*51 indirect near evaluat*52 31 or 32 or 33 or 34 or 35 or 36 or 37 or 38

or 39 or 40 or 41 or 42 or 43 or 44 or 45 or46 or 47 or 48 or 49 or 50 or 51

53 30 and 52

ERICThe ERIC database was searched using the Ovidinterface via BIDS (1966 to February 1999) and 72records were retrieved. The search strategy usedwas:

1 randomized controlled trial.tw.2 randomi#ed control? trial?.tw.3 control? clinical trial?.tw.4 clinical trial?.tw.5 trial?.tw.6 exp MATCHED GROUPS/7 exp EXPERIMENTAL GROUPS/8 1 or 2 or 4 or 5 or 6 or 79 multiple arm study.tw.

10 multiple arms study.tw.11 multiple arm studies.tw.12 multiple arms studies.tw.13 multiple arm.tw.14 multiple arms.tw.15 multi arm.tw.16 multi arms.tw.17 (multi adj2 arm).tw.18 (multi adj2 arms).tw.19 (multiple adj2 arm).tw.20 (multiple adj2 arms).tw.21 (three arm or three rms or 3 arm or 3 arms or

three limb or three limbs or 3 limb).mp. or 3 limbs.tw. [mp=abstract, title, heading word,identifiers]


115


Appendix 6

Additional electronic search strategies

22 (three arm or three arms or 3 arm or 3 armsor three limb or three limbs or 3 limb).mp. or3 limbs.tw. [mp=abstract, title, heading word,identifiers]

23 (four arm or four arms or 4 arm or 4 arms orfour limb or four limbs or 4 limb).mp. or 4limbs.tw. [mp=abstract, title, heading word,identifiers]

24 (multi? adj3 (study or studies)).tw.25 8 or 23 or 2426 meta-analysis.tw.27 (meta adj analysis).tw.28 metaanalysis.tw.29 meta-analytic.tw. 30 (meta adj analytic).tw.31 metaanalytic.tw.32 exp META ANALYSIS/33 exp LITERATURE REVIEWS 34 26 or 27 or 28 or 29 or 30 or 31 or 32 or 3335 25 and 34

EMBASEThe EMBASE database was searched using theSilverplatter interface (1980 to April 1999) and282 records were retrieved. The search strategyused was:

1 explode RANDOMIZED-CONTROLLED-TRIAL/ all subheadings

2 explode CONTROLLED-STUDY/ allsubheadings

3 explode CLINICAL-TRIAL/ all subheadings 4 trial*5 clinical trials 6 1 or 2 or 3 or 4 or 57 multiple arm study 8 multiple arms study 9 multiple arm studies

10 multiple arms studies 11 multiple arm 12 multiple arms 13 multi arm 14 multi arms 15 multi near2 arm 16 multi near2 arms 17 multiple near2 arm 18 multiple near2 arms 19 three arm or three arms or 3 arm or 3 arms or

three limb or three limbs or 3 limb or 3 limbs 20 #19 near5 (trial* or stud* or random*) 21 four arm or four arms or 4 arm or 4 arms or

four limb or four limbs or 4 limb or 4 limbs 22 #21 near5 (trial* or stud* or random*) 23 multi* near3 (study or studies) 24 #7 or #8 or #9 or #10 or #11 or #12 or #13

or #14 or #15 or #16 or #17 or #18 or #20or #22 or #23

25 #6 or #24 26 explode META-ANALYSIS/ all subheadings 27 meta-analysis 28 metaanalys* 29 meta near2 analys* 30 #26 or #27 or #28 or #29 31 #25 and #30 32 indirect near2 comparison* 33 direct near2 comparison* 34 indirect near2 evaluat* 35 direct near2 evaluat* 36 compet* near2 technolog* 37 compet* near2 intervention* 38 treatment near2 arm* 39 treatment near2 group* 40 randomi* near2 group* 41 randomi* near2 comparison* 42 therapeutic near2 arm 43 therapeutic near2 arms 44 study near2 arm 45 study near2 arms 46 trial near2 arm 47 trial near2 design 48 placebo near2 arm 49 preventive near2 arm 50 preventative near2 arm 51 competing near2 therap* 52 #32 or #33 or #34 or #35 or #36 or #37 or

#38 or #39 or #40 or #41 or #42 or #43 or#44 or #45 or #46 or #47 or #48 or #49 or#50 or #51

53 #31 and #52

MathSci The MathSci database was searched using Dialog(up to September 1999) and 1817 records were retrieved. The search strategyused was:

1 s research(W)design2 s statist?(W)models3 s indirect(2W)comparison?4 s direct(2W)comparison?5 s indirect(2W)evaluat?6 s direct(2W)evaluat?7 s compet?(2W)technolog?8 s compet?(2W)intervention?9 s treatment(2W)arm?

10 s treatment(2W)group?11 s randomi?(2W)group?12 s randomi?(2W)comparison?13 s therapeutic(2W)arm14 s therapeutic(2W)arms

Appendix 6

116

15 s study(2W)arm16 s study(2W)arms17 s 4(W)limb(W)study18 s four(W)limb(W)study19 s trial(2W)arm20 s trial(2W)design21 s placebo(2W)arm22 s preventive(2W)arm23 s preventative(2W)arm24 s multiple(W)arm(W)study25 s multiple(W)arms(W)study26 s multiple(W)arm(W)studies27 s multiple(W)arms(W)studies28 s multiple(W)arm29 s multiple(W)arms30 s multi(W)arm

31 s multi(W)arms32 s multi(2W)arm33 s multi(2W)arms34 s multiple(2W)arm35 s multiple(2W)arms36 s ((three(W)arm) or (three(W)arms) or

(3(W)arm) or (3(W)arms) or (three(W)limb) or(three(W)limbs) or (3(W) limb) or (3(W)limbs))(5W) (trial? or stud? or random?)

37 s ((four(W)arm) or (four(W)arms) or (4(W)arm)or (4(W)arms) or (four(W)limb) or(four(W)limbs) or (4(W)limb) or (4(W)limbs))(5W) (trial? or stud? or random?)

38 s competing(2W)therap?39 s multitreatment(2W)(study or studies)40 s s1:s39


117


Data setSeveral of the possible analyses are illustratedusing a set of trials of antithrombotic therapy toprevent strokes in patients with atrial fibrillation150

(Table 23). The emphasis is on methods that canbe applied using widely available software. Asthese trials are used for illustration only theimpact of certain criticisms of the systematicreview has been disregarded.151

From this set of data subsets of trials are taken toillustrate:

� an indirect comparison of BvC using A as acommon control (studies 3, 4 and 5 are two-armed comparisons of AvB, studies; 7, 8, 9 and10 are two-armed comparisons of AvC)

� a combination of the above indirect comparisonwith direct two-armed trials to estimate BvC(trial 11 is a two-armed trial of BvC, which iscombined with the indirect comparison usingthe seven trials listed above)

� a combination of the above indirect and directcomparisons with multiarmed trials of A, B andC (trials 1, 2 and 6 combined with the eighttrials listed above)

� a combination of all 15 trials and all treatments.

These subsets of trials are created for the purposeof illustrating sequentially more complexalternative models, and the results should not beregarded as definitive analyses of the above dataset.

AnalysesAnalyses were undertaken using Stata version 8 foradjusted indirect comparisons and logisticregression, and using the PROC NLMIXEDprocedure in SAS version 9.1 for the mixed models.The SAS procedure is chosen for presentation inpreference over the Stata gllamm command(which can also be used for fitting mixed models),for reasons of computational speed.


119


Appendix 7

Illustrative analyses

TABLE 23 Event rates for stroke in trials of antithrombotic therapy for patients with atrial fibrillation (from Hart et al.150)

Trial Placebo Adjusted- Aspirin Low- or fixed- Low- or fixed-dose warfarin dose warfarin dose warfarin

+ aspirinA B C D E

1 AFASAK 19/336 9/335 16/3362 SPAF 19/211 8/210 25/5523 BAATAF 13/208 3/2124 CAFA 9/191 6/1875 SPINAF 23/290 7/2816 EAFT ia 50/214 20/225 49/230a

7 EAFT iia 40/164 39/174a

8 ESPS II 23/107 17/1049 LASAF 6/182 5/194b

10 UK-TIA 8/30 8/34b

11 SPAF II 19/358 21/35712 AFASAK II 11/170 9/169 14/167 11/17113 PATAF 3/131 4/141 4/12214 SPAF III 14/523 48/52115 MWNAF 1/153 5/150

a EAFT152 included two clinical subgroups randomised to different treatment options according to eligibility foranticoagulation. The results for aspirin were combined in the publication (88/404); here, the data have been split bymaking the event rates equal.

b Combination of two groups randomised to different doses of aspirin.

Format of data setsFour different formats are required for alternativeanalyses:

(a) One row per trial (see Table 24)Each row states for treatment and control groupsthe numbers of events (event_t, event_c) and thenumber of participants (sample_t, sample_c).

This data structure is only suitable for two-armedtrials. For adjusted indirect comparisons (AICs)meta-analyses are undertaken on different subsetsof the data for each component meta-analysisindicated by an additional variable [here a variablecompb is used, indicating whether A wascompared with B (value 1) or C (value 0)]. Here,the ‘A’ arm of the trials acts as the commoncomparator and is called ‘control’.

(b) Two (or more) rows per trial, oneper trial arm (see Table 25)

Each row states the number of events (event) andparticipants (sample) and study arm (arm). Thisdata structure is the standard format for fittinglogistic and mixed models.

In Stata a data set of format (a) can be changedinto format (b) using the following commands:

rename event_c event0

rename event_t event1

rename sample_c sample0

rename sample_t sample1

reshape long event sample, i(trial)

j(treat)

generate arm=”C”

replace arm=”A” if treat==1 & compb==0

replace arm=”B” if treat==1 & compb==1

(c) Four (or more) rows per trial, tworows per trial arm (see Table 26)

Each row states the number in each outcomecategory (n) and indicator variables for whether

they suffered a stroke or not (stroke) and studyarm (arm). This data structure is only required forthe models fitted in the Stata mixed modelscommand gllamm.

In Stata a data set of format (b) can be changedinto format (c) using the following commands:

rename event n1

gene n0=sample-n1

reshape long event sample, i(trial arm)

j(stroke)

Appendix 7

120

TABLE 24 Example data set for indirect comparisons analysis

ID Trial event_c sample_c event_t sample_t compb

1 BAATAF 3 212 13 208 12 CAFA 6 187 9 191 13 SPINAF 7 281 23 290 14 EAFT ii 39 174 40 164 05 ESPS II 17 104 23 107 06 LASAF 5 194 6 182 07 UK-TIA 8 34 8 30 0

TABLE 25 Example data set for full analysis

ID Trial Event Sample Arm

1 AFASAK 19 336 A2 AFASAK 9 355 B3 AFASAK 16 336 C4 SPAK 19 211 A5 SPAK 8 210 B

… … … … …35 MWNAF 1 153 A36 MWNAF 5 150 D

TABLE 26 Example data set for full analysis in gllamm

ID Trial n Stroke Arm

1 AFASAK 19 1 A2 AFASAK 317 0 A3 AFASAK 9 1 B4 AFASAK 346 0 B5 AFASAK 16 1 C6 AFASAK 320 0 C7 SPAK 19 1 A8 SPAK 192 0 A9 SPAK 8 1 B

10 SPAK 202 0 B… … … …69 MWNAF 1 1 A70 MWNAF 152 0 A71 MWNAF 5 1 D72 MWNAF 145 0 D

(d) One row per participant, n rows pertrial (see Table 27)

Each row states whether the participant suffered astroke or not (stroke) and their study arm (arm).This data structure is only required for the modelsfitted in the Stata xtlogit.

In Stata a data set of format (c) can be changedinto format (d) using the following commands:

expand n

drop n

Indirect comparisonsNaive methodThe naive analysis is based on summing the eventsand participants in the B and C arms of the sevencomponent trials, and computing an odds ratioand confidence interval as if the data had arisen ina single study.

Across the three B trial arms 16 out of 680participants experienced a stroke. Across the fourC trial arms 69 out of 506 participantsexperienced a stroke. This comparison yields anodds ratio of 0.15 (95% CI 0.09 to 0.27), and ishighly statistically significant (z = –6.61,p < 0.00001).

Adjusted indirect comparisons andmeta-regressionAdjusted indirect comparisons are undertaken byperforming separate meta-analyses on the datasets of trials of AvB and AvC. One approach is touse the Stata meta-analysis command metan103 foreach meta-analysis (lines 5 and 9), and store thevalues of the log odds ratio (lines 6 and 10) andstandard error (lines 7 and 11) from each separatemeta-analysis. The odds ratio for indirectcomparison is estimated as the exponential of the

difference in log odds ratios from the two meta-analyses (lines 13 and 14). The standard error ofthe indirect comparison is estimated from thesquare root of the sum of the squared standarderrors (line 15).

The code below assumes that the data are informat (a):

* STATA CODE for indirect comparison by

adjusted indirect comparison

1 * compute values of the four cells for

each trial

2 gene noevent_t=sample_t-event_t

3 gene noevent_c=sample_c-event_c

4 * meta-analysis of trials of BvA (combp

equal to 1)

5 metan event_t noevent_t event_c

noevent_c if compb==1, or fixedi

nograph

6 local logor1=log($S_1)

7 local se1=$S_2

8 * meta-analysis of trials of CvA (combp

equal to 0)


noevent_c if compb==0, or fixedi

nograph

10 local logor2=log($S_1)

11 local se2=$S_2

12 * computation of log OR, OR and se for

indirect comparison

13 local logor_aic=`logor1'-`logor2'

14 local or_aic=exp(`logor_aic')

15 local se_aic=sqrt(`se1'^2+`se2'^2)

16 * computation of confidence intervals,

z-value and P-value

17 local ll_aic=exp(`logor_aic'-

(1.96*`se_aic'))

18 local

ul_aic=exp(`logor_aic'+(1.96*`se_aic')

)

19 local z_aic=`logor_aic'/`se_aic'

20 if `z_aic'>0 local p_aic=2*(1-

norm(`z_aic'))

21 if `z_aic'<=0 local

p_aic=2*norm(`z_aic')

The above analysis produces inverse varianceestimates. Alternative meta-analysis models areobtained by changing the inverse variance fixedeffect option (fixedi) on the metan command lines(lines 5 and 9) to indicate Mantel–Haenszel fixedeffects (fixed) or DerSimonian and Laird random


121


TABLE 27 Example data set for full analysis in xtlogit

ID Trial Stroke Arm

1 AFASAK 1 A2 AFASAK 1 A3 AFASAK 1 A

… … …19 AFASAK 1 A20 AFASAK 0 A21 AFASAK 0 A

… … …8139 MWNAF 0 D8140 MWNAF 0 D

effects (randomi) models. Risk ratio models can beobtained by changing the summary statistic fromodds ratio (or) to the risk ratio (rr). Riskdifference estimates are produced using the riskdifference (rd) option. If risk differences arepooled, use of a logarithmic scale is not required,requiring alterations to lines 6, 10, 14, 17 and 18of the above code. Continuous outcomes can alsobe pooled using the metan command.

Meta-regression can also be used to estimate thedifference between the groups. To undertakemeta-regression, estimates of the log odds ratioand standard error are computed for each study.Using the values stored by metan provides a short-cut for doing this (lines 5–7). A weightedregression model is then fitted with the log oddsratio as the outcome and the comparison (compb)as the predictor. The model weights each studyaccording to the inverse variance (line 8). A simpleapproach to fit the model involves weighted linearregression (line 10). A more appropriate method isto use random effects meta-regression (line 12)that allows for unexplained variability betweengroups, as provided by the metareg command.78

* STATA CODE for indirect comparison

by meta-regression

1 * compute values of the four cells for

each trial

2 gene noevent_t=sample_t-event_t

3 gene noevent_c=sample_c-event_c

4 * meta-analysis of all trials to

compute logOR and se


noevent_c, or fixedi nograph notable

6 local logor=log($S_1)

7 local se=$S_2

8 local wt=1/`se’^2

9 * weighted linear regression

10 regress logor compb [aw=wt]

11 * random effects meta-regression

12 metareg logor compb, wsse(se)

The three AIC models (inverse variance,Mantel–Haeszel, and DerSimonian and Laird) andtwo regression models (fixed and random) givesimilar estimates of the indirect comparison(Table 28), but all are very different from theflawed naive method.

The AIC DerSimonian and Laird and randomeffects meta-regression use different approaches to estimate the unexplained between studyvariation, �2. The DerSimonian and Laird method estimates separate �2 values forcomparisons of AvC (�2 = 0) and AvB (�2 = 0.023).The meta-regression model estimates a single �2 todescribe the residual heterogeneity among alltrials having accounted for the difference inestimates between trials comparing AvB and trialscomparing AvC (�2 = 0). The meta-regressionapproach is likely to be more efficient and preciseas it involves one fewer parameter and estimates �2

using data from all trials. The alternativeDerSimonian and Laird approach may be moreappropriate when there are reasons to assumedifferent �2 values in the two component analyses.

Generalised linear modelsA logistic regression model can be fitted to data informat (b). The model includes two zero–oneindicator variables, armA and armB, to indicatestudy arm (lines 1–5). Arm C is designated thebaseline category with which comparisons aremade. An estimate of the indirect comparison isobtained from the parameter estimate for armB(comparison of B with baseline C). To obtain ananalysis that is stratified by trial, indicatorvariables for trial are included to allow each trialto have a different control group risk (lines 7–8).In Stata the command blogit is used to fit alogistic regression model indicating the number ofoutcomes (event), the number of participants(people) and the dependent variables (armC,armA and trial):

* STATA CODE for indirect comparison by

logistic regression

Appendix 7

122

TABLE 28

Approach Method OR 95% CI z/t-Value p-Value

AIC Inverse variance 0.43 (0.22 to 0.87) z = –2.36 0.0184AIC Mantel–Haenszel 0.42 (0.21 to 0.83) z = –2.48 0.0131AIC DerSimonian and Laird 0.43 (0.21 to 0.89) z = –2.28 0.0225Meta-regression Weighted linear regression 0.43 (0.23 to 0.82) t = –3.37 0.0198Meta-regression Random effects meta-regression 0.43 (0.22 to 0.87) z = –2.36 0.0184

1 * generate indicator variables for armA

and armB

2 gene armA=0

3 gene armB =0

4 replace armA=1 if arm=="A"

5 replace armB=1 if arm=="B"

6 * fit a logistic regression model

7 * xi: automatically adds indicator

variables for trial

8 xi:blogit event people armC armA

i.trial, or

Alternatively, the Stata command xtlogit can beused for data in format (d). Again indicatorvariables are generated for study arms, which takevalues of 0 or –1 (lines 1–5). The i(trial) optioncombined with the fixed effect option fe (line 7)performs an analysis stratified by trial thatproduces the same results as the above logisticregression model.

The xtlogit command includes an option toperform random effects analysis, using the reoption (line 9). This analysis replaces the trialindicator variables with a distribution of effects(making an assumption that the logit controlgroup risks are a random sample from a normaldistribution). It does not, as is usually desired in ameta-analytical random effects analysis, place therandom effect on the treatment contrasts (seeChapter 4, section ‘Classical methods usingaggregate data’, p. 19).

* STATA CODE for indirect comparisonusing xtlogit1 * generate indicator variables forarmA and armB

2 gene armA=03 gene armB =04 replace armA=-1 if arm=="A"5 replace armB=-1 if arm=="B"

6 * fit a fixed effect logisticregression model

7 xtlogit stroke armC armA, i(trial)fe or

8 * fit a random effects logisticregression model

9 xtlogit stroke armC armA, i(trial)re or

Two standard software packages can be used forestimating a random effect for the treatmentcontrast for binomial data: SAS PROC NLMIXEDand the gllamm command in Stata. The use of the

SAS option is demonstrated here; this usesadaptive quadrature to identify maximumlikelihood solutions, a more advanced and fastermethod than used by gllamm. WinBugs software,81

using a fully Bayesian model specification, providesthe greatest flexibility for fitting these models.

Two alternative models can be fitted using data informat (b) with indicator variables armA and armBindicating the treatment. In the first, trial effectsare fitted as random (using the term trialr in lines4 and 7) and a random effect for treatmentcontrasts is included assumed constant acrosstreatment comparisons (using the term het in lines4 and 7). The procedure requires appropriatestarting values to be stated (line 3), which may beobtained from a fixed effect analysis.

* SAS code for indirect comparison using

PROC NLMIXED

1 * random effects for trial and

treatment contrasts

2 proc nlmixed data=indirect;

3 parms base=-2.3 a=0.2 b=-1

s2trialr=0.7 s2het=0.1;

4 logitp= (base+trialr)+ armA*(a+het) +

armB*(b+het);

5 p = exp(logitp)/(1+exp(logitp));

6 model stroke ~ binomial(n,p);

7 random trialr het ~

normal([0,0],[s2trialr,0,s2het])

subject=trial;

8 run;

This analysis estimates the trial effect variance tobe 0.68, and the random effect for treatmentcontrasts to be 0.09. If desired, separate randomeffects can be estimated for the AvC and BvCtreatment contrasts by specifying two separateheterogeneity parameters, although there arelikely to be estimation problems unless there aremany trials. This model requires the assumptionthat the logit baseline event rates are randomlysampled from a normal distribution. Thatassumption can be avoided by estimating a fixedeffect for each trial while including a randomeffect for treatment contrast. This model requiresestimation of many more parameters (additionalparameters t3–t10 are estimated for each trial inlines 7 and 8) and is less likely to produce a stablesolution. Again separate heterogeneity parameterscould be included for each treatment contrast.

* SAS code for indirect comparison

using PROC NLMIXED

1 * fixed effect for trial, random

effects for treatment contrasts


123


2 proc nlmixed data=indirect;

3 parms a=0.2 b=-0.9

4 t3=-3 t4=-3 t5=-2.7 t7=-1.3

t8=-1.6 t9=-3.6 t10=-1.2

5 s2het=1;

6 logitp=a*(armA+het) + b*(armB+het)

+

7 t3*trial3 + t4*trial4 +

t5*trial5 +

8 t7*trial7 + t8*trial8 + t9*trial9

+ t10*trial10;

9 p = exp(logitp)/(1+exp(logitp));

10 model stroke ~ binomial(n,p);

11 random het ~ normal([0],[s2het])

subject=trial;

12 run;

In the application to the example data set, theestimate of among-study heterogeneity is close tozero, but the estimates of the treatment contrastsare sensitive to the choice of starting value for theheterogeneity statistic (Table 29).

Combining indirect and directcomparisonsTrial 11 (SPAF II) is a two-armed trial of BvC. Theestimated treatment effect in this trial is OR=0.90(95% CI 0.45 to 1.79) (p=0.74). This section looksat analyses that combine this trial with the sevenindirect trials.

Combining with adjusted indirectcomparisonsA weighted combination of the results from theAIC and the direct comparison is computed as aninverse variance weighted average.

For the direct trial:

lnOR = ln [(19/339) / (21/336)] = ln(0.90) = –0.1090Var(lnOR) = 1/19 + 1/339 + 1/21 + 1/336 =0.1062

For the indirect comparison (from the inversevariance solution)

lnOR = –0.8358

Var(lnOR) = 0.1257

Inverse variance weights for the indirect and directcomparisons are 7.96 and 9.42, respectively. Theweighted average and variance of the combinationare thus:

lnOR = [(9.42 � –0.1090) + (7.96 � –0.8358)] / (7.96

+ 9.42) = –0.4456

Var(lnOR) = 1 / (7.96 + 9.42) = 0.0576

giving an overall estimate of OR = 0.64 (95% CI0.40 to 1.02) (p = 0.06).

It is also possible to test whether there is asignificant difference in the finding of the AIC andthe direct comparison, by dividing the difference inthe logOR (–0.1090 – –0.8358 = 0.7268) by thestandard error of the difference [√(0.1062+0.1257)= 0.4816], and comparing the resulting number(0.7268/0.4816 = 1.51) with a standard normaldistribution (z) to obtain a p-value (p = 0.13).

It is possible to undertake the analyses combiningany of the estimates using AICs or meta-regressionmodels with the results of the direct trial. If therewere more than one direct two-armed trial, resultsof a meta-analysis of the direct trials could becombined with the AIC.

Appendix 7

124

TABLE 29

Method Random effects OR 95% CI z/t-Value p-Value

Logistic regression None 0.42 (0.21 to 0.83) z = –2.49 0.0129(blogit, xtlogit fe)

Logistic regression Trials 0.41 (0.22 to 0.78) z = –2.71 0.007(xtlogit re)

Mixed model Trials 0.38 (0.16 to 0.94)a t = –2.76 0.03(NLMIXED)

Mixed model Trials and treatments 0.38 (0.16 to 0.94)a t = –2.76 0.04(NLMIXED)

Mixed model Treatments Unstable estimates(NLMIXED) (variance estimate is very small)

a Treatment contrast calculated using t statistic.

Generalised linear modelsThe models described above can all incorporatedata from the additional direct trial withoutchanging the code other than inclusion of anadditional indicator variable for the extra trial.The results for these models are given in Table 30.

Including three-armed AvBvC trialsTrials with more than two arms cannot be used inAICs without discarding some treatment arms.However, they can be included naturally in basicgeneralised linear models, again withoutadjustment to the code beyond addition ofindicator variables for the additional trials. There are challenges in appropriately modellingrandom effects for multiarmed trials, as outlinedby Higgins and Whitehead.53 Appropriatemodelling of random effects involves realising a different random effect for each comparison.

SAS PROC NLMIXED assumes that allcomparisons from the same trial involve the samerealisation of the treatment comparison randomeffect, which is not ideal. Correct modelling canbe undertaken using a fully Bayesian model inWinBUGS.81

Analysis of all trials and alltreatmentsAICs can be combined to include comparisonswith alternative control groups only if all trials aretwo-armed trials. Here this is not the case, andincorporation of data from trials 12–15 can onlybe done using generalised linear models. In thesesituations additional indicator variables need to beadded to the model for treatments D and E, aswell as for the additional trials. The same issuesarise in the correct modelling of random effects asdiscussed in the previous section.


125


TABLE 30


AIC Inverse variance 0.64 (0.40 to 1.02) z = –1.86 0.0632AIC Mantel–Haenszel 0.63 (0.40 to 1.01) z = –1.94 0.0530AIC DerSimonian and Laird 0.65 (0.40 to 1.04) z = –1.78 0.0747Meta-regression Weighted linear regression 0.56 (0.38 to 0.83) t = –2.91 0.0036Meta-regression Random effects meta-regression 0.64 (0.40 to 1.02) z = –1.86 0.0632Logistic regression Fixed effect 0.62 (0.39 to 0.99) z = –2.00 0.0460Logistic regression Random effect for trials 0.59 (0.37 to 0.93) z = –2.28 0.0230Mixed model Random effect for trials 0.59 (0.34 to 1.03) t = –2.24 0.0604Mixed model Random effect for trials and treatment 0.59 (0.33 to 1.05) t = –2.24 0.0667Mixed model Random effect for treatment Unstable estimates

(variance estimate is very small)

TABLE 31


Logistic regression Random effect for trials 0.57 (0.43 to 0.75) z = –3.99 0.0001Mixed model Random effect for trials 0.53 (0.38 to 0.75) t = –4.09 0.0022Mixed model Random effect for trials and treatment 0.53 (0.37 to 0.75) t = –4.09 0.0027Mixed model Random effect for treatment Unstable estimates


TABLE 32


Logistic regression Random effect for trials 0.56 (0.43 to 0.73) z = –4.37 <0.0001Mixed model Random effect for trials 0.56 (0.41 to 0.75) t = –4.15 0.001Mixed model Random effect for trials and treatment 0.55 (0.41 to 0.75) t = –4.15 0.001Mixed model Random effect for treatment Unstable estimates



127


Appendix 8

Identified meta-analyses providing sufficient data for both direct and indirect comparisons

Appendix 8

128 TA

BLE

33

Iden

tifie

d m

eta-

anal

yses

with

suf

ficie

nt d

ata

for b

oth

dire

ct a

nd in

dire

ct c

ompa

rison

s

Met

a-an

alys

isPa

tien

ts (

outc

ome)

Inte

rven

tion

s co

mpa

red:

no.

of t

rial

s (p

atie

nts)

Rel

ativ

e ef

ficac

y (9

5% C

I)

cont

inue

d

1 ATC

-I17

Patie

nts

with

an

incr

ease

d ris

k of

occ

lusiv

eva

scul

ar d

iseas

e (e

.g. p

rior

or a

cute

CH

D,

stro

ke, p

erip

hera

l vas

cula

r di

seas

e)

(Vas

cula

r ev

ents

)

Hig

h-do

se a

spiri

n vs

med

ium

-dos

e as

pirin

:3

(121

2/12

13)

Hig

h-do

se a

spiri

n vs

con

trol

: 30

(13,

667/

11,5

65)

Med

ium

-dos

e as

pirin

vs

cont

rol:

19 (2

5,37

6/25

,417

)

RR Dire

ct e

stim

ate:

0.

96 (0

.81

to 1

.15)

Adj

uste

d in

dire

ct:

1.08

(0.9

4 to

1.2

4)N

aive

indi

rect

: 1.

64 (1

.54

to 1

.74)

2 ATC

-I17

Sam

e as

abo

veA

spiri

n+di

pyrid

amol

e vs

asp

irin:

16

(282

9/28

40)

Asp

irin+

dipy

ridam

ole

vs c

ontr

ol:

23 (4

757/

4694

)A

spiri

n vs

con

trol

: 42

(40,

013/

37,5

38)

RR Dire

ct e

stim

ate:

1.

01 (0

.87

to 1

.16)

Adj

uste

d in

dire

ct:

0.91

(0.8

0 to

1.0

5)N

aive

indi

rect

: 1.

12 (1

.03

to 1

.22)

3 ATC

-I17

Sam

e as

abo

veSu

lfinp

yraz

one

vs a

spiri

n:4

(507

/656

)Su

lfinp

yraz

one

vs c

ontr

ol:

17 (2

108/

2135

) A

spiri

n vs

con

trol

:49

(41,

656/

38,7

99)

RR Dire

ct e

stim

ate:

1.

17 (0

.88

to 1

.54)

Adj

uste

d in

dire

ct:

1.02

(0.8

7 to

1.2

0)N

aive

indi

rect

: 1.

32 (1

.17

to 1

.47)

4 ATC

-I17

Sam

e as

abo

veT

iclo

pidi

ne v

s as

pirin

: 3

(173

0/17

41)

Tic

lopi

dine

vs

cont

rol:

27 (2

936/

2955

) A

spiri

n vs

con

trol

: 52

(42,

248/

39,2

30)

RR Dire

ct e

stim

ate:

0.

71 (0

.38

to 1

.34)

Adj

uste

d in

dire

ct:

0.90

(0.7

8 to

1.0

4)N

aive

indi

rect

: 0.

91 (0

.81

to 1

.03)

5 ATC

-II16

Patie

nts

with

an

incr

ease

d ris

k of

vas

cula

roc

clus

ion

(e.g

. cor

onar

y or

leg

arte

ryby

pass

gra

fting

or

angi

opla

sty)

(Vas

cula

r oc

clus

ion)

Hig

h-do

se v

s m

ediu

m-d

ose

aspi

rin:

1 (1

55/1

54)

Hig

h-do

se a

spiri

n vs

con

trol

: 8

(744

/753

)M

ediu

m-d

ose

aspi

rin v

s co

ntro

l: 4

(489

/496

)

RR Dire

ct e

stim

ate:

1.

15 (0

.76

to 1

.74)

Adj

uste

d in

dire

ct:

0.93

(0.5

8 to

1.4

8)N

aive

indi

rect

: 0.

59 (0

.46

to 0

.77)

6 ATC

-II16

Sam

e as

abo

veA

spiri

n+di

pyrid

amol

e vs

asp

irin:

10 (1

264/

1267

)A

spiri

n+di

pyrid

amol

e vs

con

trol

:14

(137

1/13

03)

Asp

irin

vs c

ontr

ol:

7 (5

97/6

10)

RR Dire

ct e

stim

ate:

1.

03 (0

.84

to 1

.27)

Adj

uste

d in

dire

ct:

1.26

(0.8

5 to

1.8

6)N

aive

indi

rect

: 1.

53 (1

.21

to 1

.92)

7 ATC

-II16

Sam

e as

abo

veSu

lfinp

yraz

one

vs a

spiri

n:

2 (1

67/3

26)

Sulfi

npyr

azon

e vs

con

trol

: 5

(276

/285

) A

spiri

n vs

con

trol

: 12

(123

3/12

49)

RR

Dire

ct e

stim

ate:

1.

01 (0

.71

to 1

.45)

Adj

uste

d in

dire

ct:

1.15

(0.7

3 to

1.8

0)N

aive

indi

rect

: 0.

81 (0

.58

to 1

.13)


129


TA

BLE

33

Iden

tifie

d m

eta-

anal

yses

with

suf

ficie

nt d

ata

for b

oth

dire

ct a

nd in

dire

ct c

ompa

rison

s (c

ont’d

)

Met

a-an

alys

isPa

tien

ts (

outc

ome)

Inte

rven

tion

s co

mpa

red:

no.

of t

rial

s (p

atie

nts)

Rel

ativ

e ef

ficac

y (9

5% C

I)

cont

inue

d

8 ATC

-II16

Sam

e as

abo

veT

iclo

pidi

ne v

s as

pirin

: 2

(41/

41)

Tic

lopi

dine

vs

cont

rol:

12 (5

46/5

42)

Asp

irin

vs c

ontr

ol:

13 (1

542/

1402

)

RR Dire

ct e

stim

ate:

1.

16 (0

.43

to 3

.16)

Adj

uste

d in

dire

ct:

1.12

(0.8

0 to

1.5

6)N

aive

Indi

rect

: 0.

75 (0

.59

to 0

.96)

9 ATC

-II16

Sam

e as

abo

veA

spiri

n+di

pyrid

amol

e vs

sul

finpy

razo

ne:

1(16

2/14

8)A

spiri

n+di

pyrid

amol

e vs

con

trol

: 19

(200

4/19

42)

Sulfi

npyr

azon

e vs

con

trol

:5

(276

/285

)

RR

Dire

ct e

stim

ate:

0.

94 (0

.62

to 1

.43)

Adj

uste

d in

dire

ct:

1.06

(0.6

9 to

1.6

5)N

aive

indi

rect

: 1.

54 (1

.13

to 2

.12)

10 ATC

-III18

Surg

ical

and

hig

h-ris

k m

edic

al p

atie

nts

(DVT

)

Asp

irin+

dipy

ridam

ole

vs a

spiri

n:

9 (2

63/2

18)

Asp

irin+

dipy

ridam

ole

vs c

ontr

ol:

11 (3

94/4

22)

Asp

irin

vs c

ontr

ol:

9 (6

49/5

97)

RR Dire

ct e

stim

ate:

0.

67 (0

.51

to 0

.89)

Adj

uste

d in

dire

ct:

0.77

(0.4

4 to

1.3

3)N

aive

indi

rect

: 0.

94 (0

.73

to 1

.20)

11 Buch

er e

t al

.14

HIV

-infe

cted

pat

ient

s

(Pne

umoc

ystis

car

inii

pneu

mon

ia)

TM

P+SM

X v

s D

+P

or D

: 8

(803

/815

)T

MP+

SMX

vs

AP:

9

(681

/613

)D

+P

or D

vs

AP:

5

(732

/718

)

RR Dire

ct e

stim

ate:

0.

45 (0

.22

to 0

.91)

Adj

uste

d in

dire

ct:

0.43

(0.2

5 to

0.7

5)N

aive

indi

rect

: 0.

55 (0

.35

to 0

.87)

12 Che

ng e

t al

.137

Wom

en a

tten

ding

ser

vice

s fo

r em

erge

ncy

cont

race

ptio

n

(No.

of p

regn

anci

es)

Levo

norg

estr

el v

s m

ifepr

iston

e:

1 (6

43/6

33)

Levo

norg

estr

el v

s yu

zpe:

2

(138

6/14

21)

Mife

prist

one

vs y

uzpe

: 2

(597

/589

)

RR Dire

ct e

stim

ate:

2.

19 (1

.00

to 4

.77)

Adj

uste

d in

dire

ct:

5.40

(0.5

3 to

54.

82)

Nai

ve in

dire

ct:

19.8

3 (1

.21

to 3

26.0

3)

13 Chi

ba e

t al

.40

Patie

nts

with

GO

RD

(Hea

ling

rate

)

H2R

A v

s PP

I: 13

(731

/884

)H

2RA

vs

plac

ebo:

11

(181

7/95

9)PP

I vs

plac

ebo:

2

(334

/75)

RR Dire

ct e

stim

ate:

0.

56 (0

.48

to 0

.66)

Adj

uste

d in

dire

ct:

0.26

(0.1

4 to

0.4

8)N

aive

indi

rect

: 0.

73 (0

.68

to 0

.78)

14 Col

lins

et a

l.138

Patie

nts

with

pos

tope

rativ

e pa

in

(No.

of p

atie

nts

with

>50

% p

ain

relie

f)

Ibup

rofe

n 40

0 m

g vs

200

mg:

5 (1

99/2

02)

Ibup

rofe

n 40

0 m

g vs

con

trol

: 26

(140

7/10

43)

Ibup

rofe

n 20

0 m

g vs

con

trol

: 3

(204

/133

)

RR Dire

ct e

stim

ate:

1.

39 (1

.08

to 1

.79)

Adj

uste

d in

dire

ct:

0.74

(0.2

7 to

2.0

2)N

aive

indi

rect

: 1.

16 (1

.00

to 1

.34)

Appendix 8

130 TA

BLE

33

Iden

tifie

d m

eta-

anal

yses

with

suf

ficie

nt d

ata

for b

oth

dire

ct a

nd in

dire

ct c

ompa

rison

s (c

ont’d

)

Met

a-an

alys

isPa

tien

ts (

outc

ome)

Inte

rven

tion

s co

mpa

red:

no.

of t

rial

s (p

atie

nts)

Rel

ativ

e ef

ficac

y (9

5% C

I)

cont

inue

d

15 Del

aney

et

al.13

9

Patie

nts

with

dys

peps

ia

(Glo

bal a

sses

smen

t)

PPI v

s H

2RA

:3

(633

/634

)PP

I vs

algi

nate

/ant

acid

:2

(595

/591

)H

2RA

vs

algi

nate

/ant

acid

:1

(119

/136

)

RR Dire

ct e

stim

ate:

0.

64 (0

.49

to 0

.82)

Adj

uste

d in

dire

ct:

0.73

(0.5

6 to

0.9

6)N

aive

indi

rect

: 0.

86 (0

.71

to 1

.05)

16 Di M

ario

et

al.14

0

Patie

nts

with

pre

viou

sly u

ntre

ated

gas

tric

ulce

r

(End

osco

pic

heal

ing)

Cim

etid

ine

vs r

aniti

dine

: 5

(tot

al 6

36)

Cim

editi

ne v

s pl

aceb

o:13

(tot

al 8

52)

Rani

tidin

e vs

pla

cebo

:8

(tot

al 7

56)

RR Dire

ct e

stim

ate:

1.21

(0.8

8 to

1.6

7)A

djus

ted

indi

rect

:0.

65 (0

.35

to 1

.20)

N

aive

indi

rect

: N

ot a

vaila

ble

17 Han

doll

et a

l.141

Patie

nts

unde

rgoi

ng s

urge

ry fo

r hi

pfr

actu

res

(DVT

)

LMW

H v

s U

FH:

3 (1

36/1

11)

LMW

H v

s pl

aceb

o:

2 (1

04/1

10)

UFH

vs

plac

ebo:

10

(407

/409

)

RR Dire

ct e

stim

ate:

0.

91 (0

.36

to 2

.31)

Adj

uste

d in

dire

ct:

1.05

(0.5

2 to

2.1

3)N

aive

indi

rect

: 0.

68 (0

.44

to 1

.07)

18 Hor

n an

dLi

mbu

rg14

2

Patie

nts

with

acu

te is

chem

ic s

trok

e

(Poo

r ou

tcom

e: d

eath

or

depe

nden

cy in

activ

ities

of d

aily

livi

ng)

Mim

odip

ine

240

mg

vs 1

20 m

g:2

(340

/341

)M

imod

ipin

e 24

0 m

g vs

con

trol

: 1

(73/

69)

Mim

odip

ine

120

mg

vs c

ontr

ol:

13 (2

081/

2106

)

RR Dire

ct e

stim

ate:

1.

07 (0

.94

to 1

.22)

Adj

uste

d in

dire

ct:

0.97

(0.6

3 to

1.4

8)N

aive

indi

rect

: 1.

06 (0

.79

to 1

.42)

19 Mar

shal

l and

Irvi

ne21

Patie

nts

with

ulc

erat

ive

colit

is

(End

osco

pic

rem

issio

n)

5-A

SA v

s re

ctal

cor

ticos

tero

ids:

7 (t

otal

682

)5-

ASA

vs

R. b

udes

onid

e:

2 (t

otal

154

)Re

ctal

cor

ticos

tero

ids

vs r

ecta

l bud

eson

ide:

5 (t

otal

463

)

OR

Dire

ct e

stim

ate:

0.

53 (0

.36

to 0

.78)

Adj

uste

d in

dire

ct:

0.92

(0.3

6 to

2.3

6)

Nai

ve in

dire

ct:

1.13

(0.8

6 to

1.4

9)

20 McI

ntos

h an

dO

lliar

o143

Patie

nts

with

unc

ompl

icat

ed m

alar

ia

(Par

asite

cle

aran

ce a

t da

y 28

)

Art

emisi

nin

vs a

rtes

unat

e:1

(20/

19)

Art

emisi

nin

vs q

uini

ne:

1 (2

7/22

)A

rtes

unat

e vs

qui

nine

: 1

(47/

39)

RR Dire

ct e

stim

ate:

0.

82 (0

.55

to 1

.22)

Adj

uste

d in

dire

ct:

0.70

(0.3

8 to

1.2

8)

Nai

ve in

dire

ct:

0.42

(0.2

6 to

0.6

6)

21 Moo

re e

t al

.24

Patie

nts

with

sev

ere

post

oper

ativ

e pa

in

(>50

% p

ain

relie

f)

Para

ceta

mol

+co

dein

e vs

par

acet

amol

: 10

(309

/313

)Pa

race

tam

ol+

code

ine

vs p

lace

bo:

5 (9

8/11

0)Pa

race

tam

ol v

s pl

aceb

o:

7 (2

81/2

54)

RR Dire

ct e

stim

ate:

1.

24 (1

.01

to 1

.54)

Adj

uste

d in

dire

ct:

1.74

(0.5

9 to

5.1

8)

Nai

ve in

dire

ct:

1.03

(0.7

8 to

1.3

5)


131


TA

BLE

33

Iden

tifie

d m

eta-

anal

yses

with

suf

ficie

nt d

ata

for b

oth

dire

ct a

nd in

dire

ct c

ompa

rison

s (c

ont’d

)

Met

a-an

alys

isPa

tien

ts (

outc

ome)

Inte

rven

tion

s co

mpa

red:

no.

of t

rial

s (p

atie

nts)

Rel

ativ

e ef

ficac

y (9

5% C

I)

cont

inue

d

22 Pagl

iaro

(use

d by

Hig

gins

and

Whi

tehe

ad53

)

Patie

nts

with

cirr

hosis

and

oeso

phag

ogas

tric

var

ices

(Rat

e of

firs

t bl

eedi

ng)

�-B

lock

ers

vs s

cler

othe

rapy

:2

(111

/115

)�

-Blo

cker

s vs

con

trol

:7

(378

/394

)Sc

lero

ther

apy

vs c

ontr

ol:

17 (7

23/7

36)

RR Dire

ct e

stim

ate:

0.

53 (0

.12

to 2

.36)

Adj

uste

d in

dire

ct:

1.00

(0.5

3 to

1.8

9)N

aive

indi

rect

: 0.

69 (0

.53

to 0

.91)

23 Poyn

ard

et a

l.26

Patie

nts

with

vira

l hep

atiti

s C

(Sus

tain

ed A

LT r

espo

nse

rate

)

Inte

rfer

on (3

MU

) 12

mon

ths

vs 6

mon

ths:

4 (2

56/2

49)

Inte

rfer

on (3

MU

) 12

mon

ths

vs c

ontr

ol:

5 (1

61/1

57)

Inte

rfer

on (3

MU

) 6 m

onth

s vs

con

trol

:6

(132

/131

)

RR Dire

ct e

stim

ate:

2.

20 (1

.52

to 3

.17)

Adj

uste

d in

dire

ct:

1.49

(0.3

5 to

6.3

1)

Nai

ve in

dire

ct:

1.67

(1.1

5 to

2.4

2)

24 Rost

om e

t al

.107

Patie

nts

who

had

tak

en N

SAID

s fo

r >

3w

eeks

(Tot

al e

ndos

copi

c ul

cers

)

PPI v

s H

2RA

: 1

(210

/215

)PP

I vs

plac

ebo:

3

(443

/331

)H

2RA

vs

plac

ebo:

6

(645

/541

)

RR Dire

ct e

stim

ate:

0.

28 (0

.15

to 0

.51)

Adj

uste

d in

dire

ct:

0.61

(0.4

0 to

0.9

3)

Nai

ve in

dire

ct:

1.02

(0.7

2 to

1.4

4)

25 Sila

gy14

4

Smok

ers

(Sm

okin

g ce

ssat

ion

rate

)

>1

visit

vs

1 vi

sit:

5 (7

33/5

21)

>1

visit

vs

cont

rol:

4 (1

931/

1529

)1

visit

vs

cont

rol:

15 (7

551/

5826

)

RR Dire

ct e

stim

ate:

1.

51 (1

.08

to 2

.12)

Adj

uste

d in

dire

ct:

1.51

(0.9

0 to

2.5

6)

Nai

ve in

dire

ct:

2.34

(2.0

2 to

2.7

1)

26 Sila

gy e

t al

.145

Patie

nts

unde

rgoi

ng N

RT

(Sm

okin

g ce

ssat

ion

rate

)

Nic

otin

e pa

tch

24 h

ours

vs

16 h

ours

1 (5

1/55

)N

icot

ine

patc

h 25

hou

rs v

s co

ntro

l:24

(541

5/43

04)

Nic

otin

e pa

tch

16 h

ours

vs

cont

rol:

8 (4

368/

1737

)

RR Dire

ct e

stim

ate:

0.

70 (0

.36

to 1

.35)

Adj

uste

d in

dire

ct:

0.82

(0.5

6 to

1.2

0)

Nai

ve in

dire

ct:

1.07

(0.9

7 to

1.1

8)

27 Sila

gy e

t al

.145

Sam

e as

abo

veN

RT w

eani

ng v

s no

wea

ning

:1

(68/

56)

NRT

wea

ning

vs

cont

rol:

24 (7

571/

4598

)N

RT n

o w

eani

ng v

s co

ntro

l:5

(701

/648

)

RR Dire

ct e

stim

ate:

0.

97 (0

.68

to 1

.38)

Adj

uste

d in

dire

ct:

0.75

(0.5

3 to

1.0

6)N

aive

indi

rect

: 0.

88 (0

.74

to 1

.05)

28 Soo

et a

l.108

Patie

nts

with

non

-ulc

er d

yspe

psia

(Glo

bal s

ympt

om a

sses

smen

t)

H2R

A v

s su

cral

fate

: 1

(47/

53)

H2R

A v

s pl

aceb

o:

8 (6

07/6

18)

Sucr

alfa

te v

s pl

aceb

o:

2 (1

29/1

17)

RR Dire

ct e

stim

ate:

2.

74 (1

.25

to 6

.02)

Adj

uste

d in

dire

ct:

0.99

(0.4

7 to

2.0

8)N

aive

indi

rect

: 1.

24 (0

.91

to 1

.70)

Appendix 8

132 TA

BLE

33

Iden

tifie

d m

eta-

anal

yses

with

suf

ficie

nt d

ata

for b

oth

dire

ct a

nd in

dire

ct c

ompa

rison

s (c

ont’d

)

Met

a-an

alys

isPa

tien

ts (

outc

ome)

Inte

rven

tion

s co

mpa

red:

no.

of t

rial

s (p

atie

nts)

Rel

ativ

e ef

ficac

y (9

5% C

I)

cont

inue

d

29 Soo

et a

l.108

Sam

e as

abo

vePr

okin

etic

s vs

H2R

A:

2 (2

08/2

15)

Prok

inet

ics

vs p

lace

bo:

10 (3

26/2

64)

Sucr

alfa

te v

s pl

aceb

o:

7 (4

96/5

08)

RR Dire

ct e

stim

ate:

0.

54 (0

.22

to 1

.33)

Adj

uste

d in

dire

ct:

0.66

(0.4

1 to

1.0

5)

Nai

ve in

dire

ct:

1.04

(0.8

4 to

1.2

8)

30 Trin

adad

e an

dM

enon

146

Patie

nts

with

maj

or d

epre

ssio

n

(No.

of d

ropo

uts)

Fluo

xetin

e vs

par

oxet

ine:

5

(327

/328

)Fl

uoxe

tine

vs c

ontr

ols:

40

(250

0/22

34)

Paro

xetin

e vs

con

trol

s:33

(147

1/14

23)

RR Dire

ct e

stim

ate:

1.

00 (0

.73

to 1

.37)

Adj

uste

d in

dire

ct:

1.00

(0.8

6 to

1.1

7)N

aive

indi

rect

: 1.

24 (1

.13

to 1

.35)

31 Trin

dade

and

Men

on14

6

Sam

e as

abo

veFl

uoxe

tine

vs fl

uvox

amin

e:

1 (4

9/51

)Fl

uoxe

tine

vs c

ontr

ols:

45

(290

9/23

78)

Fluv

oxam

ine

vs c

ontr

ols:

41

(116

8/11

66)

RR Dire

ct e

stim

ate:

0.

78 (0

.18

to 3

.31)

Adj

uste

d in

dire

ct:

0.89

(0.7

7 to

1.0

2)N

aive

indi

rect

: 1.

14 (1

.03

to 1

.26)

32 Trin

dade

and

Men

on14

6

Sam

e as

abo

vePa

roxe

tine

vs fl

uvox

amin

e:

1 (5

6/64

)Pa

roxe

tine

vs c

ontr

ols:

34

(133

5/12

88)

Fluv

oxam

ine

vs c

ontr

ols:

35

(102

5/10

19)

RR Dire

ct e

stim

ate:

0.

80 (0

.47

to 1

.35)

Adj

uste

d in

dire

ct:

0.77

(0.6

3 to

0.9

3)N

aive

indi

rect

: 0.

92 (0

.81

to 1

.03)

33 Trin

dade

and

Men

on14

6

Sam

e as

abo

veSe

rtra

line

vs fl

uvox

amin

e:

1 (4

8/49

)Se

rtra

line

vs c

ontr

ols:

9

(816

/759

)Fl

uvox

amin

e vs

con

trol

s:

23 (6

85/6

51)

RR

Dire

ct e

stim

ate:

0.

40 (0

.18

to 0

.86)

Adj

uste

d in

dire

ct:

0.81

(0.6

0 to

1.1

0)N

aive

indi

rect

: 0.

94 (0

.81

to 1

.09)

34 Trin

dede

and

Men

on14

6

Sam

e as

abo

veFl

uoxe

tine

vs s

ertr

alin

e:

3 (2

66/2

73)

Fluo

xetin

e vs

con

trol

s:35

(250

2/19

67)

Sert

ralin

e vs

con

trol

s:

9 (8

16/7

59)

RR Dire

ct e

stim

ate:

0.

90 (0

.38

to 2

.14)

Adj

uste

d in

dire

ct:

0.88

(0.7

0 to

1.1

1)N

aive

indi

rect

:1.

10 (0

.98

to 1

.23)

35 van

Pinx

tere

net

al.10

9

Patie

nts

with

GO

RD-li

ke s

ympt

oms

(Hea

rtbu

rn r

emiss

ion)

PPI v

s H

2RA

: 3

(122

8/66

4)PP

I vs

plac

ebo:

1

(161

/159

)H

2RA

vs

plac

ebo:

2

(511

/502

)

RR Dire

ct e

stim

ate:

0.

67 (0

.57

to 0

.80)

Adj

uste

d in

dire

ct:

0.45

(0.3

1 to

0.6

6)

Nai

ve in

dire

ct:

0.53

(0.4

0 to

0.7

1)


133


TA

BLE

33

Iden

tifie

d m

eta-

anal

yses

with

suf

ficie

nt d

ata

for b

oth

dire

ct a

nd in

dire

ct c

ompa

rison

s (c

ont’d

)

Met

a-an

alys

isPa

tien

ts (

outc

ome)

Inte

rven

tion

s co

mpa

red:

no.

of t

rial

s (p

atie

nts)

Rel

ativ

e ef

ficac

y (9

5% C

I)

cont

inue

d

36 Zha

ng a

nd

Li W

an P

o147

Patie

nts

with

dys

men

orrh

oea

(No.

of p

atie

nts

with

at

leas

t m

oder

ate

pain

relie

f)

Nap

roxe

n vs

ibup

rofe

n:

3 (1

22/1

13)

Nap

roxe

n vs

pla

cebo

: 17

(904

/877

)Ib

upro

fen

vs p

lace

bo:

10 (3

45/3

46)

RR Dire

ct e

stim

ate:

1.

08 (0

.79

to 1

.48)

Adj

uste

d in

dire

ct:

1.40

(0.9

4 to

2.0

9)N

aive

indi

rect

: 0.

82 (0

.75

to 0

.89)

37 Zha

ng a

nd

Li W

an P

o147

Sam

e as

abo

veN

apro

xen

vs m

efen

amic

aci

d:

1 (2

4/20

)N

apro

xen

vs p

lace

bo:

18 (9

42/9

16)

Mef

enam

ic v

s pl

aceb

o:

4 (3

07/3

07)

RR Dire

ct e

stim

ate:

2.

40 (1

.39

to 4

.13)

Adj

uste

d in

dire

ct:

1.53

(1.1

1 to

2.1

2)N

aive

indi

rect

: 0.

88 (0

.80

to 0

.97)

38 Zha

ng a

nd

Li W

an P

o147

Sam

e as

abo

veN

apro

xen

vs a

spiri

n:

1 (3

2/32

)N

apro

xen

vs p

lace

bo:

17 (8

90/8

72)

Asp

irin

vs p

lace

bo:

5 (2

20/2

23)

RR Dire

ct e

stim

ate:

2.

29 (1

.16

to 4

.52)

Adj

uste

d in

dire

ct:

2.45

(1.6

5 to

3.6

4)N

aive

indi

rect

: 1.

83 (1

.49

to 2

.23)

39 Zha

ng a

nd

Li W

an P

o147

Sam

e as

abo

veIb

upro

fen

vs a

spiri

n:1

(43/

43)

Ibup

rofe

n vs

pla

cebo

: 10

(322

/327

)A

spiri

n vs

pla

cebo

: 5

(187

/184

)

RR Dire

ct e

stim

ate:

1.

90 (1

.30

to 2

.77)

Adj

uste

d in

dire

ct:

1.80

(1.1

2 to

2.8

9)N

aive

indi

rect

: 2.

32 (1

.84

to 2

.93)

40 Aus

ejo14

8

Chi

ldre

n w

ith c

roup

(Impr

ovem

ent

in c

roup

sev

erity

sco

re)

Bude

soni

de v

s de

xam

etha

sone

:1

(65/

69)

Bude

soni

de v

s pl

aceb

o:

5 (1

66/1

61)

Dex

amet

haso

ne v

s pl

aceb

o:

8 (3

65/3

74)

Stan

dard

ised

mea

n di

ffere

nce

Dire

ct e

stim

ate:

0.

09 (–

0.25

to

0.43

)A

djus

ted

indi

rect

: 0.

32 (–

0.52

to

1.16

)N

aive

indi

rect

: 0.

20 (0

.02

to 0

.38)

41 Li W

an P

o an

dZ

hang

20

Patie

nts

with

pos

tsur

gica

l pai

n

(Sum

of d

iffer

ence

in p

ain

inte

nsity

)

Para

ceta

mol

+de

xam

etha

sone

vs

para

ceta

mol

:3

(103

/99)

Para

ceta

mol

+de

xam

etha

sone

vs

plac

ebo:

5

(181

/178

)Pa

race

tam

ol v

s pl

aceb

o:

14 (5

58/5

34)

Mea

n di

ffere

nce

Dire

ct e

stim

ate:

1.

22 (0

.00

to 2

.45)

Adj

uste

d in

dire

ct:

0.51

(–0.

43 t

o 1.

45)

Nai

ve in

dire

ct:

0.13

(–0.

61 t

o 0.

88)

42 Pack

er e

t al

.69

Patie

nts

with

hea

rt fa

ilure

(Cha

nges

in le

ft ve

ntric

ular

eje

ctio

nfr

actio

n)

Car

vedi

lol v

s m

etop

rolo

l: 4

(123

/125

)C

arve

dilo

l vs

plac

ebo:

9

(534

/668

)M

etop

rolo

l vs

plac

ebo:

6

(376

/408

)

Mea

n di

ffere

nce

Dire

ct e

stim

ate:

0.

029

(0.0

07 t

o 0.

051)

Adj

uste

d in

dire

ct:

0.02

7 (0

.013

to

0.04

1)

Nai

ve in

dire

ct:

0.01

6 (0

.012

to

0.02

04)

Appendix 8

134 TA

BLE

33

Iden

tifie

d m

eta-

anal

yses

with

suf

ficie

nt d

ata

for b

oth

dire

ct a

nd in

dire

ct c

ompa

rison

s (c

ont’d

)

Met

a-an

alys

isPa

tien

ts (

outc

ome)

Inte

rven

tion

s co

mpa

red:

no.

of t

rial

s (p

atie

nts)

Rel

ativ

e ef

ficac

y (9

5% C

I)

AP,

aero

lised

pen

tam

idin

e; D

, dap

sone

; NA

, not

ava

ilabl

e; N

RT, n

icot

ine

repl

acem

ent

ther

apy;

P, p

yrim

etha

min

e; S

MX

, sul

fam

etho

xazo

le; T

MP,

trim

etho

prim

.

43 Saur

iol e

t al

.149

Patie

nts

with

sch

izop

hren

ia

(Cha

nges

in b

rief p

sych

iatr

ic r

atin

g sc

ale)

Ola

nzap

ine

vs r

isper

idon

e:

1 (1

72/1

67)

Ola

nzap

ine

vs p

lace

bo:

3 (1

620/

786)

Risp

erid

one

vs p

lace

bo:

8 (1

044/

416)

Mea

n di

ffere

nce

Dire

ct e

stim

ate:

1.

80 (–

1.43

to

5.03

)A

djus

ted

indi

rect

: 1.

33 (–

0.63

to

3.29

) N

aive

indi

rect

: 1.

07 (–

0.28

to

2.42

)

44 Zha

ng a

nd

Li W

an P

o147

Patie

nts

with

sur

gica

l pai

n

(Sum

of d

iffer

ence

in p

ain

inte

nsity

)

Para

ceta

mol

+co

dein

e vs

par

acet

amol

: 13

(449

/448

)Pa

race

tam

ol+

code

ine

vs p

lace

bo:

12 (N

A)

Para

ceta

mol

vs

plac

ebo:

31

(NA

)

Mea

n di

ffere

nce

Dire

ct e

stim

ate:

6.

97 (3

.56

to 1

0.37

)A

djus

ted

indi

rect

: –1

.16

(–6.

95 t

o 4.

64)

Nai

ve in

dire

ct:

–9.8

9 (–

11.6

5 to

–8.

13)


145

Health Technology AssessmentProgramme

Prioritisation Strategy GroupMembers

Chair,Professor Tom Walley, Director, NHS HTA Programme,Department of Pharmacology &Therapeutics,University of Liverpool

Professor Bruce Campbell,Consultant Vascular & GeneralSurgeon, Royal Devon & ExeterHospital

Dr Edmund Jessop, MedicalAdvisor, National Specialist,Commissioning Advisory Group(NSCAG), Department ofHealth, London

Professor Jon Nicholl, Director,Medical Care Research Unit,University of Sheffield, Schoolof Health and Related Research

Dr John Reynolds, ClinicalDirector, Acute GeneralMedicine SDU, RadcliffeHospital, Oxford

Dr Ron Zimmern, Director,Public Health Genetics Unit,Strangeways ResearchLaboratories, Cambridge

HTA Commissioning BoardMembers

Programme Director, Professor Tom Walley, Director, NHS HTA Programme,Department of Pharmacology &Therapeutics,University of Liverpool

Chair,Professor Jon Nicholl,Director, Medical Care ResearchUnit, University of Sheffield,School of Health and RelatedResearch

Deputy Chair, Professor Jenny Hewison,Professor of Health CarePsychology, Academic Unit ofPsychiatry and BehaviouralSciences, University of LeedsSchool of Medicine

Dr Jeffrey AronsonReader in ClinicalPharmacology, Department ofClinical Pharmacology,Radcliffe Infirmary, Oxford

Professor Deborah Ashby,Professor of Medical Statistics,Department of Environmentaland Preventative Medicine,Queen Mary University ofLondon

Professor Ann Bowling,Professor of Health ServicesResearch, Primary Care andPopulation Studies,University College London

Dr Andrew Briggs, PublicHealth Career Scientist, HealthEconomics Research Centre,University of Oxford

Professor John Cairns, Professorof Health Economics, PublicHealth Policy, London School ofHygiene and Tropical Medicine,London

Professor Nicky Cullum,Director of Centre for EvidenceBased Nursing, Department ofHealth Sciences, University ofYork

Mr Jonathan Deeks, Senior Medical Statistician,Centre for Statistics inMedicine, University of Oxford

Dr Andrew Farmer, SeniorLecturer in General Practice,Department of Primary Health Care, University of Oxford

Professor Fiona J Gilbert,Professor of Radiology,Department of Radiology,University of Aberdeen

Professor Adrian Grant,Director, Health ServicesResearch Unit, University ofAberdeen

Professor F D Richard Hobbs,Professor of Primary Care &General Practice, Department ofPrimary Care & GeneralPractice, University ofBirmingham

Professor Peter Jones, Head ofDepartment, UniversityDepartment of Psychiatry,University of Cambridge

Professor Sallie Lamb, Professor of Rehabilitation,Centre for Primary Health Care, University of Warwick

Professor Stuart Logan,Director of Health & SocialCare Research, The Peninsula Medical School, Universities of Exeter &Plymouth

Dr Linda Patterson, Consultant Physician,Department of Medicine,Burnley General Hospital

Professor Ian Roberts, Professorof Epidemiology & PublicHealth, Intervention ResearchUnit, London School ofHygiene and Tropical Medicine

Professor Mark Sculpher,Professor of Health Economics,Centre for Health Economics,Institute for Research in theSocial Services, University of York

Dr Jonathan Shapiro, SeniorFellow, Health ServicesManagement Centre,Birmingham

Ms Kate Thomas,Deputy Director,Medical Care Research Unit,University of Sheffield

Ms Sue Ziebland,Research Director, DIPEx,Department of Primary HealthCare, University of Oxford,Institute of Health Sciences

Current and past membership details of all HTA ‘committees’ are available from the HTA website (www.ncchta.org)


Health Technology Assessment Programme

146

Diagnostic Technologies & Screening PanelMembers

Chair,Dr Ron Zimmern, Director ofthe Public Health Genetics Unit,Strangeways ResearchLaboratories, Cambridge

Ms Norma Armston,Lay Member, Bolton

Professor Max BachmannProfessor of Health Care Interfaces, Department of Health Policy and Practice,University of East Anglia

Professor Rudy BilousProfessor of Clinical Medicine &Consultant Physician,The Academic Centre,South Tees Hospitals NHS Trust

Dr Paul Cockcroft, Consultant MedicalMicrobiologist and ClinicalDirector of Pathology,Department of ClinicalMicrobiology, St Mary'sHospital, Portsmouth

Professor Adrian K Dixon,Professor of Radiology,University Department ofRadiology, University ofCambridge Clinical School

Dr David Elliman, Consultant Paediatrician/Hon. Senior Lecturer,Population Health Unit, Great Ormond St. Hospital,London

Professor Glyn Elwyn,Primary Medical Care Research Group,Swansea Clinical School,University of Wales Swansea

Mr Tam Fry, HonoraryChairman, Child GrowthFoundation, London

Dr Jennifer J Kurinczuk,Consultant ClinicalEpidemiologist,National PerinatalEpidemiology Unit, Oxford

Dr Susanne M Ludgate, MedicalDirector, Medicines &Healthcare Products RegulatoryAgency, London

Professor William Rosenberg,Professor of Hepatology, LiverResearch Group, University ofSouthampton

Dr Susan Schonfield, Consultantin Public Health, SpecialisedServices Commissioning NorthWest London, HillingdonPrimary Care Trust

Dr Phil Shackley, SeniorLecturer in Health Economics,School of Population andHealth Sciences, University ofNewcastle upon Tyne

Dr Margaret Somerville, PMSPublic Health Lead, PeninsulaMedical School, University ofPlymouth

Dr Graham Taylor, ScientificDirector & Senior Lecturer,Regional DNA Laboratory, TheLeeds Teaching Hospitals

Professor Lindsay WilsonTurnbull, Scientific Director,Centre for MR Investigations &YCR Professor of Radiology,University of Hull

Professor Martin J Whittle,Associate Dean for Education,Head of Department ofObstetrics and Gynaecology,University of Birmingham

Dr Dennis Wright, Consultant Biochemist &Clinical Director, Pathology & The KennedyGalton Centre, Northwick Park & St Mark’sHospitals, Harrow

Pharmaceuticals PanelMembers

Chair,Dr John Reynolds, ChairDivision A, The John RadcliffeHospital, Oxford RadcliffeHospitals NHS Trust

Professor Tony Avery, Head of Division of PrimaryCare, School of CommunityHealth Services, Division ofGeneral Practice, University ofNottingham

Ms Anne Baileff, ConsultantNurse in First Contact Care,Southampton City Primary CareTrust, University ofSouthampton

Professor Stirling Bryan,Professor of Health Economics,Health Services Management Centre,University of Birmingham

Mr Peter Cardy, ChiefExecutive, Macmillan CancerRelief, London

Professor Imti Choonara,Professor in Child Health,Academic Division of ChildHealth, University ofNottingham

Dr Robin Ferner, ConsultantPhysician and Director, WestMidlands Centre for AdverseDrug Reactions, City HospitalNHS Trust, Birmingham

Dr Karen A Fitzgerald,Consultant in PharmaceuticalPublic Health, National PublicHealth Service for Wales,Cardiff

Mrs Sharon Hart, Head of DTB Publications, Drug &Therapeutics Bulletin, London

Dr Christine Hine, Consultant inPublic Health Medicine, SouthGloucestershire Primary CareTrust

Professor Stan Kaye,Cancer Research UK Professor of Medical Oncology,Section of Medicine, The Royal Marsden Hospital,Sutton

Ms Barbara Meredith,Lay Member, Epsom

Dr Andrew Prentice, SeniorLecturer and ConsultantObstetrician & Gynaecologist,Department of Obstetrics &Gynaecology, University ofCambridge

Dr Frances Rotblat, CPMPDelegate, Medicines &Healthcare Products RegulatoryAgency, London

Professor Jan Scott, Professor of Psychological Treatments,Institute of Psychiatry,University of London

Mrs Katrina Simister, AssistantDirector New Medicines,National Prescribing Centre,Liverpool

Dr Richard Tiner, MedicalDirector, Medical Department,Association of the BritishPharmaceutical Industry,London

Dr Helen Williams,Consultant Microbiologist,Norfolk & Norwich UniversityHospital NHS Trust



147

Therapeutic Procedures PanelMembers

Chair, Professor Bruce Campbell,Consultant Vascular andGeneral Surgeon, Departmentof Surgery, Royal Devon &Exeter Hospital

Dr Aileen Clarke,Reader in Health ServicesResearch, Public Health &Policy Research Unit, Barts &the London School of Medicine& Dentistry, London

Dr Matthew Cooke, Reader inA&E/Department of HealthAdvisor in A&E, WarwickEmergency Care andRehabilitation, University ofWarwick

Dr Carl E Counsell, ClinicalSenior Lecturer in Neurology,Department of Medicine andTherapeutics, University ofAberdeen

Ms Amelia Curwen, ExecutiveDirector of Policy, Services andResearch, Asthma UK, London

Professor Gene Feder, Professorof Primary Care R&D,Department of General Practiceand Primary Care, Barts & theLondon, Queen Mary’s Schoolof Medicine and Dentistry,London

Professor Paul Gregg,Professor of OrthopaedicSurgical Science, Department ofGeneral Practice and PrimaryCare, South Tees Hospital NHSTrust, Middlesbrough

Ms Bec Hanley, Co-Director,TwoCan Associates,Hurstpierpoint

Ms Maryann L Hardy, Lecturer, Division ofRadiography, University ofBradford

Professor Alan Horwich,Director of Clinical R&D,Academic Department ofRadiology, The Institute ofCancer Research, London

Dr Simon de Lusignan,Senior Lecturer, Primary Care Informatics,Department of CommunityHealth Sciences,St George’s Hospital MedicalSchool, London

Professor Neil McIntosh,Edward Clark Professor of Child Life & Health,Department of Child Life &Health, University of Edinburgh

Professor James Neilson,Professor of Obstetrics andGynaecology, Department ofObstetrics and Gynaecology,University of Liverpool

Dr John C Pounsford,Consultant Physician,Directorate of Medical Services,North Bristol NHS Trust

Karen Roberts, NurseConsultant, Queen ElizabethHospital, Gateshead

Dr Vimal Sharma, ConsultantPsychiatrist/Hon. Senior Lecturer,Mental Health Resource Centre,Cheshire and Wirral PartnershipNHS Trust, Wallasey

Dr L David Smith, ConsultantCardiologist, Royal Devon &Exeter Hospital

Professor Norman Waugh,Professor of Public Health,Department of Public Health,University of Aberdeen


Health Technology Assessment Programme

148Current and past membership details of all HTA ‘committees’ are available from the HTA website (www.ncchta.org)

Expert Advisory NetworkMembers

Professor Douglas Altman,Director of CSM & CancerResearch UK Med Stat Gp,Centre for Statistics inMedicine, University of Oxford,Institute of Health Sciences,Headington, Oxford

Professor John Bond,Director, Centre for HealthServices Research, University ofNewcastle upon Tyne, School ofPopulation & Health Sciences,Newcastle upon Tyne

Mr Shaun Brogan, Chief Executive, RidgewayPrimary Care Group, Aylesbury

Mrs Stella Burnside OBE,Chief Executive, Office of theChief Executive. TrustHeadquarters, AltnagelvinHospitals Health & SocialServices Trust, Altnagelvin AreaHospital, Londonderry

Ms Tracy Bury, Project Manager, WorldConfederation for PhysicalTherapy, London

Professor Iain T Cameron,Professor of Obstetrics andGynaecology and Head of theSchool of Medicine,University of Southampton

Dr Christine Clark,Medical Writer & ConsultantPharmacist, Rossendale

Professor Collette Clifford,Professor of Nursing & Head ofResearch, School of HealthSciences, University ofBirmingham, Edgbaston,Birmingham

Professor Barry Cookson,Director, Laboratory ofHealthcare Associated Infection,Health Protection Agency,London

Professor Howard Cuckle,Professor of ReproductiveEpidemiology, Department ofPaediatrics, Obstetrics &Gynaecology, University ofLeeds

Dr Katherine Darton, Information Unit, MIND – The Mental Health Charity,London

Professor Carol Dezateux, Professor of PaediatricEpidemiology, London

Mr John Dunning,Consultant CardiothoracicSurgeon, CardiothoracicSurgical Unit, PapworthHospital NHS Trust, Cambridge

Mr Jonothan Earnshaw,Consultant Vascular Surgeon,Gloucestershire Royal Hospital,Gloucester

Professor Martin Eccles, Professor of ClinicalEffectiveness, Centre for HealthServices Research, University ofNewcastle upon Tyne

Professor Pam Enderby,Professor of CommunityRehabilitation, Institute ofGeneral Practice and PrimaryCare, University of Sheffield

Mr Leonard R Fenwick, Chief Executive, Newcastleupon Tyne Hospitals NHS Trust

Professor David Field, Professor of Neonatal Medicine,Child Health, The LeicesterRoyal Infirmary NHS Trust

Mrs Gillian Fletcher, Antenatal Teacher & Tutor andPresident, National ChildbirthTrust, Henfield

Professor Jayne Franklyn,Professor of Medicine,Department of Medicine,University of Birmingham,Queen Elizabeth Hospital,Edgbaston, Birmingham

Ms Grace Gibbs, Deputy Chief Executive,Director for Nursing, Midwifery& Clinical Support Services, West Middlesex UniversityHospital, Isleworth

Dr Neville Goodman, Consultant Anaesthetist,Southmead Hospital, Bristol

Professor Alastair Gray,Professor of Health Economics,Department of Public Health,University of Oxford

Professor Robert E Hawkins, CRC Professor and Director ofMedical Oncology, Christie CRCResearch Centre, ChristieHospital NHS Trust, Manchester

Professor Allen Hutchinson, Director of Public Health &Deputy Dean of ScHARR,Department of Public Health,University of Sheffield

Dr Duncan Keeley,General Practitioner (Dr Burch& Ptnrs), The Health Centre,Thame

Dr Donna Lamping,Research Degrees ProgrammeDirector & Reader in Psychology,Health Services Research Unit,London School of Hygiene andTropical Medicine, London

Mr George Levvy,Chief Executive, MotorNeurone Disease Association,Northampton

Professor James Lindesay,Professor of Psychiatry for theElderly, University of Leicester,Leicester General Hospital

Professor Julian Little,Professor of Human GenomeEpidemiology, Department ofEpidemiology & CommunityMedicine, University of Ottawa

Professor Rajan Madhok, Medical Director & Director ofPublic Health, Directorate ofClinical Strategy & PublicHealth, North & East Yorkshire& Northern Lincolnshire HealthAuthority, York

Professor David Mant, Professor of General Practice,Department of Primary Care,University of Oxford

Professor Alexander Markham, Director, Molecular MedicineUnit, St James’s UniversityHospital, Leeds

Dr Chris McCall, General Practitioner, TheHadleigh Practice, Castle Mullen

Professor Alistair McGuire,Professor of Health Economics,London School of Economics

Dr Peter Moore, Freelance Science Writer, Ashtead

Dr Sue Moss, Associate Director,Cancer Screening EvaluationUnit, Institute of CancerResearch, Sutton

Mrs Julietta Patnick, Director, NHS Cancer ScreeningProgrammes, Sheffield

Professor Tim Peters,Professor of Primary CareHealth Services Research,Academic Unit of PrimaryHealth Care, University ofBristol

Professor Chris Price, Visiting Chair – Oxford, ClinicalResearch, Bayer DiagnosticsEurope, Cirencester

Professor Peter Sandercock,Professor of Medical Neurology,Department of ClinicalNeurosciences, University ofEdinburgh

Dr Eamonn Sheridan,Consultant in Clinical Genetics,Genetics Department,St James’s University Hospital,Leeds

Dr Ken Stein,Senior Clinical Lecturer inPublic Health, Director,Peninsula TechnologyAssessment Group, University of Exeter

Professor Sarah Stewart-Brown, Professor of Public Health,University of Warwick, Division of Health in theCommunity Warwick MedicalSchool, LWMS, Coventry

Professor Ala Szczepura, Professor of Health ServiceResearch, Centre for HealthServices Studies, University ofWarwick

Dr Ross Taylor, Senior Lecturer, Department ofGeneral Practice and PrimaryCare, University of Aberdeen

Mrs Joan Webster, Consumer member, HTA –Expert Advisory Network

How to obtain copies of this and other HTA Programme reports.An electronic version of this publication, in Adobe Acrobat format, is available for downloading free ofcharge for personal use from the HTA website (http://www.ncchta.org). A fully searchable CD-ROM isalso available (see below).

Printed copies of HTA monographs cost £20 each (post and packing free in the UK) to both public andprivate sector purchasers from our Despatch Agents, York Publishing Services.

Non-UK purchasers will have to pay a small fee for post and packing. For European countries the cost is£2 per monograph and for the rest of the world £3 per monograph.

You can order HTA monographs from our Despatch Agents, York Publishing Services by:

– fax (with credit card or official purchase order) – post (with credit card or official purchase order or cheque)– phone during office hours (credit card only).

Additionally the HTA website allows you either to pay securely by credit card or to print out yourorder and then post or fax it.

Contact details are as follows:York Publishing Services Email: [email protected] Box 642 Tel: 0870 1616662YORK YO31 7WX Fax: 0870 1616663UK Fax from outside the UK: +44 1904 430868

NHS libraries can subscribe free of charge. Public libraries can subscribe at a very reduced cost of £100 for each volume (normally comprising 30–40 titles). The commercial subscription rate is £300 per volume. Please contact York Publishing Services at the address above. Subscriptions can only bepurchased for the current or forthcoming volume.

Payment methods

Paying by chequeIf you pay by cheque, the cheque must be in pounds sterling, made payable to York PublishingDistribution and drawn on a bank with a UK address.

Paying by credit cardThe following cards are accepted by phone, fax, post or via the website ordering pages: Delta, Eurocard,Mastercard, Solo, Switch and Visa. We advise against sending credit card details in a plain email.

Paying by official purchase orderYou can post or fax these, but they must be from public bodies (i.e. NHS or universities) within the UK.We cannot at present accept purchase orders from commercial companies or from outside the UK.

How do I get a copy of HTA on CD?

Please use the form on the HTA website (www.ncchta.org/htacd.htm). Or contact York PublishingServices (see contact details above) by email, post, fax or phone. HTA on CD is currently free of chargeworldwide.

The website also provides information about the HTA Programme and lists the membership of the variouscommittees.

HTA

Health Technology Assessm

ent 2005;Vol. 9: No. 26

Indirect comparisons of com

peting interventions


AM Glenny, DG Altman, F Song, C Sakarovitch, JJ Deeks, R D’Amico, M Bradburn and AJ Eastwood


HTAHealth Technology AssessmentNHS R&D HTA Programme

The National Coordinating Centre for Health Technology Assessment,Mailpoint 728, Boldrewood,University of Southampton,Southampton, SO16 7PX, UK.Fax: +44 (0) 23 8059 5639 Email: [email protected]://www.ncchta.org ISSN 1366-5278

FeedbackThe HTA Programme and the authors would like to know

your views about this report.

The Correspondence Page on the HTA website(http://www.ncchta.org) is a convenient way to publish

your comments. If you prefer, you can send your comments to the address below, telling us whether you would like

us to transfer them to the website.

We look forward to hearing from you.

July 2005

nhs r&d hta programmepure-oai.bham.ac.uk/ws/files/17501744/deeks... · random samples of...

Documents