• Design and Reporting Design and Reporting Modifications in Industry-Modifications in Industry-Sponsored ComparativeSponsored Comparative

• Psychopharmacology TrialsPsychopharmacology Trials• DANIEL J. SAFER, M.D.1DANIEL J. SAFER, M.D.1• J nerve mental dis 2002J nerve mental dis 2002

BMJ 2006;333;782-; originally published online 6 Oct 2006;Anders W Jørgensen, Jørgen Hilden and Peter C Gøtzschereviewmeta-analyses of the same drugs: systematicsupported meta-analyses and otherCochrane reviews compared with industry

Are you at risk?Are you at risk?

falling victim to falling victim to misleading presentations as misleading presentations as a result of conflict of a result of conflict of interest of investigatorsinterest of investigators

Not at allNot at all A littleA little Moderate riskModerate risk High riskHigh risk

SpinningSpinning

• get in groups of 2 to 5get in groups of 2 to 5

• write down write down 3 ways3 ways in which in which papers can misleadpapers can mislead

• 3 minutes3 minutes

SpinningSpinning

• get in groups of 2 to 5get in groups of 2 to 5

• write down three ways to write down three ways to avoid being misleadavoid being mislead

• 3 minutes3 minutes

Conclusions and Conclusions and Funding, Funding,

Als-Nielsen, JAMA 2003Als-Nielsen, JAMA 2003• 25 Cochrane reviews, 370 RCTs25 Cochrane reviews, 370 RCTs

– random selection of 167 of 1081 (issue 2, 2001)random selection of 167 of 1081 (issue 2, 2001)– excluded excluded

• 101 < 5 studies 101 < 5 studies • 6 no concealment variability 6 no concealment variability • 2 no binary outcome 2 no binary outcome • 16 non-pharmacological 16 non-pharmacological

• abstractedabstracted– sources of fundingsources of funding– effect on primary outcomeeffect on primary outcome– adverse effectsadverse effects– methodological quality (concealment, blinding)methodological quality (concealment, blinding)

Scale Used to Grade Conclusions Scale Used to Grade Conclusions in Trialsin Trials

6 Points6 PointsExperimental intervention highly preferred and should now be Experimental intervention highly preferred and should now be

considered the standard intervention in all patients, or similarconsidered the standard intervention in all patients, or similar5 Points5 PointsExperimental intervention preferred to control, but further trials Experimental intervention preferred to control, but further trials

still indicated, experimental may be more costly, or similarstill indicated, experimental may be more costly, or similar4 Points4 PointsExperimental and control intervention about equal, but the Experimental and control intervention about equal, but the

experimental cheaper, easier to administer, or similar minor experimental cheaper, easier to administer, or similar minor advantageadvantage

3 Points3 PointsExperimental and control intervention about equal, but the control Experimental and control intervention about equal, but the control

may be cheaper, easier to administer, or similar minor advantagemay be cheaper, easier to administer, or similar minor advantage2 Points2 PointsControl intervention preferred to experimental intervention, but Control intervention preferred to experimental intervention, but

experimental intervention might be promising under some experimental intervention might be promising under some circumstances, or similarcircumstances, or similar

1 Point1 PointControl intervention highly preferred and should now be considered Control intervention highly preferred and should now be considered

the standard intervention in all patients, or similarthe standard intervention in all patients, or similar

Relation Between Funding Source Relation Between Funding Source and Conclusions in 370 Randomized and Conclusions in 370 Randomized

Drug TrialsDrug Trials

Abbreviation: IQR, interquartile range.*Conclusions in trials were assessed by a 1-6 point scale. If the conclusions recommended the experimental drug as the treatment of choice without disclaimers, 6 points was assigned, and if not, 1-5 points was assigned.+p<.001, using Kruskal-Wallis test (medians) or X2 test (proportions)

FundingFunding # of # of TrialTrial

ss

Median Median ScoreScore

(IQR)+(IQR)+

# (%) of # (%) of Trials Trials

Scoring 6 Scoring 6 Points+Points+

Nonprofit Nonprofit organizationsorganizations

6767 4 (3-5)4 (3-5) 11 (16.4)11 (16.4)

Not reportedNot reported 106106 5 (3-6)5 (3-6) 32 (30.1)32 (30.1)

Nonprofit and for-Nonprofit and for-profit profit organizationsorganizations

5151 5 (4-6)5 (4-6) 18 (35.2)18 (35.2)

For-profit For-profit organizationsorganizations

146146 6 (5-6)6 (5-6) 74 (50.6)74 (50.6)

TotalTotal 370370 5 (4-6)5 (4-6) 135 (36.4)135 (36.4)

Als-Nielson, JAMA, Als-Nielson, JAMA, 20032003

• bigger effect size, more likely bigger effect size, more likely stronger recommendationstronger recommendation

• blinding, more likely stronger blinding, more likely stronger recommendationrecommendation

• after adjustment, industry after adjustment, industry funding, odds ratio 5.3 (95% CI funding, odds ratio 5.3 (95% CI 2.0 to 14.4)2.0 to 14.4)

Read methods and Read methods and resultsresultsbypass the discussionbypass the discussion

1

ACP Journal Club?ACP Journal Club?

• secondary journal since 1991secondary journal since 1991– provides structured summaries provides structured summaries – articles important to internistsarticles important to internists– high methodologic qualityhigh methodologic quality

• therapy, randomizationtherapy, randomization• diagnosis, blind comparison to a gold standarddiagnosis, blind comparison to a gold standard

• may provide more information than may provide more information than original articleoriginal article– concealment of randomizationconcealment of randomization– who is blindwho is blind

• patients, caregivers, outcome data collectors, patients, caregivers, outcome data collectors, adjudictors of outcome, data analystsadjudictors of outcome, data analysts

• loss to follow-uploss to follow-up

PROGRESS, Lancet 2001PROGRESS, Lancet 2001

PROGRESS, Lancet 2001PROGRESS, Lancet 2001

MethodsMethods

ResultsResults

InterpretationInterpretation

Ask MD ifpatient willgive two drugs

NO

perindopril

placebo

YES RR

R

placebo

perindopril + indapamide

PROGRESS Trial – ACP PROGRESS Trial – ACP JCJC

PROGRESSPROGRESS

PROGRESSPROGRESS

ACP Journal Club

Read the abstract Read the abstract reported in evidence-reported in evidence-based secondary based secondary publications (ACPJC)publications (ACPJC)

2

Use of placebo when active comparator optimal

Angiotensin receptor blockers for diabetic nephropathy

vs.

ACE inhibitors

Parving H-H et al. N Engl J Med 2001;345(12):870-878Brenner BM N Engl J Med 2001;345(12):861-869Lewis EJ et al. N Engl J Med 2001;345(12):851-60.

Hostetter TH. N Engl J Med 2001;345(12):910-912

Do you manage patients with type II diabetes?

Is there any agent you would routinely suggest forpatients with diabetic nephropathy?

Incomparable Incomparable comparatorscomparators

• 8 RCTS of 28 RCTS of 2ndnd generation generation neuroleptics vs. 20 mg/d neuroleptics vs. 20 mg/d haloperidolhaloperidol– fewer extrapyramidal symptomsfewer extrapyramidal symptoms– standard dose haloperidol < 12 mgstandard dose haloperidol < 12 mg

• RCT paroxetine qd vs. RCT paroxetine qd vs. amitriptyline amitriptyline bidbid– less daytime somnolenceless daytime somnolence– standard amitriptyline qhsstandard amitriptyline qhs

Safer J Nerv Ment Dis 2002;190(9):583-92.Geddes J et al BMJ 2000;321(7273):1371-6.Christiansen PE, et al. Acta Psychiatr Scand 1996;93(3):158-63

• Neurology 2006;66;1294-1295Neurology 2006;66;1294-1295• Jacqueline A. French and Jacqueline A. French and Richard J. KryscioRichard J. Kryscio

• trialstrials• Active control trials for Active control trials for epilepsy : Avoiding bias in epilepsy : Avoiding bias in head-to-headhead-to-head

136 trials of newtreatments for multiple myeloma

For profit agency Nonprofit agency

40% 79%

Use of appropriate active comparatorinstead of placebo

Mann H et al. James Lind Library, 2003

• Goetschze in methods pharmaceutical other than reporting bias (though has reporting bias implications as well)

Beware Beware faulty faulty comparators comparators

3

Irbesartan vs Irbesartan vs amlodipine in diabetic amlodipine in diabetic

nephropathynephropathy• in comparison to amlodipine, irbesartan in comparison to amlodipine, irbesartan reduced the combined endpoint of all reduced the combined endpoint of all cause mortality, progression to end cause mortality, progression to end stage renal disease, and doubling of stage renal disease, and doubling of serum creatinine (RRR 20%, 95% CI 7.5% serum creatinine (RRR 20%, 95% CI 7.5% to 32%)to 32%)

• did irbesartan reduce all-cause did irbesartan reduce all-cause mortality?mortality?

-40 -24 -8 8 24 40 56

Risk reduction with irbesartan (vs. amlodipine)

RRR (95% CI)

Composite endpoint

All-cause mortality

End-stage renal disease

Doubling of creatinine concentration

RCT 1715 DM 2 nephropathy, HTN irbesartan vs amlodipine NEJM 2001; 345: 851

RRR 20% (7.5-32%)

RRR 33% (16-47%)

RRR 23% (-0.5-41%)

RRR -3% (-35-22%)

What has gone wrong?What has gone wrong?

• widely varying importancewidely varying importance

• biggest effect on least importantbiggest effect on least important– most important no effectmost important no effect

• criteria for use of compositecriteria for use of composite– similar patient-importancesimilar patient-importance– similar effectsimilar effect

2.7%

UKPDSUKPDSDiabetes-related endpointsSudden death Death from hypoglycemiaDeath from hyperglycemia Fatal MINonfatal MI AnginaHeart failureStroke Renal failureAmputation Vitreous hemorrhageRetinal photocoagulation Blindness in one eyeCataract extraction

After 10 years of treatment with diet + SU or insulin: 35.3%with diet: 38.5%RD: 3.2%NNT for 10 years: 31

SIMILAR PATIENT IMPORTANCE?SIMILAR TREATMENT EFFECT?

1 in 35 reviews note this dominance

Beware composite Beware composite endpointsendpoints

4

5,269 patients with abnormal glucose tolerance test

randomized to lifestyle advice, or advice + rosiglitazone

followed for 3 to 5 years

What is the authors’ message?

• rosiglitazone to prevent diabetes:

• strong indication (for all)

• weak indication (for some)

• not indicated

Doctor, what do I gain Doctor, what do I gain by taking rosiglitazone?by taking rosiglitazone?

• Doc: less chance of diabetesDoc: less chance of diabetes

• Pt: what happens if I get diabetesPt: what happens if I get diabetes

• Doc: you have to take a drugDoc: you have to take a drug

• Pt: the same drug I’m taking to prevent Pt: the same drug I’m taking to prevent diabetes?diabetes?

• Doc: I could give you a drug with less problemsDoc: I could give you a drug with less problems

• Pt: I’ll take a drug every day for 3 years to Pt: I’ll take a drug every day for 3 years to lower my risk of taking the same or a less lower my risk of taking the same or a less toxic drug from 25 to 10%???toxic drug from 25 to 10%???

drug no drug difference

diabetes medication 3 years 30000 3650 26350

new diagnosis of diabetes 1060 2500HR 0.38 (CI, 0.33-

0.44)

Anxiety about diabetes ?? ?? ??

Costs inconvenience self-monitoring

?? ?? ??

Costs and inconvenience HbA1c, lipoprotein testing, retinal exam, etc.

?? ?? ??

MI, stroke, CV death at 3 years

120 90HR 1.39 (CI 0.81-

2.37)

Heart failure, at 3 years 50 10HR 7.03 (CI 1.6-

30.9)

kidney, eyes, neuropathy ?? ?? ??

Peripheral edema, at 3 years 680 490RR 1.4 (CI 1.1-

1.8)

Weight gain (kg), at 3 years + 1.1 -1.1 + 2.2

Rare (fractures, macular edema)

?? ?? ??

What is your view?

• rosiglitazone to prevent diabetes:

• strong indication (for all)

• weak indication (for some)

• not indicated

• what has gone wrong here?

Other problematic Other problematic surrogatessurrogates

• tests of cognitive function instead of tests of cognitive function instead of function and behavior in Alzheimer’sfunction and behavior in Alzheimer’s

• bone density instead of fractures in bone density instead of fractures in osteoporosisosteoporosis

• oxygenation instead of mortality in ARDSoxygenation instead of mortality in ARDS

• asymptomatic DVT instead of symptomatic asymptomatic DVT instead of symptomatic thromboembolism thromboembolism

• pulmonary function instead of exacerbations pulmonary function instead of exacerbations and qol in respiratory diseaseand qol in respiratory disease

• lipids instead of CV events in atherosclerosislipids instead of CV events in atherosclerosis

Another surrogateAnother surrogate• patients with CV diseasepatients with CV disease

• higher HDL associated with lower CV riskhigher HDL associated with lower CV risk– strongly agreestrongly agree– agreeagree– disagreedisagree

• raising HDL should decrease CV riskraising HDL should decrease CV risk– strongly agreestrongly agree– agreeagree– disagreedisagree

Beware substitute Beware substitute endpointsendpoints

5

Five vs Four Courses of Five vs Four Courses of Therapy for Acute Therapy for Acute Myeloid LeukemiaMyeloid Leukemia

Wheatley K, Clayton D. Controlled Clinical Trials 2003;24:66-70

Five vs Four Courses of Five vs Four Courses of Therapy for Acute Therapy for Acute Myeloid LeukemiaMyeloid Leukemia

Wheatley K, Clayton D. Controlled Clinical Trials 2003;24:66-70

Five vs Four Courses of Five vs Four Courses of Therapy for Acute Therapy for Acute Myeloid LeukemiaMyeloid Leukemia

Wheatley K, Clayton D. Controlled Clinical Trials 2003;24:66-70

Five vs Four Courses of Five vs Four Courses of Therapy for Acute Therapy for Acute Myeloid LeukemiaMyeloid Leukemia

Wheatley K, Clayton D. Controlled Clinical Trials 2003;24:66-70

Tifacogin [Recombinant Tifacogin [Recombinant Tissue Factor Pathway Tissue Factor Pathway

Inhibitor (TFPI) in Sepsis]Inhibitor (TFPI) in Sepsis]• BackgroundBackground: :

– multicentre RCT (245 hospitals) multicentre RCT (245 hospitals) comparing TFPI to control in severe comparing TFPI to control in severe sepsissepsis

– major endpoint: 28 day mortalitymajor endpoint: 28 day mortality– 2 interim analyses2 interim analyses

• ResultsResults: : – 1754 patients randomized over 18 1754 patients randomized over 18 monthsmonths

– baseline characteristics similarbaseline characteristics similarAbraham et al. JAMA 2003:290;238-247

Results: 3-Month Moving Results: 3-Month Moving Average MortalityAverage Mortality

Interim analysis at 722 patients:Mortality 38.9% vs 29.1% (p<.006)

Abraham et al. JAMA 2003:290;238-247

Results: 3-Month Moving Results: 3-Month Moving Average MortalityAverage Mortality

Abraham et al. JAMA 2003:290;238-247

True beneficial effect

No effect

Stopping boundary

True beneficial effect

No effect

Stopping boundary

Look after every patient or event

stop

True beneficial effect

No effect

Stopping boundary

Interim analyses every q patients or events

stopstop

stop

Poldermans, NEJM, 1999Poldermans, NEJM, 1999

• elective vascular surgeryelective vascular surgery–positive dobutamine stress echopositive dobutamine stress echo

• compared bisoprolol to placebocompared bisoprolol to placebo–unblindedunblinded

• primary endpoint death or primary endpoint death or nonfatal MInonfatal MI

Poldermans NEJM 1999Poldermans NEJM 1999• planned sample size 266planned sample size 266

• prior planned single look at 100 ptsprior planned single look at 100 pts– stop if exceeded O’Brien-Fleming boundarystop if exceeded O’Brien-Fleming boundary

• p < 0.001p < 0.001

• stopped after 112 patientsstopped after 112 patients

• primary endpointprimary endpoint– 2 of 59 (3.4%) in bisoprolol group2 of 59 (3.4%) in bisoprolol group– 18 of 53 (34%) in placebo 18 of 53 (34%) in placebo

• RR 0.09, 95% CI 0.02 to 0.37, P< 0.001RR 0.09, 95% CI 0.02 to 0.37, P< 0.001

Quality Assessment

Summary of Findings

QualityRelative Effect

(95% CI)

Absolute risk difference

OutcomeNumber of

participants(studies)

Risk of Bias

Consistency Directness PrecisionPublication

bias

Myocardial infarction

10,125(9)

No serious limitations

No serious imitations

No serious limitations

No serious limitations

Not detected

High0.71

(0.57 to 0.86)15/1,000 fewer (7 fewer to 21

fewer)

Mortality10,205

(7)No serious limitations

Possiblly inconsistent

No serious limitations

SeriousImprecision

Not detected

Moderate or low

1.23(0.98 – 1.55)

5/1,00 more(1 fewer

to 13 more)

Stroke10,889

(5)No serious limitaions

No serious limitations

No serious limitations

Possible imprecision

Not detected

Moderate or High

2.21(1.37 – 3.55)

5/1,000 more (2 more to 1.3 more0

Beta blockers in non-cardiac surgery

Beware trials stopped Beware trials stopped early early

6

Users Guides to SpinUsers Guides to Spin

1.1. read methods and results; read methods and results; bypass the discussion sectionbypass the discussion section

2.2. read the abstract reported in read the abstract reported in evidence-based secondary evidence-based secondary publicationspublications

3.3. beware faulty comparatorsbeware faulty comparators

4.4. beware composite endpointsbeware composite endpoints

5.5. beware of substitute beware of substitute endpointsendpoints

6.6. beware trials stopped earlybeware trials stopped early

Wall St. Journal, November, 2006

Wall St. Journal, November, 2006

New York Times, May 9, 2007

Available at: www.mhprofessional.com/jama