Systematic reviews of clinical versus animal studies:

Parallels and challenges

Maroeska M. Rovers

Acknowledgments: Carlijn Hooijmans, Malcolm Macleod, Michael Bracken

•  Archie Cochrane in 1979: "It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials”

•  1993: founding of Cochrane Collaboration • Mission: to enable people to make well-informed decisions about healthcare

My background

Why so important?

The growth in RCTs

0

5000

10000

15000

20000

25000

30000

35000

1950 1960 1970 1980 1990 2000

Year of Publication

Num

ber o

f Tria

ls /

Yea

r .

MEDLINE

Cochrane CCTR

Multicentre Trials

Systematic review in detail

Research question

Extensive literature search

Objective selection process

Critical appraisal

Objective data extraction

Synthesis of results

Structured reporting

How close is Cochrane’s dream?

How many reviews in Cochrane? 3,800 reviews with 6 trials each = 22,800 How many reviews are needed? A. Trials: Number of RCTs over 500,000

So 5% included so far B. Questions: Over 11,000 diseases:

If 2 treatments each; 22,000 “questions” So 17% of questions

What about animal studies

How many systematic reviews on animal studies? 248 reviews > 7 million animal studies

“Much animal research into potential treatments for humans is wasted because it is poorly conducted and not evaluated in SRs”

Potential sources of bias in animal studies

Bias Solution Selection bias Randomisation Performance bias Allocation concealment Detection bias Blinded outcome assessment Attrition bias ITT/ reporting drop outs

Six interventions compared: 3 similar outcomes 3 discordant results

Quality of the studies included in the SR

BMJ, 2007

Consequences risk of bias

Macleod et al. Stroke, 2008

Methodological problems

1.  Heterogeneity in animal species and strains used 2.  Different models of inducing illness with varying

similarity to the human condition 3.  Variation in dosis/intervention schedule 4.  Variability in how animals are selected 5.  Loss to follow-up not reported 6.  Small groups with inadequate power

Methodological problems

7.  Flawed statistical analyses (not taking into account confounding, no ITT)

8.  Variety of outcomes (with uncertain relevance to human condition)

9.  Length of follow-up may not correspond to disease latency inhumans

10. External validity often nog thought of

Other challenges….

Plos. Med. 2005

Solutions: •  Better powered evidence

àLarger studies or low bias meta-analyses •  Registrations of studies

And remember…

A systematic review does NOT replace a good quality study

Future directions

•  Methodology •  Search strategies •  Analyses •  Individual animal data?

•  Improving experimental designs •  Real sample size calculation •  Demanding highest quality standards in conducting and

reporting •  Multicenter animal studies?

•  Registry of animal studies