using primary care databases to evaluate drug benefits and harms: are the results replicable and...
DESCRIPTION
Databases of electronic medical records and in particular primary care databases (PCDs) are increasingly used in research. The largest PCDs contain full data on all primary care consultations by millions of patients over two or more decades. They provide a means for investigating important healthcare questions which cannot be practically addressed in a Randomised Controlled Trial. However, concerns remain about the validity of studies based on data from PCDs. Most work around validity has attempted to confirm individual data values within a dataset. We take a different approach and instead replicate published PCD studies in a second, independent, PCD. Agreement of results then implies that the conclusions drawn are independent of the data source (though this doesn’t rule out that such as confounding by indication are commonly influencing both). We replicated two previous PCD studies using the Clinical Practice Research Datalink (CPRD). The first was a retrospective cohort study of the effect of Beta-blocker therapy on survival in cancer patients using DIN-LINK. The second was a nested case-control analysis of the effects of Statins on mortality of patients with ischaemic heart disease using QRESEARCH. Our analyses produced several important quantitative differences compared to the original studies, altering conclusions. These could not be fully explained by either demographic differences in the patient samples or structural differences between the datasets. Our study highlights both the caution that needs to be applied when assessing the findings from analysis of just a single database and the difficulties in performing replications of existing PCD studies.TRANSCRIPT
Using primary care databases toevaluate drug benefits and harms:are the results valid andreplicable?
David A. Springate, University of ManchesterCentres for Primary Care/Biostatistics
Outline
1. Primary Care Database (PCD) study validity
2. PCD Replications
– Statins and Ischaemic heart disease– β-blockers and Cancer
3. Lessons to be learned
PCD studies are all the rage. . .Number of UK PCDpublications is rapidlyincreasing
1990 1995 2000 2005 2010
050
100
150
PCD articles in pubmed
year
Num
ber
of a
rtic
les
There is global interest in UK PCDresearch
Institutions affiliated with UK PCD publications
xx
x x
xxxxxxx
x x x
x
xx
x
x
x
xxx
xx
xxxx xxxxx
xxx
xxxxxx
xxxx
x
xxx
x
x xx xx
x
xx
x
xxxxxxx
xxxx x
xxx
x xxx
x
xxxxx
xxxxxx
x
xx
xxx
xxxxx
x xxx
x
xxx
xxx
x
x
xx
xx
x
x
x
xx xx
x
xxx
x x
xx
xx
xx
xx
xx x
xx
xxxxx
xx
xxx
x
xx
x
xxx
x xx
xx
xxx
x
xx
x
xxxxxx x
x
xxx
xxxxxx
xxx
x
x
xx
xxxxxxxx
x
xxxxxxxx
xxxx
xx
x
x
xx x
xxxx
x
xxxx
x xxxxxx
xxxxxx
xxxxx
x
x
x
xxxx
xxxxx
xx
xxx
xx
xx
x
x
x
xxx
xxxxxxxx x
x
x
xx
x
x
x
x
xx
xxx
x x
xx
x
xxx
x
x
xxxx
xxxxxx
x
x xxxxx x
xxxxx
xxxx
x
x
xxxxxxxx xxx
x
xxxxx
x
x xx xxx xx
xx xxxxx
x
xxxxx
x
xxxxxxx
xxx
x
xxx
x
xx
xx
x
x
xxxx
xxx
xxxxx
xxxxx
xx
x
xx x
xxx
xxxxxxx
x
x
xxxx
x xxxxx
x
xx
xx
x
x
x
x
x
x
x
x xxxxxx xxxx xx
x
xxxx
xx
x
x xx x
xx
x
xx
x
x
xxx
xx
xxx
xxxxx
x
x
x
xx
xxx x
xx
xxx
xxxx
xxxx
xx
x
x
xxxx
x
xx
xxxx
x xxxx
xx
xxxxxxxxxxx
xxxx
x
xxxx
xx
x
xxxx xx
xx x
x
x
x xx
x
xxx
x
xx
xxxxxx xxx
xx
x
x
x
xxxx
xx
xxxxx
xx
xxx
xxx
xxx
xx x
x
xxxxx
x
xx xxxx
xxxxxxx
x
x
x
xx
x
x
xx
x
xxx
xx
x
xxxxxx x
xx
xxx
xx
xxxxxxxxx
x
x
xxxxxx xx
x
x x x
xxxx
xx
xx
xxx
xxx
x xx
xxx
xxx
x
x
xxx
xxx
x
x
x
x
xxxxx
x
x x
xxxxxxxxxxxxxxx x
xxxx
xxx
xxxx x
x
x
xxxx
xx
x
xx
xxxx
xx
x xx
xxx
xxx
x
x
x xxxxxxx xxxx
x
xxxx
x
x
xxx
x xx
xxxxx
x
x
x
x
x
x
x
xxxxxxxxx
x
xxx
xxx
xxx
x
xx xxxxxx
x
xxxxxxxx xxxxx
x
xxxx
xxxxxxxx
x
x
xx
xxx
x
x xxx
xxxx
xx
xx
x
x
x
xxx
x x
xxxx
x
x
xxxx
xx
xxxx
x
xxx x
x
xxx x
x
x x
xx
xx
xxx
xxx
xxxxxxxxx x
xxx
xxx
xxxxxxx
xx
xxxx
xxxx
xxxx
xx
x xxx
x
xxx
x
xx
xx
x
xx
x xx
x xxx x
xxxx x
xx
xx
x
x
x
xxxxxxx
xxx
xxx
xx
x
x
xx
xx
xx
x
x
xx
x
xxxxxxx
x
xxxxxxxxxxx xxxxxxxxxxxxxx xx
xxx
xxx
xxxxxxxxxxxxxxxxxxxxxx xxxxx
Uses of Primary care databases. . .
• Prevalence / incidence studies
• Associations between conditions
• Harms and risks of treatments
• Comparative effectiveness
• RCT comparisions / replications (and replacements?)
BUT, There are still concerns about thevalidity of PCD-based studies. . .
Threat Refs
Data quality Herrett 2009,
Khan 2010, Jordan 2004
Data completeness Marston 2010, Delaney 2007,
Collins 2010
Confounding Tannen 2008, Lewis 2007
Clinical coding www.ClinicalCodes.org
PCD Replications“Non-reproducible singleoccurrences are of nosignificance to science.”
—– Karl Popper (1959)
An approach to validity that askswhether flaws and differences inthe data make any difference tothe ultimate conclusions ratherthan looking at validity andcompleteness of the underlyingindividual data
http://xkcd.com/242
Replicating studies in another, independentPCD
• Agreement implies that conclusions are not dependenton data source
• Some factors could influence both (such asconfounding by indication)
• First completely independent PCD replication (but seeVinogradova (BMJ 2013): relationship betweenbisphosphate exposure and cancer in QResearch andCPRD)
Replications were performed in CPRD
• Largest UK primary care database (CPRD-GOLD)
• ˜ 14 million patients
• ˜ 650 practices across the UK
• Uses the Vision GP computer system
Criteria for replication
1. Effectiveness studies
2. Different GP computer system from CPRD (notVision)
3. No practice overlap with CPRD
4. Representative coverage
5. Primary Care Database (Not integratedprimary/secondary/pharmacy)
PCD replications
Hippisley-Cox and Coupland (2006) Effect of statins on themortality of patients with ischaemic heart disease: populationbased cohort study with nested case-control analysis. Heart92:752-758 (QResearch)
Shah, Carey et al. (2011) Does β-adrenoceptor blockertherapy improve cancer survival? Findings from apopulation-based retrospective cohort study
Br J Clin Pharmacol 72:157-161 (DIN-LINK)
We then compare
• summary statistics
• mortality rates
• model coefficients and standard errors
Effect of statins on the mortality of patientswith ischaemic heart disease (QResearch)
Objective To measure the effect of statins on mortality forcommunity based patients with IHD
Design Cohort survival analysis and nested case-control
Setting 1.18 million patients in 89 practices
Subjects Patients with first diagnosis of IHD betweenJanuary 1996 and December 2003
Outcomes - Cohort: Adjusted hazard ratios (+/-95%CI) for all-cause mortality
- Case-control: Odds ratio (+/-95%CI) forcurrent use of statins, previous use andduration of use
Effect of statins on the mortality of patientswith ischaemic heart diseaseSummary statistics
Measure Analysis CPRD Qresearch
Number of practices Cohort 661 89Number of patients Cohort 91589 13029
Cases 15591 2266Controls 62356 9064
Median age Cases 80 80Controls 79 80
Percent female Cases 45.5% 44.3%Controls 45.5% 44.3%
Median followup (months) Cases 22.1 20.3Controls 22.5 21
Percent on statins Cases 17% 19.6%Controls 23.6 25.4%
Mortality rates for patients with IHD
Age Comorbidity Sex
0
100
200
300
400
Tota
l0−
44
45−5
4
55−6
4
65−7
4
75−8
4
85−9
4
95_p
lus
No dia
bete
s
Diabet
es
No hy
perte
nsion
Hyper
tens
ion
No CCF
CCF
Fem
aleM
ale
Patient group
Mor
talit
y ra
te (
per
1000
per
son
year
s)
PCD
CPRD
QResearch
Survival analyses for patients on Statins
QResearch
Adjusted HR 0.47 (0.41 to 0.53)
CPRD
0 2 4 6 8
Time since diagnosis of IHD (Years)
Sur
viva
l
0.00
0.25
0.50
0.75
1.00
Adjusted HR 0.43 (0.40 to 0.46)
Case-control analysis: Odds ratios for effectsof Statins on mortality in IHD patients
Odds are relative topatients not on statins.Dotted line represents1:1 odds
Previously on statins Currently on statins
0.0
0.5
1.0
1.5
2.0
All Sta
tins
Atorv
asta
tin
Ceriva
statin
Fluvas
tatin
Prava
statin
Simva
statin
All Sta
tins
Atorv
asta
tin
Ceriva
statin
Fluvas
tatin
Prava
statin
Simva
statin
Statin type
Odd
s ra
tio PCD
CPRD
Qresearch
Misleading pooled odds ratios
Combining effects of e.g. two drugs of the same BNFchapter
Group Y N Odds ratio
Drug 1 Cases 600 2003/3 = 1
Drug 1 Controls 75 25
Drug 2 Cases 10 300.333/1 = 0.333
Drug 2 Controls 30 30
Pooled Cases 610 2302.65/1.91 = 1.39
Pooled Controls 105 55
Case-control analysis: Adjusted OR forduration of use of statins on survival
Odds are relative topatients not on statins.Dotted line represents1:1 odds
0.00
0.25
0.50
0.75
1.00
0−12
13−2
4
25−3
6
37−4
8
49−6
0>6
0
Duration (months)
Adj
uste
d O
dds
ratio
PCD
CPRD
Qresearch
Summary — Statins study
1. Strikingly similar results in the two studies, despitedifferent GP computer systems (Vision vs EMIS)
2. As expected, narrower confidence intervals due tolarger study
3. Original study was well designed (Matching,appropriate analyses etc.)
4. Given the results, pooling of “all statins” isquestionable
Does β-adrenoceptor blocker therapy improvecancer survival?(DIN-LINK)
Objective To examine the effect of β-blocker treatment oncancer survival
Design Survival analyses for 9 cancer types
Setting 3462 cancer patients on β-blocker or otherantihypertensive therapy
Subjects Patients 40-85 with first cancer diagnosisbetween 1997 and 2006
Outcomes - Adjusted hazard ratios (+/- 95%CI) forall-cause mortality in each cancer type
- Pooled hazard ratio (random effects)
Comparison of summary statisticsCohort size:
CPRD 11316
DIN-LINK 3462
BPLM = Blood pressure
lowering medicines
Deaths Alive at 1 year Diagnosed 1997−2001 Diagnosed 2002−2006
0
25
50
75
100
Beta−
block
ers
Other
BPLM
Beta−
block
ers
Other
BPLM
Beta−
block
ers
Other
BPLM
Beta−
block
ers
Other
BPLM
Drug combination
Per
cent
age
of p
atie
nts
PCD
CPRD
DIN−LINK
Comparison of patient samples by cancer site
BPLM = Blood pressure
lowering medicines
0
10
20
30
40
brea
st
colon lun
g
oeso
phag
us
ovar
ian
panc
reas
pros
tate
rena
l
stom
ach
Cancer site
Per
cent
age
of p
atie
nts
PCD
CPRD
DIN−LINK
Survival analyses: comparison of hazard ratios
Cancer typebreastcolonlungoesophagusovarianpancreasprostaterenalstomach
Overall
Hazard ratio1.09 (0.8, 1.49)1 (0.77, 1.3)1.12 (0.89, 1.41)1.05 (0.69, 1.6)1.14 (0.63, 2.06)1.88 (1.09, 3.25)1.54 (1.13, 2.09)1.14 (0.52, 2.52)1.44 (0.76, 2.74)
1.18 (1.04, 1.33)
% weight14.8520.9325.78.174.274.9115.192.373.61
100
0 1 2 3 4
all Beta−blockers vs controls DIN−LINK
Cancer typebreastcolonlungoesophagusovarianpancreasprostaterenalstomach
Overall
Hazard ratio1.19 (1.03, 1.37)0.85 (0.74, 0.97)1.04 (0.91, 1.19)1.27 (1.01, 1.59)1.05 (0.74, 1.5)0.94 (0.74, 1.21)1.03 (0.92, 1.15)0.46 (0.26, 0.83)1.03 (0.78, 1.36)
1.01 (0.91, 1.13)
% weight14.8715.1115.3610.76.329.7616.272.898.71
100
0 1 2 3 4
all Beta−blockers vs controlsCPRD
Summary — β-blocker study
1. Different individual cancer HR’s and overallconclusions
2. Important differences in some cohort statistics
3. Differences remain after correcting to give the samepatient:practice
4. Differences remain after reducing the size of theCPRD study
5. Databases appear to be demographically similar (Careyet al. 2004)
WHY?
Summary — β-blocker study
1. Different individual cancer HR’s and overallconclusions
2. Important differences in some cohort statistics
3. Differences remain after correcting to give the samepatient:practice
4. Differences remain after reducing the size of theCPRD study
5. Databases appear to be demographically similar (Careyet al. 2004)
WHY?
”an experiment is reproducable until anotherlaboratory tries to repeat it.” — Alexander Kohn
• Artifact of differences in GP computer systems?DIN-LINK uses Torex/iSoft systems (SeeKontopantelis et al 2013)
• Analysis methods?
– No matching - potential confounding– No clustering by practice– Limited control for covariates– Is meta-analysis the most appropriate method
(Assumes independence)?
• Data quality?
”an experiment is reproducable until anotherlaboratory tries to repeat it.” — Alexander Kohn
• Artifact of differences in GP computer systems?DIN-LINK uses Torex/iSoft systems (SeeKontopantelis et al 2013)
• Analysis methods?
– No matching - potential confounding– No clustering by practice– Limited control for covariates– Is meta-analysis the most appropriate method
(Assumes independence)?
• Data quality?
”an experiment is reproducable until anotherlaboratory tries to repeat it.” — Alexander Kohn
• Artifact of differences in GP computer systems?DIN-LINK uses Torex/iSoft systems (SeeKontopantelis et al 2013)
• Analysis methods?
– No matching - potential confounding– No clustering by practice– Limited control for covariates– Is meta-analysis the most appropriate method
(Assumes independence)?
• Data quality?
Conclusions / recommendations
These replications add to the evidencethat PCD results are valid
• PCD Replication is hard!
– Methods details are inadequate forreplication
– Clinical codes not provided withthe original article
– Relies on active cooperation ofauthors of original studies
– Even then, ambiguity can remain
• Publish full methods (in onlineappendix?)
• Publish full clinical code lists...
ClinicalCodes.org
Conclusions / recommendations
These replications add to the evidencethat PCD results are valid
• PCD Replication is hard!
– Methods details are inadequate forreplication
– Clinical codes not provided withthe original article
– Relies on active cooperation ofauthors of original studies
– Even then, ambiguity can remain
• Publish full methods (in onlineappendix?)
• Publish full clinical code lists...
ClinicalCodes.org
Conclusions / recommendations
These replications add to the evidencethat PCD results are valid
• PCD Replication is hard!
– Methods details are inadequate forreplication
– Clinical codes not provided withthe original article
– Relies on active cooperation ofauthors of original studies
– Even then, ambiguity can remain
• Publish full methods (in onlineappendix?)
• Publish full clinical code lists...
ClinicalCodes.org
Conclusions / recommendations
These replications add to the evidencethat PCD results are valid
• PCD Replication is hard!
– Methods details are inadequate forreplication
– Clinical codes not provided withthe original article
– Relies on active cooperation ofauthors of original studies
– Even then, ambiguity can remain
• Publish full methods (in onlineappendix?)
• Publish full clinical code lists...
ClinicalCodes.org
Thanks. . .
Research team
David ReevesEvan KontopantelisIvan OlierDarren Ashcroft
Authors of the originalstudies
Iain Carey (St. GeorgesUniversity, London)Carol Coupland(University ofNottingham)
Contact: [email protected]