leite, a; andrews, nj; thomas, sl (2016) near real...

Leite, A; Andrews,NJ; Thomas,SL (2016) Near real-time vaccinesafety surveillance using electronic health records-a systematic re-view of the application ofstatisticalmethods. Pharmacoepidemi-ology and drug safety,25 (3). pp. 225-37. ISSN 1053-8569 DOI:https://doi.org/10.1002/pds.3966

Downloaded from:http://researchonline.lshtm.ac.uk/2532230/

DOI: 10.1002/pds.3966

Usage Guidelines

Please refer to usage guidelinesat http://researchonline.lshtm.ac.uk/policies.htmlor alterna-tively contact [email protected].

Available under license:http://creativecommons.org/licenses/by/2.5/

REVIEW

Near real-time vaccine safety surveillance using electronic healthrecords—a systematic review of the application of statistical methods†

Andreia Leite1*, Nick J. Andrews2 and Sara L. Thomas1

1Faculty of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, London, UK2Statistics, Modelling and Economics Department, Public Health England, London, UK

ABSTRACTPurpose Pre-licensure studies have limited ability to detect rare adverse events (AEs) to vaccines, requiring timely post-licensure studies.With the increasing availability of electronic health records (EHR) near real-time vaccine safety surveillance using these data has emerged asan option. We reviewed methods currently used to inform development of similar systems for countries considering their introduction.Methods Medline, EMBASE and Web of Science were searched, with additional searches of conference abstract books. Questionnaireswere sent to organizations worldwide to ascertain unpublished studies. Eligible studies used EHR and regularly assessed pre-specified AEto vaccine(s). Key features of studies were compared descriptively.Results From 2779 studies,31 were included from the USA (23),UK (6), and Taiwan and New Zealand (1 each).These werepublished/conducted between May 2005 and April2015.Thirty-eightdifferentvaccines were studied,focusing mainly on influenza(47.4%), especially 2009 H1N1 vaccines. Forty-six analytic approaches were used, reflecting frequency of EHR updates and the AE studied.Poisson-based maximized sequential probability ratio test was the most common (43.5%), followed by its binomial (23.9%) and conditionalversions (10.9%). Thirty-seven of 49 analyses (75.5%) mentioned control for confounding, using an adjusted expected rate (51.4% of thoseadjusting), stratification (16.2%) or a combination of a self-controlled design and stratification (13.5%). Guillain-Barré syndrome (11.9%),meningitis/encephalitis/myelitis (11.9%) and seizures (10.8%) were studied most often.ConclusionsNear real-time vaccine safety surveillance using EHR has developed over the past decade but is not yet widely used. As morecountries have access to EHR, it will be important that appropriate methods are selected, considering the data available and AE of interest.© 2016 The Authors. Pharmacoepidemiology and Drug Safety Published by John Wiley & Sons Ltd.

key words—electronic health records; safety; sequential tests; statistical process control; vaccines; pharmacoepidemiology

Received 19 June 2015; Revised 16 December 2015; Accepted 17 December 2015

INTRODUCTIONVaccines are considered to be one of the mostcost-effective interventionsin public health.1,2 As withother drugs,vaccines are nottotally safe,3 butsafetyrequirementsareparticularly high asvaccinesaregiven to healthy individuals,mostoften children.4

All vaccines go through extensive safety assessmentbefore licensure;however,pre-licensure studies havelimited ability to detectrare adverse events (AEs) tovaccines (with frequency <1/10 000-1/100 000)5, AE

occurring among specific sub-populations who werenot included in clinical trials,and long-term AE.6 Toovercome these limitations, timely post-licensure stud-ies are required.These can be broadly divided intopassive (spontaneous reports) and active studies andshould be followed by confirmatory epidemiologicstudies. While spontaneous reporting of AE is widelyimplemented worldwideas a simpleand low-costmethod,usefulto detectnew,unanticipated AE,ithas limitations.2 These include difficulties in denomi-nator calculation, potential reporting biases (e.g. over-reporting ofpotentialAE receiving extensive mediacoverage) and incomplete reporting. In contrast, activesurveillance triesto identify allthose experiencing(or atleastseeking medicalattention for) a potentialAE to vaccines.This approach includesanalysesof large population datasets (using electronic healthrecords (EHR)),targeted hospital-based surveillance

*Correspondence to: A. Leite, Department of Infectious Disease Epidemiology,London School of Hygiene & Tropical Medicine,Keppel Street,WC1E 7HT,London, UK. E-mail: [email protected]†Prior postings and presentations statement: This work has not been submittedor accepted elsewhere.Preliminary results have been presented atthe NIHRHealth Protection Research Uniton Immunisation annualmeeting in March2015 and have been presented as a poster presentation to the 31st InternationalConference on Pharmacoepidemiology & Therapeutic Risk Management.

© 2016 The Authors. Pharmacoepidemiology and Drug Safety Published by John Wiley & Sons Ltd.

pharmacoepidemiology and drug safety 2016; 25: 225–237Published online 28 January 2016 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/pds.3966

This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distributionand reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.

(where trained health workersdaily seek potentialcases of conditions of interest) and recruitment of vac-cinated cohorts for detection of AE (using face-to-faceinterviews,phone interviews,short-message servicesor web-based tools).7,8With the increased availabilityof large population datasets,nearreal-time vaccinesafety surveillance(NRTVSS) hasemerged asanoption.9

Near real-timevaccinesafetysurveillance,alsoknown as rapid cycle analysis,involves regular inter-rogation ofEHR to investigate pre-specified AE tovaccines.2 By testing these AE on a regular basis afterintroduction of a new vaccine, these methods ensure atimely detection of possible safety problems.10When asignal is detected by this approach, it needs to be fur-ther analysed, including a signal refinement stage andeventualconfirmatory analyses.These stepsshouldbe predetermined and willlead to the decision ofwhether to validate or invalidate the signal. NRTVSSis thus partof a systematic approach to signaldetec-tion, with a dual role of signalling possible AE to vac-cines and reassuring authorities and populations thatevents are being monitored.11 For a given vaccine,NRTVSS only considers a small number of suspectedAE (e.g. 5 to 10); complementary information is pro-vided by existingmethodssuch as spontaneousreports.12

The growing use of NRTVSS methods,along withthe increasing availability of EHR, highlights the needto review studies using this approach.Such a reviewcan provide crucialinformation on the developmentof systems for vaccine safety surveillance for countriesconsidering their introduction.

OBJECTIVEThe aim of this study was to carry out a systematic re-view of publishedand unpublisheddataon themethods used for NRTVSS using EHR.

METHODSStudies were included in the review if they (i) usedroutinely collected health data (atleastfor the ex-pected numberof events);(ii) studied pre-specifiedoutcome(s) to assess the safety of one or more vac-cines; and (iii) regularlytestedthe outcomes.Studies(i) includingonly informationbasedonspontaneous reporting systems,(ii) aimed attestinghypothesis/confirming previously generated/suspectedsignals or (iii) aimed atdeveloping new methods forNRTVSS (unlessa specific application ofthe new

method was given) were excluded. No limits were im-posed in terms of language or year.

Medline and EMBASE were searched forstudiespublished until6 January 2015,using a combinationof thesaurus and free-text terms (search strategy is pro-vided in Supporting Information Appendix A).Titlesand abstracts were reviewed to determine eligibilitystatus,followed by the fulltextfor those consideredpotentially eligible.References from the papers col-lected were also reviewed. Reviews of the topic wereselected forreference mining.A. L. was responsiblefor evaluating eligibility of the identified studies.Toensure quality, eligibility of a random sample of 10%of the results was evaluated by S. T.and N. A.Wheneligibility was unclear, the study was discussed amongthe authors until a consensus was reached.

To complementthe database searches,a citationsearch was conducted. To the best of our knowledge,the methods under study were first applied to the fieldof vaccinesafety by theVaccineSafety Datalink(VSD). Two key VSD papers that describe the testingand implementation of rapid cycle analysis using rou-tinely collected health data were selected to perform acitation search.9,13

The same search strategy was used in the Web ofScience Core Collection to cover meetings and confer-ences,restricting the search to meeting abstractsorproceedings papers.Also,the AnnualConference onVaccine Research and the Vaccine and ISV Congressabstract book and programme, respectively, were analysed(Supporting Information Appendix B).The BrightonCollaboration newsletter was also searched as a potentialsource of relevant new studies or contacts.14

A second stage ofthe review included contactingexperts in vaccine safety, as follows:

• Specialists in vaccine safety (from the Global Advi-sory Committee on Vaccine Safety (GACVS),15

Brighton Collaboration16 and Accelerated Devel-opmentof Vaccine benefit–risk collaboration inEurope (ADVANCE)17) were asked if they wereaware ofwork being conducted in the area andfulfilling our inclusion criteria.

• Authors with known work using routinely collecteddata and the potential to have implemented/conductedeligiblestudieswere contacted(MedicinesandHealthcare products Regulatory Agency (MHRA),18

VSD19 and Statens Serum Institute20). Further con-tacts were also asked for at this stage.

• Finally, authors with a previous published work butincompleteinformation,and thosesuggested byotherexperts,were contacted to ask forfurtherinformation to characterize the methods.

a. leite et al.226

© 2016 The Authors. Pharmacoepidemiology and Drug SafetyPublished by John Wiley & Sons Ltd.

Pharmacoepidemiology and Drug Safety, 2016; 25: 225–237DOI: 10.1002/pds

An online questionnaire was used to capture infor-mation on studies conducted (Supporting InformationAppendix C).When othersourcesof information(e.g.reports)were available and shared by the con-tactsthesewereused.Expertcontactstook placefrom February to March 2015.

The information identified wasextracted using astandardized extraction form.Data extracted includedtimeline, country/institutions where the study was con-ducted,vaccines studied,study population,outcomesassessed and their method of ascertainment,methodsused to perform the analyses, frequency of assessment,confounding,data-accruallag (i.e.delays in the dataavailable to perform surveillance,which may affectthe results), assessment of the validity of the outcomesof interest (e.g.chart review) and main results.A de-scriptivesummary ofcountry/institution,vaccines,outcomes studied,confounding and data-accruallaghandling was drawn up.

RESULTSA totalof 29 reports were included for data extrac-tion (includinginformationprovidedby expertcontacts),9,13,21–45representing31 studies/systems(Figure 1). A brief description of the studies/systems in-cluded by country,methods used and adjustmentforconfounding strategies is given in Table 1.A detailedcharacterization of the studies is provided in SupportingInformation Appendix D.

Nearreal-timevaccinesafety surveillanceusingEHRs was first reported by Davis et al. in 2005, whena retrospectivestudyassessingthe feasibilityofimplementing such methods was published. Since thistime,we identified a further13 studies conductedby the VSD and 17 other studies in three countries(Figure 2).The firststudy conducted outside theVSD was conducted in New Zealand and publishedin 2007.The reportfrom the laststudy includedwas published online in 2015.Fourstudies (allinthe USA) were conducted completely orpartiallyin a retrospective manner,to testthe feasibility ofimplementing this kind ofsystem (Table 1).Twoof these studies attempted to replicate known sig-nals (rotavirusvaccineand intussusceptionandacellulardiphtheria-tetanus-pertussis (DTaP)/wholecelldiphtheria-tetanus-pertussis vaccine and febrileseizures).Of the prospectivestudies,mostwereconducted in the USA (n = 20),with studiesalsoconductedin the UK (n = 6), and Taiwan andNew Zealand(n = 1for each).The prospectivestudies looked mainly at influenza vaccines (n = 16), es-pecially the 2009 H1N1 pandemic influenza vaccine

(n = 7). Rotavirus (n = 5), DTaP-based (n = 3) and humanpapillomavirus vaccines (n = 3) also received attention.

The outcomes studied were most often neurological(58.5%). Looking at specific outcomes, Guillain-Barrésyndrome (GBS) (11.9% of studied known outcomes),meningitis/encephalitis/myelitis (11.9%) and seizures(10.8%) were the most often included. Outcome ascer-tainmentfor the near real-time analysis was,in mostcases, based on automated data (with no a priori con-firmation of the diagnosis). In these cases, chart reviewand confirmation were used whenever a potential AEwas signalled.Only two studies performed this kindof confirmation forthe nearreal-time analysis,21,35

and one compared the analysis considering the chart-reviewed and non-reviewed outcome for GBS.33Fromthe outcomes studied,11 signals were identified,butonly threeconfirmed (measles-mumps-rubella-vari-cella combinationvaccineand febrileseizures,27

2010–2011trivalentinactivatedinfluenzavaccineand febrile seizures,37 and monovalentrotavirus vac-cine and intussusception41).

Table 2 summarizes the methods used by the stud-ies included in this review. These can be broadly di-vided into continuoussequentialtesting,whichallows examination ofthe data as often as desired(n = 25),9,13,22–34,37,38,40–43,45group sequential testing(n = 4)35,36,38,39and statisticalprocess control(SPC;n = 3).21,44The choice ofthe group of methods hasbeen determined by the frequency of updates to theEHR data used (Table 2).

When considering specificversionsof the testsavailable, the choice has been guided by the increasingavailability of new methods and knowledge of thesemethods over time,as shown in Figure 2,as wellasthe frequency of AE studied.In VSD, the sequentialprobability ratio test(SPRT) was firstapplied9 beingsubsequentlyreplacedby its maximizedversion(MaxSPRT) with the advantage of not having to spec-ify a singlealternativehypothesis.13 The use ofMaxSPRT and its variations also evolved over time.While in the beginning the Poisson and binomial ver-sionswere simultaneously used forthe same out-come,13 from 2010,a targeted selection ofthe testversion and its extensions,based on the strengths ofeach method (Table 2) and the characteristics of theoutcomeunderstudy,was preferred.24,33,34,42,43Inparticular,Poisson-basedMaxSPRT (PMaxSPRT)has been used when less than 50 events were antici-pated and the conditional version when the ratio of ob-served historicaleventsto upperlimit was ≤2.5.Outside VSD, a pattern in the use of continuous sequen-tial methods was less clear. Overall, these tests were themost often employed—PMaxSPRT(45.7%),10,50

near real-time vaccine safety surveillance 227



followed by the binomial(BMaxSPRT—23.9%)10,50

and conditional (10.9%) versions.51

More recently,fourstudies used group sequentialtesting.Two of theseused an alpha-spending ap-proach,38,39(a function controlling how much of thealpha willbe ‘spent’every time a new analysis isrun52), one the Updating Sequential Probability RatioTest53 and otherthe Abt’s modification ofSPRT.54

An alpha-spending approach was thus preferred overthe two othertests employed in a group sequentialway. Both the Pocock-type and O’Brien–Fleming-typefunctions have been used.12,55The remaining methodsdid notfollow a clearevolution and include use ofSPC56at different times by two non-USA institutions

(New Zealand Ministry of Health,Health ProtectionScotland).21,44

Thirty-seven of 49 analyses (75.5%) mentioned con-trol for confounding.Strategieschosen were oftendesign-based and included (alone or in combination)the following: (i) using a self-controlled design, whichautomatically addressestime-invariantconfounders;(ii) matching baseline confounders, through a concur-rentcomparatordesign;(iii) adjusting the expectedrateobtained from ahistoricalcomparison groupbased on the confounders’distribution in the studycohort (iv) stratifying the results according to relevantconfounder categories.Analyses adjusting for poten-tial confounders used mainly an expected rate adjusted

Figure 1. Flowchart of included studies. Studies were excluded for (i) not considering vaccines (nonvaccine), (ii) not analysing the safety of a vaccine (notsafety), (iii) considering safety issues but not applying the methods of interest (other safety), (iv) only developing new methods (methods only) and (v) havingno abstract available (not available)

a. leite et al.228



Tabl

e 1.

Inclu

ded

stud

ies a

ccor

ding

to th

e co

untry

, met

hods

use

d an

d co

ntro

l for

con

foun

ding

stra

tegi

es (s

ee S

uppo

rting

Info

rmat

ion

Appe

ndix

D fo

r fur

ther

det

ails)

Stud

yCo

untry

,or

gani

zatio

nM

etho

dCo

nfou

ndin

gDa

ta‐a

ccru

al la

g or

unde

rrepo

rting

adj

ustm

ent

Retro

spec

tive

Davi

s9US

A, V

SDSP

RTRi

sk a

djus

tmen

t* (s

ite, a

ge, t

ime,

seas

on, s

ex)

Retro

spec

tive

Lieu13

†US

A, V

SDPM

axSP

RTUn

clear

Retro

spec

tive

Brow

n22US

A, i3

Dru

gSa

fety

PMax

SPRT

Expe

cted

cou

nts

(sex

, age

, reg

ion,

mon

th,

conc

omita

nt v

accin

atio

n)Re

trosp

ectiv

e; d

ata

lags

ass

esse

d du

ring

the

stud

yGr

eene24

†US

A, V

SDPM

axSP

RTEx

pect

ed ra

tes (

age

and

site)

Retro

spec

tive—

data

ass

umed

to a

ccru

ewi

thou

t del

ayBM

axSP

RT‡SC

; stra

tifica

tion

(age

, sea

son)

Pros

pect

ive

Lieu13

USA,

VSD

PMax

SPRT

No a

djus

tmen

tAn

alys

es w

aite

d at

leas

t 6 w

eeks

from

the

vacc

inat

ion

or p

reve

ntiv

e vi

sitBM

axSP

RTM

atch

ing

(age

, wee

k, si

te)

McN

ichol

as21Ne

wZe

alan

d,M

oH

SPC

Stra

tifica

tion

(age

)Da

ily re

view

of d

atab

ases

, med

ical

char

ts, d

ischa

rge

lette

rs a

nd la

bora

tory

reco

rds

Yih23

USA,

VSD

PMax

SPRT

Expe

cted

cou

nts

(GBS

/sei

zure

s—ag

e;ot

her A

E—ag

e, se

x)An

alys

is st

arte

d at

leas

t 8 w

eeks

from

the

date

of v

accin

atio

n46

and

redo

neat

the

end

of th

e st

udy

Belo

ngia25

†US

A, V

SDPM

axSP

RTEx

pect

ed ra

tes (

intu

ssus

cept

ion—

trend

,ag

e, si

te b

y Po

isson

regr

essio

n; o

ther

AE—

site)

Anal

ysis

star

ted

at le

ast 8

wee

ks fr

omth

e da

te o

f vac

cinat

ion

46

Brya

n28UK

, MHR

APM

axSP

RTEx

pect

ed ra

tes (

age

and

gend

er)

Adju

sted

for u

nder

repo

rting

(yel

low-

card

dat

a)Hu

ang30

†Ta

iwan

, CDC

PMax

SPRT

Stra

tifica

tion

(age

)Da

taba

se u

pdat

ed d

aily

BMax

SPRT

‡SC

Enge

r29US

A, i3

Dru

gSa

fety

Uncle

arUn

clear

Uncle

ar

DMSS

26,3

2,47

,48

USA,

DoD

PMax

SPRT

Uncle

arUn

clear

VA26

,32,

48US

A, V

APM

axSP

RTUn

clear

Uncle

arIH

S26,3

2,48

.49

USA,

IHS/

FDA

PMax

SPRT

Uncle

arUn

clear

PRIS

M26,3

2,48

USA,

FDA

/NV

POPM

axSP

RTUn

clear

Uncle

arBM

axSP

RT‡

Klei

n27†

USA,

VSD

BMax

SPRT

Mat

chin

g (a

ge g

roup

, site

, cal

enda

rye

ar a

nd re

spira

tory

viru

s sea

son)

Anal

ysis

dela

yed

at le

ast 8

wee

ksfro

m d

ate

of v

accin

atio

n46

Gee34

USA,

VSD

PMax

SPRT

Expe

cted

rate

s (ag

e, si

te)

Uncle

arBM

axSP

RT§

Mat

chin

g (a

ge, s

ite, v

accin

atio

n da

te)

Lee33

USA,

VSD

PMax

SPRT¶

Expe

cted

rate

s (ag

e an

d sit

e)Ad

just

ed fo

r par

tially

ela

psed

risk

inte

rval

and

del

ay in

the

arriv

al o

fin

patie

nt d

ata

BMax

SPRT‡

SCBo

thSt

ratifi

catio

n** (

age)

Brya

n31†

MHR

A, U

KPM

axSP

RTEx

pect

ed ra

tes (

age)

Adju

sted

for u

nder

repo

rting

(yel

low-

card

dat

a)Bu

rwen36

USA,

FDA

USPR

TNo

Criti

cal l

imits

adj

uste

d fo

rde

lays

in th

e cl

aim

s (ba

sed

onpr

evio

us s

easo

ns)

Loug

hlin35

†US

A,Op

tum

Insig

htAb

t’sm

odifi

catio

n of

SPRT

NoNo

(Con

tinue

s)near real-time vaccine safety surveillance 229



Tabl

e 1.

(Con

tinue

d)

Stud

yCo

untry

,or

gani

zatio

nM

etho

dCo

nfou

ndin

gDa

ta‐a

ccru

al la

g or

unde

rrepo

rting

adj

ustm

ent

Tse37

USA,

VSD

PMax

SPRT¶

Stra

tifica

tion

(age

, site

)Ad

just

ed fo

r par

tially

ela

psed

risk

inte

rval

and

dela

y in

the

arriv

al o

f inp

atie

nt d

ata

BMax

SPRT‡

SCDo

nega

n40†

UK, M

HRA

PMax

SPRT

Stra

tifica

tion

(age

)—fir

st y

ear o

fsu

rvei

llanc

eSe

nsiti

vity

ana

lyse

s ass

umin

g va

rious

deg

rees

of u

nder

repo

rting

(yel

low-

card

dat

a)Ne

lson38

USA,

VSD

GS PMax

SPRT

Expe

cted

cou

nts

(site

, gen

der,

age

grou

p,sit

e ×

age—

Poiss

on re

gres

sion)

No††

Tsen

g39US

A, V

SDGS

Stra

tifica

tion

(age

, dos

e nu

mbe

r—on

ly fo

r feb

rile

seizu

res,

urtic

aria

/ang

ione

urot

ic o

edem

a, a

sthm

a)No

††

Dale

y42†

USA,

VSD

PMax

SPRT¶

Expe

cted

rate

s (sit

e—ex

cept

for G

BS a

nd S

JS—

weig

hted

ave

rage

use

d)Ex

clusio

n of

the

mos

t rec

ent 1

4 we

eks o

fda

ta11

Kawa

i43US

A, V

SDPM

axSP

RT¶Ex

pect

ed ra

tes a

djus

ted

(age

, site

)De

laye

d an

alys

is un

til e

stim

ated

dat

a la

gac

crua

l and

follo

w-up

tim

e wa

s com

plet

edBM

axSP

RT‡SC

, stra

tifica

tion

(age

)W

eint

raub41

†US

A, V

SDPM

axSP

RTEx

pect

ed ra

tes (

age,

site

)An

alys

is de

laye

d 2

week

sM

urdo

ch44†

UK, H

PSSP

CSt

ratifi

catio

n (a

ge, s

ite)

NoYi

h45US

A, F

DAPM

axSP

RT¶Ex

pect

ed ra

tes (

age

for a

naph

ylax

is an

dse

izure

s and

dat

a pa

rtner

for s

eizu

res)

Adju

sted

for p

artia

lly e

laps

ed ri

sk in

terv

al a

ndde

lay

in th

e ar

rival

of i

npat

ient

dat

aBM

axSP

RT‡SC

, stra

tifica

tion

(sei

zure

s—ag

e,co

ncom

itant

PCV

13 6

–23

mon

ths)

HPS†

(unp

ublis

hed)

UK, H

PSSP

CSt

ratifi

catio

n (a

ge, s

ex fo

r her

pes

zost

er, s

ite)

No

MHR

A†(u

npub

lishe

d)MHR

A, U

KPM

axSP

RTEx

pect

ed ra

tes (

age)

Adju

sted

for u

nder

repo

rting

(yel

low-

card

dat

a)

Stud

ies

in it

alic

are

the

ones

iden

tified

from

exp

ert c

onta

cts.

AE‐A

dver

se e

vent

; BM

axSP

RT, b

inom

ial‐b

ased

max

imize

d se

quen

tial p

roba

bilit

y ra

tio te

st; C

DC, C

ente

rs fo

r Dise

ase

Cont

rol a

nd P

reve

ntio

n; D

MSS

, Def

ense

Med

ical S

urve

illan

ce S

yste

m; D

oD,

Depa

rtmen

t of D

efen

se; F

DA, F

ood

and

Drug

Adm

inist

ratio

n; H

PS, H

ealth

Pro

tect

ion

Scot

land

; IHS

, Ind

ian

Heal

th S

ervi

ce; M

HRA,

Med

icine

s and

Hea

lthca

re p

rodu

cts R

egul

ator

y Ag

ency

; MoH

,M

inist

ry o

f Hea

lth; N

VPO,Na

tiona

l Vac

cine

Prog

ram

Offi

ce; P

CV13

,13

‐val

ent p

neum

ococ

cal c

onju

gate

vac

cine;

PM

axSP

RT,

Poiss

on‐b

ased

max

imize

d se

quen

tial p

roba

bilit

y ra

tio te

st; P

RISM

,Po

st‐L

icens

ure

Rapi

d Im

mun

izatio

n Sa

fety

Mon

itorin

g; S

C, se

lf‐co

ntro

lled

desig

n; S

JS, S

teve

ns–Jo

hnso

n sy

ndro

me;

SPC

, sta

tistic

al p

roce

ss c

ontro

l; SP

RT, s

eque

ntia

l pro

babi

lity

ratio

test

; USP

RT,

upda

ting

sequ

entia

l pro

babi

lity

ratio

test

; VA,

Vet

eran

s Affa

irs; V

SD, V

accin

e Sa

fety

Dat

alin

k.*E

ach

uniq

ue c

ombi

natio

n of

pot

entia

l con

foun

ders

is id

entifi

ed, f

orm

ing

a st

ratu

m, a

nd a

bas

elin

e ris

k is

calcu

late

d. F

or e

ach

stra

tum

, a te

st st

atist

ic is

calc

ulat

ed, a

nd th

e te

st s

tatis

tics a

re c

ombi

ned.

† Add

ition

al in

form

atio

n ob

tain

ed fr

om th

e au

thor

s.‡ U

ses a

self‐

cont

rolle

d de

sign.

§ Use

s an

exac

t ver

sion

of th

e te

st, w

ith fl

exib

le m

atch

ing.

¶ Use

s the

con

ditio

nal v

ersio

n of

the

test

.**

Only

for i

nact

ivat

ed v

accin

es a

nd s

peci

fic o

utco

mes

(dem

yelin

atin

g di

seas

e of

the

cent

ral n

ervo

us sy

stem

, diso

rder

s of t

he p

erip

hera

l ner

vous

syst

em a

nd n

euro

path

y, se

izure

s, B

ell’s

pal

sy a

ndot

her c

rani

al n

erve

diso

rder

s).

††An

alys

is ba

sed

on th

e nu

mbe

r of d

oses

mig

ht m

inim

ize d

elay

s for

initi

al p

erio

ds o

f sur

veill

ance

.

a. leite et al.230



for potentialconfounders (51.4% of those adjusting),stratification (16.2%)or a combination ofa self-controlleddesignand stratification(13.5%).Thechoice of approaches also depended on the analyticalmethod selected.For group sequentialmethods andSPC, strategies to dealwith confounders were evenmore limited.When employinggroupsequentialmethods,only expected rate calculationsbased onthe confounders’distribution and stratification wereconsidered. For SPC, only stratification was used. Po-tentialconfounders considered include age,sex,geo-graphicsite, concomitantvaccineadministration,season and trend (Table 1).

Some ofthe prospective studies considered data-accrual lags in their analysis. Most often, the analysiswas delayed by some weeks (n = 7).Others adjustedfor partially elapsed risk intervals and delays in the ar-rival of inpatient data (n = 3).46For studies using spon-taneous report for the observed number of events (andEHR for the expected number of events),sensitivity

analyses with several degrees of underreporting wereconducted(n = 4).28,31,40Updatesto the previousdatasets already analysed were notconsidered a spe-cific strategy to adjustfor data-accruallags as theywould notreduce the time to signal.The majority ofstudies did notmention ways ordid notadjustfordata-accrual lags (n = 11).

DISCUSSIONOurcomprehensive systematic review has identifiedan increasingnumberof studiesand systemsimplementing NRTVSS.All the studiesidentifiedwere performed in high-income countries/regions withmost in the USA. This might reflect limited capacity inmany settings to provide registry data in a timely fash-ion and theinfra-structurerequired to setup thesystem.

A clear effortwas putinto using these methods toassess pandemic influenza vaccine safety. This vaccine

Figure 2.Studies included in the review, ordered by the year of publication. Continuous sequential test are underlined with single line, group sequential withbold line,and statisticalprocess controlwith dashed line.Grey background indicates non-published studies.*Results with previous published results.maxSPRT – Maximized Sequential Probability Ratio Test, P – Poisson version (†use of the conditional version), B – binomial version (‡use of self-controlledcase-series or extensions of the test). DMSS – Defense Medical Surveillance System, DTaP – acellular diphtheria-tetanus-pertussis vaccine, DTwP – wholecell diphtheria-tetanus-pertussis vaccine, GBMV – Group-B Meningococcal Vaccine, HPS – Health Protection Scotland, HPV2 – bivalent human papillomavirusvaccine, HPV4 – quadrivalent human papillomavirus vaccine, IHS – (US) Indian Health Service, IPV – inactivated poliovirus vaccine, MCV – meningococcalconjugate vaccine,MHRA – Medicines and Healthcare products Regulatory Agency,MMRV – Measles-Mumps-Rubella-Varicella combination vaccine,PCV13 – 13-valent pneumococcal conjugate vaccine, PRISM – Post-Licensure Rapid Immunization Safety Monitoring, RRV – Rhesus-Rotavirus vaccine,RV5 – pentavalent rotavirus vaccine, VA – Veteran’s Affaires, VSD – Vaccine Safety Datalink




Tabl

e 2.

Met

hods

and

resp

ectiv

e ex

tens

ions

use

d by

the

elig

ible

stud

ies.

Mai

n ad

vant

ages

and

cha

lleng

es o

f eac

h m

etho

d ar

e pr

ovid

edGe

neric

met

hod

Vers

ion

Gene

ral d

escr

iptio

nCo

mpa

rato

rAd

vant

ages

and

disa

dvan

tage

sCo

nfou

ndin

g

Cont

inuo

us s

eque

ntia

l—al

low

exam

inat

ion

of th

e da

ta a

s ofte

n as

des

ired,

the

vario

us v

ersio

ns a

re d

escr

ibed

late

r (SP

RT a

nd M

axSP

RT)

Wal

d’s S

PRT

Gene

ral

desc

riptio

nTh

is is

the

gene

ricm

etho

d pr

opos

edby

Wal

d in

the

1940

s.

For v

accin

e sa

fety

, a P

oiss

on m

odel

wou

ldty

pica

lly b

e us

ed w

ith th

e ob

serv

ed c

ount

com

pare

d wi

th a

fixe

d ex

pect

ed c

ount

.9,

50

Adva

ntag

e—Ea

sy im

plem

enta

tion

of th

e Po

isson

mod

el.

Cova

riate

adj

uste

d ex

pect

ed le

vels

can

be o

btai

ned

to a

llow

for

poss

ible

con

foun

ding

.9

Disa

dvan

tage

(com

pare

d wi

thM

axSP

RT)—

Fixe

d sin

gle

alte

rnat

ive

hypo

thes

is (e

.g. R

R =

3)wh

ose

choi

ce w

ill u

sual

ly b

e ar

bitra

ry.

50

Max

SPRT

Gene

ral

desc

riptio

nTh

is ge

neric

ally

desc

ribes

all

SPRT

met

hods

that

hav

e a

com

posit

eal

tern

ativ

ehy

poth

esis

(RR

> 1

).

Depe

nds o

n th

e ve

rsio

n of

the

test

(refe

r to

succ

eedi

ng d

ata)

.No

nee

d to

spe

cify

a sin

gle

alte

rnat

ive.

50De

pend

s on

the

vers

ion

of th

e te

st (r

efer

tosu

ccee

ding

dat

a).

Poiss

on—

This

impl

emen

tatio

n as

sum

es a

Poiss

on d

istrib

utio

n fo

r obs

erve

dco

unts

and

com

pare

s to

a fix

edex

pect

ed m

ean.

50

Adva

ntag

e—Si

mpl

e to

impl

emen

t.Th

e us

e of

a fi

xed

expe

cted

leve

lin

crea

ses p

ower

.10

Cova

riate

adj

uste

d ex

pect

edle

vels

can

be o

btai

ned

toal

low

for p

ossib

le c

onfo

undi

ng.

Pote

ntia

l for

con

foun

ding

due

tose

ason

al o

r tem

pora

l cha

nges

indi

seas

e in

cide

nce

or c

odin

g.10

Disa

dvan

tage

—Re

lies o

n ac

cura

teda

ta fo

r the

exp

ecte

d le

vel,

whic

hm

ay n

ot b

e th

e ca

se if

dat

a ar

elim

ited

or o

nly

hist

oric

al.

51

Bino

mia

l—

Base

d on

a b

inom

ial d

istrib

utio

nev

ents

occ

urrin

g am

ong

vacc

ine

expo

sed

indi

vidu

als/

perio

dsve

rsus

com

paris

on (u

nexp

osed

indi

vidu

als/

perio

ds).

50

Adva

ntag

e—Do

es n

ot re

ly o

n a

fixed

exp

ecte

d va

lue

and

can

mat

ch o

n co

nfou

nder

s or

com

pare

to o

ther

per

iods

with

in in

divi

dual

s.50

Can

be u

sed

in d

iffer

ent v

ersio

ns—

mat

chin

g co

ntro

ls (fi

xed

or fl

exib

lem

atch

ing

ratio

—ex

act s

eque

ntia

lan

alys

is57) o

r sel

f-con

trolle

d de

sign

(SCC

S or

SCR

I) or

con

sider

ing

prev

ious

sea

sons

, avo

idin

g th

ehe

alth

y va

ccin

ee e

ffect

(DID

24).

Disa

dvan

tage

—Le

ss p

ower

ful

than

Poi

sson

unl

ess m

ultip

leun

vacc

inat

ed a

vaila

ble

per v

accin

ated

.50

The

use

of a

self-

cont

rolle

dde

sign

with

pos

t-exp

osur

eco

mpa

rison

inte

rval

s mig

htre

sult

in d

elay

s.46

Pote

ntia

l for

con

foun

ding

dep

ends

on th

e ve

rsio

n of

the

test

use

d.

Cond

ition

al—

Assu

mes

a P

oiss

on p

roce

ss fo

rth

e cu

mul

ativ

e pe

rson

-tim

e to

obse

rve

a nu

mbe

r of a

dver

seev

ents

.51

Adva

ntag

e—Do

es n

ot a

ssum

e th

eex

pect

ed n

umbe

r of c

ases

is k

nown

(as

the

Poiss

on-b

ased

Max

SPRT

).

Sam

e as

Poi

sson

Acco

unts

for u

ncer

tain

ty in

hist

orica

l dat

a.51

Disa

dvan

tage

—As

sum

es c

onst

ant

even

t rat

es a

re in

hist

orica

l and

surv

eilla

nce

data

.51

Grou

pse

quen

tial

test

ing

Gene

ral

desc

riptio

nDa

ta a

re e

xam

ined

at d

iscre

tepo

ints

in ti

me.10

Seve

ral a

ppro

ache

s use

d a

grou

p se

quen

tial w

ay (P

Max

SPRT

,Ab

t’s m

odifi

catio

n of

SPR

T,US

PRT)

ofte

n im

plem

entin

g an

alph

a-sp

endi

ng a

ppro

ach

(usin

g a

func

tion

to d

eter

min

e ho

w to

‘spe

nd’

the

alph

a in

the

diffe

rent

test

s).

12

Adva

ntag

e—Re

quire

sle

ss fr

eque

nt u

pdat

es.

Depe

nds o

n th

e sp

ecifi

cve

rsio

n us

ed.

Disa

dvan

tage

—Co

ntin

uous

test

s are

mor

e po

werfu

l.58

Less

exp

lore

d(c

ompa

red

with

con

tinuo

us te

sts)

in th

e ob

serv

atio

nal s

ettin

g,in

cludi

ng a

djus

tmen

t for

con

foun

ders

.M

ore

com

plex

des

igns

.12

(Con

tinue

s)a. leite et al.232



is a good example of the importance of post-licensuresurveillance due to potential safety concerns.32Menin-gococcalgroup B vaccine in New Zealand21 repre-sents a similar situation,where NRTVSS,along withenhancedpassivesurveillanceand other activemethods,was implemented after the vaccine was ap-proved without phase III trials. Other situations wherethese methods have been particularly usefulincludevaccines/AE ofconcerndue to experienceswithpreviousversionsof the vaccine—forexample,rotavirus/intussusception25 and influenza/GBS.32 Forpreviously suspected AE,the setof methodsherereviewed has the advantage of informing in a timelymanner the existence of a safety concern or reassuringregulatory authoritiesand the public aboutvaccinesafety.

In this review, we have identified different methodsto perform NRTVSS using EHR and the way thesehave been applied, both by VSD and by other institu-tions.All the methodsidentified are derived fromWald’s sequential test.50,59,60When choosing a partic-ular method, it is important to be aware of its proper-ties. Propertiesof the continuousand groupsequentialmethods have been studied in the contextof drug safety.12 Group sequentialmethodsweredeemed to be more appropriate when data updatesare less frequent,12 butmore recentwork comparingthese methods has found that for any group sequentialdesign, there is a better continuous method and recom-mended thatthe data are looked atas frequently aspossible.58 After selecting themethodologicalap-proach, it is necessary to choose the specific test to em-ploy. For example,using the PMaxSPRT andBMaxSPRT simultaneously mightbe a more robustapproach owing to complementary strengths.How-ever,as previously suggested,BMaxSPRT might failto identify a signalwhen investigating very rareevents.Hence,an alternative is to use PMaxSPRTwhen less than 50 events are anticipated and the condi-tionalversion when the ratio ofobserved historicalevents to upperlimitis ≤2.5.The use ofa targetedapproach has been considered in VSD’s more recentwork.24,33,34,42,43

On the otherhand,the propertiesof SPC-basedmethods applied to vaccine safety have notbeen ex-tensively studied. Both Kulldorff et al.50and Musondaet al.61have argued that SPC-based methods such ascumulative sum are not appropriate to perform surveil-lance for newly introduced products as the aim is todetecta safety problem thatis already presentandnot a sudden change. These authors defend the use ofsuch methodsin the contextof surveillanceforbatch-related problems (problems arising atthe timeTa

ble

2.(C

ontin

ued)

Gene

ric m

etho

dVe

rsio

nGe

nera

l des

crip

tion

Com

para

tor

Adva

ntag

es a

nd d

isadv

anta

ges

Conf

ound

ing

Stat

istica

lpr

oces

s con

trolGe

nera

lde

scrip

tion

Grap

hica

l app

roac

hwh

ere

the

num

ber o

f eve

nts i

sco

mpa

red

with

an

uppe

r lim

it(th

e th

resh

old

is ty

pica

lly—

mea

n+

a ce

rtain

num

ber o

f SD)

.56

Expe

cted

cou

nt.

Adva

ntag

e—Ea

sy to

impl

emen

t.St

ratifi

catio

n ca

n be

use

dto

han

dle

conf

ound

ing.

Disa

dvan

tage

—Le

ss m

etho

dolo

gica

lwo

rk o

n ap

plic

atio

ns to

vac

cine

safe

ty.

No fo

rmal

way

to c

ontro

l for

mul

tiple

test

.

AE, a

dver

se e

vent

; DID

, diff

eren

ce-in

-diff

eren

ce; (

P)M

axSP

RT, (

Poiss

on-b

ased

) max

imize

d pr

obab

ility

ratio

test

; RR,

rela

tive

risk;

SCC

S, se

lf-co

ntro

lled

case

serie

s; S

CRI,

self-

cont

rolle

d ris

k in

terv

al;

SD, s

tand

ard

devi

atio

n; S

PRT,

sequ

entia

l pro

babi

lity

ratio

test

; USP

RT, u

pdat

ing

sequ

entia

l pro

babi

lity

ratio

test

; UL,

upp

er li

mit.




of manufactureratherthan related to theproductitself).However,we should consider thatatthe timeof introduction,if there is a safety problem with thatspecific vaccine and an appropriate comparison groupis used, a sudden change would be observable as well.Given its ease of implantation,SPC is attractive,butrecommendations on the use of SPC are deferred untilfurther research on their properties is available.

Controlfor potentialconfounders has been limitedin both the strategies employed and factors adjustedfor. This observation isin agreementwith Nelsonet al.,12who have argued for better methods for con-founder adjustment, in particular at the analysis stage.Recent work has been performed in this area, adaptinggroup sequentialmethods with regression adjustmentand comparing thisto existing approaches.62,63Tothe best of our knowledge, these promising approachesare stillat the developmentstage and have notyetbeen applied to new studies.As pointed outbyYih,11it might not be possible to adjust for all possi-ble confounders in this setting,which can lead tospurious signals.However,it should be noted that,as a near real-timeanalysis,aimedat quicklyidentifying/strengthening signals,priority isgivento rapid results.As such,confounding adjustment isnot deemedas critical—morecompleteanalysescan be performed atconfirmatory stages.11 Thesemightinclude adjusting foradditionalconfoundersor a more detailed adjustment (e.g.using finer cate-gorization of a variable) to avoid residual confound-ing. The specific confounders to adjust for should bedecided on the basis of the vaccine, outcome and agegroups studied.In addition to those factors consid-ered by studies,adjustmentsfor day-of-the-weekeffects or co-morbidities might be required.11Never-theless,12 studies13,24–27,29,30,35,36did notrefertopotential confounding in at least one of the analysesreported in their published texts.

Best practice using EHR apply equally to NRTVSSas to any study using these kind of data. For example,Lanes et al. provide an approach to identify outcomesin healthcare databases.64One of the aspects to considerwhile doing so is misclassification. In some occasions,manualreview ofindividualmedicalrecords can beused,particularly if a signalis found.In this review,only two studies21,35performed this confirmation beforerunning the NRTVSS analysis, as doing so might delaythe surveillance process.Alternatively,multiple algo-rithms mightbe developed,providing a trade-off be-tween sensitivity and positive predictive values (PPV).In the NRTVSS,an algorithm with higher sensitivityand moderate PPV is generally considered to be timelierthan algorithms with moderate sensitivity algorithm and

high PPV. This should be considered for the specificoutcome understudy,its seriousnessand the dataavailable.65 Misclassification ofthe exposure mightalso be problematic. A possible approach is to restrictthe analysis to vaccinated individuals, avoiding poten-tial biases.11

A key aspect to consider while using these methods isthe availability of timely data. ‘Real-time’ analyses aredifficult to achieve, and thus, the expression ‘near real-time’ is preferred.In fact,delays can occur at variousstages, including delays in diagnosis (e.g. for conditionswith more insidious onset), recording (e.g. retrospectiverecording of vaccination administration or diagnosis),receiving the data for analysis (due to either incompletedata accrualor partially accrued risk windows)andreporting.The timeliness of data should thus be con-sidered.Some studies have delayed the analysis forsome weeks.13,23,25,27,41–43While this approach givestime for data to accrue,it will notreduce the time tosignal.The use of group sequential methods with lessfrequenttesting portraysa similarsituation wheremore time has been given fordata to accrue.35,38,39

Nevertheless,for events occurring closerto the timeof testing,data-accruallags may stillbe problematic.Finally,adjustments for partially elapsed risk intervaland delays in the arrivalof inpatientdata have beenproposed (through the expected numberof events)46

or integrated in the criticallimits calculation36. Thesecan decrease the time to signal,based on previouslyobserved data-accrualpatterns.They have been ap-plied in a few,influenza vaccine,studies.Influenzavaccinespose particularchallengeswhen using de-layed data asfailure to detecta signalbefore theseason ends willimpede adequate action.Strategiesproposed so fardo not specifically addressdelaysbetween illness onset and diagnosis.

Only three of the 11 outcomes identified in the pro-spective studies were confirmed as true signals. In ad-dition to issues already raised (confounding factorsthathave notbeen considered,misclassification ofthe outcome),unconfirmed signalswere due to (i)changes in the true incidence or coding practices; (ii)inappropriate comparison groups;(iii) uncertainty inbackground rates; and (iv) type I errors.11,33For typeI errors,additionalstrategies to reduce the false dis-covery rate are available atthe planning stage:theseinclude delaying the first test,66requiring a minimumnumber of events to occur before rejecting the null hy-pothesis67 or, in the case ofgroup sequentialtests,selecting an O’Brien–Fleming threshold.The latterspends lessalpha in earliertests and wasused byNelson et al.38During the surveillance period, it is im-portant to update the critical limits as data arrive, as the

a. leite et al.234



observed data might differ from those planned.66As inthe case ofoutcome identification,these consider-ations should be balanced againstthe importance ofdetecting signals in a timely manner. Even after care-ful consideration of all these aspects before and duringsurveillance,possible spurious signals may still arise.This emphasizes the need for a predetermined plan ofaction forsignalrefinementif a signalis found.11

The plan should include a careful decision on the datasource to use to test the hypothesis in subsequent anal-yses if needed, owing to potential biases with the useof the same data to identify and testthe signals.NRTVSS is thus nota stand-alone method butpartof the signal detection and evaluation process.

This review aimed at capturing studies and systemsworldwide using EHR to perform NRTVSS.Our rig-orous search strategy and further contacts with manyexpertson vaccinesafety from differentcountriesand institutions(with a satisfactory responserate,70.6%) should have minimized the risk of missing sys-tems currently in use. However, we cannot exclude theexistence of similar systems elsewhere.Furthermore,some information was missing from the studies in-cluded,which we have tried to reduce by contactingthe authors.The missing information mostoften re-lated to confounding controlstrategies and the data-accruallag adjustmentemployed.This mightreflectthe limited options to address these issues,especiallyfor the earlier studies.

Countries considering introduction of these methodsshould benefitfrom the work developed so farandfrom strategies under development.There should bea cautious reflection on the availability of timely dataand theircharacteristics(including discussion withthe data providers),the vaccine(s) and outcome(s) tobe studied and the infra-structure needed in case a sig-nal is detected.Future directions forresearch mightinclude further development and application of strate-gies for adjustmentfor confounding and data-accruallag, as well as consideration of other methods not yetapplied to observational settings but in use in clinicaltrials,for example,Bayesian approaches to group se-quentialtests.68 Bayesian methodscan incorporateprevious information (such as the data generated bypre-licensure studies) and potentially provide a moreflexible approach.

In conclusion,NRTVSS using EHR to assess thesafety of newly introduced vaccines is being increas-ingly used in the USA,with limited introduction in afew other countries. These methods ensure timely de-tection of safety signals. New methods have been inte-grated over time, but strategies to account for potentialconfounders and data-accruallags have received less

attention.As new vaccines are expected to be intro-duced and the public questionsvaccine safety,thedemand for strong post-licensure surveillance systemswill increase.

CONFLICT OF INTERESTThe authors declare no conflict of interest.

KEY POINTS• Near real-time vaccine safety surveillance using

electronic health records (EHR) is one of the op-tions available to identify vaccine safety signals.

• Use of near real-time vaccine safety surveillanceusing EHR has been increasing in the USA but todate has only been considered in a few othercountries.

• Methods available have developed over time andhave been integrated into systems using this kindof surveillance.Continuoussequentialtestinghas been the preferred approach.

• Strategiesto addresspotentialconfoundingfactors are currently limited, but further develop-ments may address this in the near future.

• Timelinessand allowing fordata-accruallagare importantfactorsfor consideration whenimplementing nearreal-time surveillance usingEHR. Lags haveonly beenaddressedin afew studies.

ETHICS STATEMENTThe authors state that no ethical approval was needed.

ACKNOWLEDGEMENTSThe research was funded by the National Institute forHealthResearchHealthProtectionResearchUnit(NIHR HPRU) in Immunisation at the London Schoolof Hygiene & TropicalMedicine in partnership withPublic Health England (PHE).The views expressedare those of the author(s) and not necessarily those ofthe NHS,the NIHR, the Departmentof Health orPublic Health England. The funders had no role in thestudy design, data collection, analysis or interpretation.The authors thank Dr Abdoulreza Esteghamati (TehranUniversity of MedicalSciences/GACVS),Dr AnandaAmarasinghe (Ministry of Health, Sri Lanka/GACVS),ProfBrigitte Keller-Stanislawski(PaulEhrlich-Institut/GACVS),Bruce Fireman (Kaiser Permanente),ClaireCameron (Health Protection Scotland), Dr Daniel Salmon




(JohnsHopkinsUniversity Schoolof Public Health),Dr David Martin (Food and Drug Administration),Dr Edward Belongia(Marshfield ClinicResearchFoundation), Dr Gagandeep Kang (Christian MedicalCollege),Dr Hanna Nohynek (NationalInstitute forHealth and Welfare),Heather Murdoch (Health Pro-tection Scotland),Jeanne Loughlin (OptumInsight),Jorgen Bauwens (Brighton Collaboration), Dr KatherineDonegan (MHRA), Dr Katherine Yih (Harvard PilgrimHealth Care Institute), Kevin Pollock (Health ProtectionScotland),Lorenz Von Seidlein,Dr Matthew Daley(KaiserPermanente Colorado),Dr Melinda Wharton(Centers for Disease Control and Prevention/GACVS),Dr Nicola Klein (Kaiser Permanente), Ned Lewis (KaiserPermanente),Dr Patrick Garman (US Army),Dr PhilBryan (MHRA),Dr Punam Mangtani (London Schoolof Hygiene & TropicalMedicine),Dr RogerBaxter(Kaiser Permanente), Dr Silvia Perez-Vilar (Foundationfor the Promotion of Health and BiomedicalResearchof Valencia Region), Dr Sharon K. Greene (New YorkCity Departmentof Health and MentalHygiene),ProfStephen Evans (London School of Hygiene & TropicalMedicine),Dr SteveBlack (CincinnatiChildren’sHospital Medical Center), Dr Suzie Seabroke (MHRA),Dr Wan-Ting Huang (Taiwan Centers for Disease Control),Dr Xavier Kurz (European Medicines Agency/GACVS)and allother researchers who generously gave theirtime to answer queries arising from this research.

REFERENCES1. Bonhoeffer J, Black S, Izurieta H, Zuber P, Sturkenboom M. Current status and

future directions of post-marketing vaccine safety monitoring with focus on USAand Europe. Biologicals 2012 Sep; 40: 393–397.

2. Chen RT,Glanz JM, VellozziC. Pharmacoepidemiologic studies ofvaccinesafety.In Pharmacoepidemiology.Strom BL,KimmelSE and Hennessy S(eds). Wiley-Backwell: Chicester, 2011; 423–468.

3. O’Hagan DT, Rappuoli R. The safety of vaccines. Drug Discov Today 2004; 9:846–854.

4. McPhillips H, Marcuse EK. Vaccine safety. Curr Probl Pediatr 2001; 31: 95–121.5. Lopalco PL, Johansen K, Ciancio B, Gomes HC, Kramarz P, Giesecke J. Mon-

itoring and assessing vaccine safety:a European perspective.ExpertRev Vac-cines 2010; 9: 371–380.

6. Dal Pan GJ, Lindquist M, Gelperin K. Postmarketingspontaneouspharmacovigilance reporting systems.In Pharmacoepidemiology.Strom BL,Kimmel SE and HennessyS (eds). Wiley-Backwell:Chicester,2011;137–157.

7. Crawford NW,ClothierH, Hodgson K,SelvarajG, Easton ML,Buttery JP.Active surveillance foradverse eventsfollowing immunization.ExpertRevVaccines 2014; 13: 265–76.

8. Waldman EA, Luhm KR, Monteiro SA, Freitas FR. Surveillance of adverse ef-fectfollowing vaccination and safety ofimmunization programs.Rev SaudePublica 2011; 45: 173–84.

9. Davis RL, Kolczak M, Lewis E, Nordin J, Goodman M, Shay DK, et al. Activesurveillance of vaccine safety: a system to detect early signs of adverse events.Epidemiology 2005; 16: 336–41.

10. Kulldorff M. Sequential statistical methods for prospective postmarketing safetysurveillance. In Pharmacoepidemiology. Strom BL, Kimmel SE and Hennessy S(eds). Wiley-Backwell: Chicester, 2011; 852–867.

11. Yih WK, Kulldorff M, Fireman BH, Shui IM, Lewis EM, Klein NP, et al. Activesurveillance for adverse events:the experience of the Vaccine Safety Datalinkproject. Pediatrics 2011; 127(S1): S54–64.

12. Nelson JC, Cook AJ, Yu O, Zhao S, Jackson LA, Psaty BM. Methods for obser-vationalpost-licensure medicalproductsafety surveillance.StatMethods MedRes 2015; 24: 177–193.

13. Lieu TA, Kulldorff M, Davis RL, Lewis EM,Weintraub E, Yih K, et al. Real-time vaccine safety surveillance for the early detection of adverse events.MedCare 2007; 45(S2): S89–95.

14. BrightonCollaboration.Newsletter.2015;.https://brightoncollaboration.org/public/newsletter.html (accessed 19 June 2015).

15. World Health Organization. The Global Advisory Committee on Vaccine Safety.2015; http://www.who.int/vaccine_safety/committee/en/(accessed19 May2015).

16. Brighton Collaboration.Who we are.2015;https://brightoncollaboration.org/public/who-we-are.html (accessed 19 May 2015).

17. ADVANCE. About ADVANCE. 2015; http://www.advance-vaccines.eu/(accessed 2015-05-19).

18. GovernmentDigitalService.Medicines& HealthcareproductsRegulatoryAgency.2015;https://www.gov.uk/government/organisations/medicines-and-healthcare-products-regulatory-agency (accessed 19 May 2015).

19. Centers forDisease Controland Prevention.Vaccine Safety Datalink (VSD).2015; http://www.cdc.gov/vaccinesafety/Activities/VSD.html (accessed 19 May2015).

20. Statens Serum Institut.Statens Serum Institut.2015; http://www.ssi.dk/english.aspx (accessed 19 May 2015).

21. McNicholas A,Galloway Y,Stehr-Green P,Reid S,Radke S,Sexton K,et al.Post-marketing safety monitoring of a new group B meningococcalvaccine inNew Zealand, 2004–2006. Hum Vacc 2007; 3: 196–204.

22. Brown JS, Moore KM, Braun MM, Ziyadeh N, Chan KA, Lee GM, et al. Activeinfluenza vaccine safety surveillance: potential within a healthcare claims envi-ronment. Med Care 2009; 47: 1251–7.

23. Yih WK, Nordin JD, Kulldorff M, Lewis E, Lieu TA, Shi P, et al. An assessmentof the safety of adolescent and adult tetanus-diphtheria-acellular pertussis (Tdap)vaccine,using active surveillance foradverse eventsin the Vaccine SafetyDatalink. Vaccine 2009; 27: 4257–62.

24. Greene SK, Kulldorff M, Lewis EM, Li R, Yin R, Weintraub ES, et al. Near real-time surveillance for influenza vaccine safety:proof-of-conceptin the VaccineSafety Datalink Project. Am J Epidemiol 2010; 171: 177–88.

25. Belongia EA, Irving SA, Shui IM, Kulldorff M, Lewis E, Yin R, et al. Real-timesurveillance to assess risk of intussusception and other adverse events after pen-tavalent, bovine-derived rotavirus vaccine. Pediatr Infect Dis J 2010; 29: 1–5.

26. National Vaccine Advisory Committee (NVAC). Report on 2009 H1N1 vaccinesafety risk assessment.NVAC. 2010;http://www.hhs.gov/nvpo/nvac/reports/vsrawg_report_apr2010.html (accessed 19 June 2015).

27. Klein NP,Fireman B,Yih WK, Lewis E,Kulldorff M,Ray P,etal. Measles-mumps-rubella-varicella combination vaccine and the risk offebrile seizures.Pediatrics 2010; 126: e1–8.

28. Bryan P, Seabroke S, Davies C. H1N1 vaccine safety: real-time surveillance inthe UK. Lancet 2010; 376: 417–8.

29. Enger C, Turnbull B, Gately R, Loughlin J, Chan KA. H1N1 influenza vaccinesurveillance project. Pharmacoepidemiol Drug Saf 2010; 19: S14–S5.

30. Huang WT, Chen WW, Yang HW, Chen WC, Chao YN, Huang YW,et al. Design ofa robustinfrastructureto monitorthe safety ofthe pan-demicA(H1N1)2009 vaccination program in Taiwan.Vaccine2010;28:7161–6.

31. Bryan P, Seabroke S. No increased risk of febrile convulsions after seasonal in-fluenza immunisation in UK. Lancet 2011; 377: 904.

32. Salmon DA, Akhtar A, Mergler MJ, Vannice KS, Izurieta H, Ball R, et al. Immu-nization-safety monitoring systems forthe 2009 H1N1 monovalentinfluenzavaccination program. Pediatrics 2011; 127(S1): S78–86.

33. Lee GM,Greene SK, Weintraub ES, Baggs J, Kulldorff M, Fireman BH, et al.H1N1 and seasonalinfluenza vaccine safety in the Vaccine Safety Datalinkproject. Am J Prev Med 2011; 41: 121–8.

34. Gee J, Naleway A, Shui I, Baggs J, Yin R, Li R, et al. Monitoring the safety ofquadrivalenthuman papillomavirus vaccine:findings from the Vaccine SafetyDatalink. Vaccine 2011; 29: 8279–84.

35. Loughlin J, Mast TC, DohertyMC, Wang FT, Wong J, SeegerJD.Postmarketing evaluation of the short-term safety of the pentavalentrotavirusvaccine. Pediatr Infect Dis J 2012; 31: 292–6.

36. Burwen DR,Sandhu SK,MaCurdy TE,Kelman JA,GibbsJM, Garcia B,et al. Surveillancefor Guillain-Barresyndromeafterinfluenzavaccinationamong the Medicare population,2009–2010.Am J Public Health 2012;102:1921–7.

37. Tse A,Tseng HF,Greene SK,VellozziC, Lee GM.Signalidentification andevaluation for risk of febrile seizures in children following trivalent inactivatedinfluenza vaccine in the Vaccine Safety Datalink Project,2010–2011.Vaccine2012; 30: 2024–31.

38. Nelson JC, Yu O, Dominguez-Islas CP, Cook AJ, Peterson D, Greene SK, et al.Adapting group sequential methods to observational postlicensure vaccine safetysurveillance: results of a pentavalent combination DTaP-IPV-Hib vaccine safetystudy. Am J Epidemiol 2013; 177: 131–41.

a. leite et al.236



39. Tseng HF, Sy LS, Liu IL, Qian L, Marcy SM, Weintraub E, et al. Postlicensuresurveillance for pre-specified adverse events following the 13-valent pneumococ-cal conjugate vaccine in children. Vaccine 2013; 31: 2578–83.

40. Donegan K,Beau-Lejdstrom R,King B, Seabroke S,Thomson A,Bryan P.Bivalent human papillomavirus vaccine and the risk of fatigue syndromes in girlsin the UK. Vaccine 2013; 31: 4961–7.

41. Weintraub ES, Baggs J, Duffy J, Vellozzi C, Belongia EA, Irving S, et al. Risk ofintussusception after monovalent rotavirus vaccination. N Engl J Med 2014; 370:513–9.

42. Daley MF, Yih WK, Glanz JM, Hambidge SJ, Narwaney KJ, Yin R, et al. Safetyof diphtheria, tetanus, acellular pertussis and inactivated poliovirus (DTaP-IPV)vaccine. Vaccine 2014; 32: 3019–24.

43. Kawai AT, Li L, Kulldorff M, Vellozzi C, Weintraub E, Baxter R, et al. Absenceof associationsbetween influenza vaccinesand increased risksof seizures,Guillain-Barre syndrome, encephalitis, or anaphylaxis in the 2012–2013 season.Pharmacoepidemiol Drug Saf 2014; 23: 548–53.

44. Murdoch H, McFadden M, Smith-Palmer A, von Wissmann B, Cameron C. Ac-tive monitoring of potentialadverse immunisation events with hospitaladmis-sion data and linked analysis in Scotland. Lancet 2014; 384(S2): S10.

45. Yih K, Zichitella L, Sandhu S, Nguyen M, Kuldorff M, Cole D, et al. Accessingthe freshestfeasibledatafor conductingactiveinfluenzavaccinesafetysurveillance.Mini-Sentinel.2015; http://www.mini-sentinel.org/work_products/PRISM/Mini-Sentinel_PRISM_Active-Influenza-Vaccine-Safety-Surveillance-Report.pdf (accessed 19 June 2015).

46. Greene SK, Kulldorff M, Yin R, Yih WK, Lieu TA, Weintraub ES, et al. Near real-time vaccine safety surveillance with partially accrued data.PharmacoepidemiolDrug Saf 2011; 20: 583–90.

47. Garman P. Enhanced surveillance of novel H1N1 vaccines among military person-nel.Defense MedicalSurveillance System.2009;http://www.fda.gov/downloads/AdvisoryCommittees/CommitteesMeetingMaterials/BloodVaccines-BloodVaccinesandO/VaccinesandRelatedBiologicalProductsAdvisoryCommittee/UCM198078.ppt(accessed 19 June 2015), ND.

48. National Vaccine Advisory Committee (NVAC). H1N1 Vaccine Safety Risk As-sessment Working Group Report. NVAC. 2012; http://www.hhs.gov/nvpo/nvac/re-ports/vsrawg_report_january_2012.pdf (accessed 19 June 2015).

49. SandhuSK. Updateon Surveillancefor Guillain-BarréSyndromeafterVaccination with PandemicInfluenza A/H1N1 2009-containing Vaccines,2009–2011. Vaccines and Related Biological Products Advisory Committee. 2011;http://fda.yorkcast.com/webcast/Viewer/?peid=75dcd91903204870aff160cb9d5528151d (accessed 19 June 2015).

50. Kulldorff M,Davis RL,Kolczak M,Lewis E,Lieu T, PlattR. A maximizedsequentialprobability ratio testfor drug and vaccine safety surveillance.SeqAnal 2011; 30: 58–78.

51. Li L, Kulldorff M. A conditional maximized sequential probability ratio test forpharmacovigilance. Stat Med 2010; 29: 284–95.

52. Ellenberg SS,Fleming TR,DeMets DL.Statistical,philosophicaland ethicalissues in data monitoring. In Data Monitoring Committees in Clinical Trials: APractical Perspective. Wiley: Chicester, 2003; 119–152.

53. Franks R,Sandhu S,Avagyan A,Lu Y, Hong H,Garcia B,etal. Robustnesspropertiesof a sequentialtest for vaccinesafetyin the presenceofmisspecification. Stat Anal Data Min 2014; 7: 368–75.

54. Abt K. Poisson sequential sampling modified towards maximal safety in adverseevent monitoring. Biometrical J 1998; 40: 21–41.

55. Cook AJ, Tiwari RC, Wellman RD, Heckbert SR, Li LL, Heagerty P, et al. Sta-tistical approaches to group sequential monitoring of postmarket safety surveil-lancedata:currentstateof the art for use in the Mini-Sentinelpilot.Pharmacoepidemiol Drug Saf 2012; 21: 72–81.

56. Carey RG,Stake LV.Improving Healthcare with ControlCharts:Basic andAdvanced SPC Methods and Case Studies.ASQ Quality Press:Milwakukee,Wisconsin, 2003.

57. Lewis E, Fireman B,Klein NP, BaxterR. Exactsequentialanalysisforvaccine safety surveillance.12th AnnualConference on Vaccine Research,Bethesda; 2009. http://www.nfid.org/professional-education/archives/acvr/acvr09.pdf(accessed 19 June 2015).

58. Silva IR,KulldorffM. Continuous versus group sequentialanalysis forpost-marketdrug and vaccine safety surveillance.Biometrics2015.doi:10.1111/biom.12324.

59. Jennison C,TurnbullBW. Introduction.In Group SequentialMethodswithApplications to Clinical Trials. CRC Press: Florida, 1999; 1–19.

60. Grigg O, FarewellV, SpiegelhalterD. Use of risk-adjusted CUSUM andRSPRTcharts for monitoring in medical contexts.Stat Methods Med Res 2003;12: 147–70.

61. Musonda P, Hocine MN, Andrews NJ, Tubert-Bitter P, Farrington CP. Monitoringvaccine safety using case series cumulative sum charts. Vaccine 2008; 26: 5358–67.

62. Cook AJ,Wellman RD,Nelson JC,Jackson LA,Tiwari RC.Group sequentialmethod for observational data by using generalized estimating equations: appli-cation to Vaccine Safety Datalink.J Roy Stat Soc C–App 2015; 64: 319–38.

63. Stratton KG, Cook AJ, Jackson LA, Nelson JC. Simulation study comparing ex-posure matching with regression adjustmentin an observationalsafety settingwith group sequential monitoring. Stat Med 2014; 34: 1117–1133.

64. Lanes S, Brown JS, Haynes K, Pollack MF, Walker AM. Identifying health out-comes in healthcare databases. Pharmacoepidemiol Drug Saf 2015; 24: 1009–16.

65. Maro JC, Brown JS, Dal Pan GJ, Kulldorff M. Minimizing signal detection timein postmarket sequential analysis: balancing positive predictive value and sensi-tivity. Pharmacoepidemiol Drug Saf 2014; 23: 839–48.

66. Nelson JC,Cook AJ, Yu O, et al. Challenges in the design and analysis ofsequentially monitored postmarket safety surveillance evaluations using electronicobservational health care data. Pharmacoepidemiol Drug Saf 2012; 21: 62–71.

67. KuldorffM, Silva IR. Continuous post-marketsequentialsafety surveillancewith minimum events to signal. Rev Stat 2015; http://arxiv.org/abs/1503.01978.

68. GsponerT, GerberF, Bornkamp B,Ohlssen D,VandemeulebroeckeM,Schmidli H. A practical guide to Bayesian group sequential designs. Pharm Stat2014; 13: 71–80.

SUPPORTING INFORMATIONAdditional supporting information may be found in theonline version of this article at the publisher’s web site.




leite, a; andrews, nj; thomas, sl (2016) near real...

Documents