Data Mining AERSData Mining AERSFDA’s (Spontaneous) Adverse Event Reporting SystemFDA’s (Spontaneous) Adverse Event Reporting System

Division of Drug Risk EvaluationDivision of Drug Risk EvaluationOffice of Drug SafetyOffice of Drug Safety

Data Mining AERSData Mining AERSFDA’s (Spontaneous) Adverse Event Reporting SystemFDA’s (Spontaneous) Adverse Event Reporting System

Division of Drug Risk EvaluationDivision of Drug Risk EvaluationOffice of Drug SafetyOffice of Drug Safety

Carolyn McCloskey, M.D., M.P.H.

Drug Safety and Risk Management Advisory CommitteeMay 18, 2005

Carolyn McCloskey, M.D., M.P.H.

Drug Safety and Risk Management Advisory CommitteeMay 18, 2005

2

OutlineOutlineOutlineOutline1. Brief History of data mining (DM) activities

at the FDA

2. Current Use in the Division of Drug Risk Evaluation (DDRE) – CRADA

• Application Development - WebVDME• Pilot - Examples and Selected

Conclusions• Other CRADA Activities

3. Future Directions in DDRE Pharmacovigilance

1. Brief History of data mining (DM) activities at the FDA

2. Current Use in the Division of Drug Risk Evaluation (DDRE) – CRADA

• Application Development - WebVDME• Pilot - Examples and Selected

Conclusions• Other CRADA Activities

3. Future Directions in DDRE Pharmacovigilance

3

Historical OverviewHistorical OverviewHistorical OverviewHistorical Overview

February 1998: Office of Women’s Health Grant (Ana Szarfman)

March 2003 – July 2005:

Cooperative Research & Development Agreement (CRADA)

Research in more advanced methodology (Drug-drug interaction & logistic regression modeling)

4

CRADA - Cooperative Research and Development

Agreement

CRADA - Cooperative Research and Development

Agreement

WebWeb VVisual isual DData ata MMining ining EEnvironment nvironment (WebVDME)(WebVDME)

With Lincoln Technologies, Inc.

March 2003 – July 2005

WebWeb VVisual isual DData ata MMining ining EEnvironment nvironment (WebVDME)(WebVDME)

With Lincoln Technologies, Inc.

March 2003 – July 2005

5

CRADA ObjectivesCRADA ObjectivesCRADA ObjectivesCRADA Objectives

• User-friendly application

• Web-based environment

• Performance Evaluations by User Groups

• Training

• Continued development and refinement

• User-friendly application

• Web-based environment

• Performance Evaluations by User Groups

• Training

• Continued development and refinement

6

Empirical Bayes Geometric Empirical Bayes Geometric Mean (EBGM)Mean (EBGM)

Empirical Bayes Geometric Empirical Bayes Geometric Mean (EBGM)Mean (EBGM)

• An observed/expected score

• Adjusts for sampling variation (e.g., sample size)

• No adjustment for reporting bias

• Allows data stratification in DM software

– Standard stratification: gender, age group, year

• An observed/expected score

• Adjusts for sampling variation (e.g., sample size)

• No adjustment for reporting bias

• Allows data stratification in DM software

– Standard stratification: gender, age group, year

7

EB05 – EB95 IntervalEB05 – EB95 IntervalEB05 – EB95 IntervalEB05 – EB95 Interval

• Interval in which the EBGM score could be found because of sampling variability

• EB05 is the lower bound

• EB95 is the upper bound

• 90% probability of EBGM occurring between EB05 and EB95

• Interval in which the EBGM score could be found because of sampling variability

• EB05 is the lower bound

• EB95 is the upper bound

• 90% probability of EBGM occurring between EB05 and EB95

8

Example of Sampling Variability Example of Sampling Variability Adjustment Adjustment (for small numbers)(for small numbers)

Example of Sampling Variability Example of Sampling Variability Adjustment Adjustment (for small numbers)(for small numbers)

Adverse Event (AE)

Observed Count (N)

Expected Count (E)

Obs/Exp* N/E (RR)

EBGM** EB05 EB95

Myalgia 1,665 334 4.99 4.97 4.78 5.18

SpinalOsteoarthritis 17 3 6.16 4.54 3.03 6.60

* Obs = Observed; Exp = expected

** EBGM = Empirical Bayes Geometric Mean - adjusts for sampling variability

9

CRADA Pre-PilotCRADA Pre-PilotPerformance EvaluationsPerformance Evaluations

May 2003 – October 2003May 2003 – October 2003

CRADA Pre-PilotCRADA Pre-PilotPerformance EvaluationsPerformance Evaluations

May 2003 – October 2003May 2003 – October 2003

• WebVDME record retrieval validation with AERS case retrieval

Multiple trade & ingredient nomenclature

Drug assignment allocations (suspect & concomitant)

Duplicate removal logic

• OIT system performance evaluations

• WebVDME record retrieval validation with AERS case retrieval

Multiple trade & ingredient nomenclature

Drug assignment allocations (suspect & concomitant)

Duplicate removal logic

• OIT system performance evaluations

10

CRADA PilotCRADA PilotMedical Safety EvaluatorsMedical Safety Evaluators

CRADA PilotCRADA PilotMedical Safety EvaluatorsMedical Safety Evaluators

• Evaluated data mining scores for drugs and biologics

Indication vs. new signal Labeled vs. unlabeled Innocent bystanders or concomitant

medications Drug names Safety Evaluators’ ease of use

• Evaluated data mining scores for drugs and biologics

Indication vs. new signal Labeled vs. unlabeled Innocent bystanders or concomitant

medications Drug names Safety Evaluators’ ease of use

11

CRADA PilotCRADA PilotEpidemiologistsEpidemiologists

CRADA PilotCRADA PilotEpidemiologistsEpidemiologists

• Evaluated

Temporal trends Drug name selections Suspect & Concomitant selections Stratification Signal strengths Epidemiologists’ ease of use

• Evaluated

Temporal trends Drug name selections Suspect & Concomitant selections Stratification Signal strengths Epidemiologists’ ease of use

12

Pilot ExamplesPilot ExamplesPilot ExamplesPilot Examples

• New vs. Old Drug

• EBGM Rankings & Confidence Intervals

• New vs. Old Drug

• EBGM Rankings & Confidence Intervals

13

New Drug (1 Year)New Drug (1 Year)EBGM (EB05-EB95)EBGM (EB05-EB95)

New Drug (1 Year)New Drug (1 Year)EBGM (EB05-EB95)EBGM (EB05-EB95)

0

10

20

30

40

50

60

EB

05

-EB

GM

-EB

95

0

10

20

30

40

50

60

EB

05

-EB

GM

-EB

95

EB05 =2.0

14

OLDER DRUG (>10 Years)OLDER DRUG (>10 Years)EBGM (EB05-EB95)EBGM (EB05-EB95)

OLDER DRUG (>10 Years)OLDER DRUG (>10 Years)EBGM (EB05-EB95)EBGM (EB05-EB95)

0

10

20

30

40

50

60

70

80

EB

05

-EB

GM

-EB

95

EB05 =2.0

15

Selected Pilot Conclusions - 1Selected Pilot Conclusions - 1Selected Pilot Conclusions - 1Selected Pilot Conclusions - 1

• WebVDME DM - Statistical tool assists in identifying unusual patterns with AERS data but

–! Patterns Need Interpretation!

• WebVDME DM - Statistical tool assists in identifying unusual patterns with AERS data but

–! Patterns Need Interpretation!

16

Selected Pilot Conclusions - 2Selected Pilot Conclusions - 2Selected Pilot Conclusions - 2Selected Pilot Conclusions - 2

• Knowledge of data in database imperative to interpret

–Clinical & pharmacologic activities of drug

–Other - reporting disproportionalities which also reflect limitations of AERS data

• Knowledge of data in database imperative to interpret

–Clinical & pharmacologic activities of drug

–Other - reporting disproportionalities which also reflect limitations of AERS data

17

AERS DATABASE AERS DATABASE LIMITATIONSLIMITATIONS

AERS DATABASE AERS DATABASE LIMITATIONSLIMITATIONS

18

Continuing CRADA Activities - DDREContinuing CRADA Activities - DDREMarch 2004 - PresentMarch 2004 - Present

Continuing CRADA Activities - DDREContinuing CRADA Activities - DDREMarch 2004 - PresentMarch 2004 - Present

• Access by interested Reviewers to WebVDME– Training– Application

• Refinements addressing– Technical problems identified– Customization by user needs

• Access by interested Reviewers to WebVDME– Training– Application

• Refinements addressing– Technical problems identified– Customization by user needs

19

Summary – 1 Summary – 1 DM Signals in DDREDM Signals in DDRE

Summary – 1 Summary – 1 DM Signals in DDREDM Signals in DDRE

• Assist in prioritizing evaluations of case series

• Identify associations, NOT a cause or degree of risk

• Reflect limitations of data

• Assist in prioritizing evaluations of case series

• Identify associations, NOT a cause or degree of risk

• Reflect limitations of data

20

Summary – 2 Summary – 2 DM Signals in DDREDM Signals in DDRE

Summary – 2 Summary – 2 DM Signals in DDREDM Signals in DDRE

• Threshold a compromise between sensitivity and specificity (false positives & negatives)

Absence of a DM signal ≠ absence of a drug-event association

Magnitude of DM scores ≠ magnitude of risk

• Always require clinical case report and reporting bias evaluation

• Threshold a compromise between sensitivity and specificity (false positives & negatives)

Absence of a DM signal ≠ absence of a drug-event association

Magnitude of DM scores ≠ magnitude of risk

• Always require clinical case report and reporting bias evaluation

21

Future Directions of DMFuture Directions of DMFuture Directions of DMFuture Directions of DM

• Prospective signal detection

• Parallel use with traditional pharmacovigilance methods in DDRE

• Continued research in more advanced methodology (Drug-drug interaction & logistic regression modeling)

• Prospective signal detection

• Parallel use with traditional pharmacovigilance methods in DDRE

• Continued research in more advanced methodology (Drug-drug interaction & logistic regression modeling)

22

AcknowledgmentsAcknowledgmentsAcknowledgmentsAcknowledgments

• Division of Drug Risk Evaluation– Rita Ouellet-Hellstrom, Ph.D., M.P.H.– Mary Willy, Ph.D.– Mark Avigan, M.D., C.M.

• Division of Drug Risk Evaluation– Rita Ouellet-Hellstrom, Ph.D., M.P.H.– Mary Willy, Ph.D.– Mark Avigan, M.D., C.M.

THANK YOUTHANK YOUTHANK YOUTHANK YOU