1 audit of the columbia suicidality classification project september 13, 2004 meeting of...
DESCRIPTION
3 Objective of the FDA Audit Assess the reproducibility and reliability of the Columbia suicidality classification methodology and processTRANSCRIPT
1
Audit of the Columbia Suicidality Classification Project
September 13, 2004Meeting of Psychopharmacological Drugs Advisory Committee and
Pediatric Advisory Committee
Solomon Iyasu, M.D., M.P.H.Medical Team LeaderDPDD/OCTAP/CDERFood and Drug Administration
2
Outline
• Background
• Objectives of the FDA audit
• FDA audit method, process
• Results of the audit
• Limitations/strengths
• Conclusion
3
Objective of the FDA Audit
• Assess the reproducibility and reliability of the Columbia suicidality classification methodology and process
4
The FDA Audit Methods and Process:Event Narratives
• Columbia reviewed and classified ALL sponsor-submitted event narratives
• FDA audit team received a computerized line listing of these event narratives from the Div. Neuro-pharmacological Drug Products
• Drew a sample of events for review
5
The FDA Audit Methods and Process
• Grouped event narratives into 4 predefined strata
• Selected event narratives for review via a stratified simple random sampling strategy
• Over-sampled difficult to classify and reclassified events
6
The FDA Audit Methods and Process:Strata and Definitions
• Stratum 1: Events reclassified by Columbia to non-suicidal or other events (n=2)
• Stratum 2: Events newly identified and classified by Columbia as possibly suicide related or other categories (n=29)
• Stratum 3: Events that were difficult to classify defined as events with discordant initial independent ratings by Columbia reviewers (n=56)
• Stratum 4: Events that are straight forward cases defined as events with concordant initial ratings by Columbia reviewers (n=336)
7
The FDA Audit Methods and Process:Sample Size and Selection
• Sample size: 64 of the 423 event narratives selected for review (15%)
– Stratum 1: All selected ( n=2)– Stratum 2: 1/3rd selected (n=10)* – Stratum 3: 1/3rd selected (n=19) – Stratum 4: 1/10th selected (n=33)
• Events from stratum 2 – 4 were selected by a simple random technique using a random number table
* Four of the sampled records were also ‘difficult to classify”
8
Composition of the FDA Audit TeamPlanning Group
• Div. of Pediatric Drug Development, OCTAP, CDER– Solomon Iyasu, MD, M.P.H.,
Audit Team Leader– Susan Cummins, M.D., M.P.H. – Rosemary Addy, Project Manager– Kristin Phucas, Project Manager
• Div. of Neuro-pharmacological Drug Products, CDER– Thomas Laughren, M.D.
• OND– Armando Oliva, M.D.
Clinical Reviewers
• Div. of Pediatric Drug Development, OCTAP, CDER– Hari Sachs, M..D., (Pediatrician)– ShaAvhree Buckman, M..D., Ph.D.
(Pediatrician)
• Div. of Neuro-pharmacological Drug Products, CDER– Cara Alfaro, PharmD. (Pharmacist)– Robert Levin, M..D. (Psychiatrist)
Consensus meeting facilitator
• Div of Scientific Investigations – Robert Stasko, M.D. (psychiatrist),
None of the clinical reviewers had previously reviewed the event narratives
9
FDA Audit Team Training
• FDA Audit Team received a 2-hour teleconference training by Kelly Posner about the suicidality classification scale, method and process including– A review of the classification scale, categories and
construct
– A review of examples of case narratives for each of the classification categories
– Independent classification of test event narratives by participants to evaluate the training
10
The FDA Audit Methods and Process:Review Assignments
• Each sampled event randomly assigned to three of four reviewers for independent and blinded review
• Each reviewed and rated 48 events (total of 192 reviews)
• Reviewers blinded to treatment assignment, sponsor, diagnosis and to the Columbia ratings
11
The FDA Audit Methods and ProcessReview Procedures/Instructions
• A memo outlining the procedures of the audit was prepared and provided to the audit team
• Reviewers were not allowed to discuss the events among themselves or with colleagues during the independent review
• Allowed reviewers to call Kelly Posner, Columbia University to obtain clarification on the classification scale but no discussion of the specifics of any case was allowed
• Required reviewers to record on rating form if they consulted with Columbia during the review process
• Modified and pre-coded rating form used
12
13
The FDA Audit Methods and Process• Reviewers returned completed rating forms in sealed envelopes
• Ratings double key entered into an Excel database • Discordant ratings identified and discussed during a consensus
meeting facilitated– by a board certified child and adolescent psychiatrist external
to DNDP– previously not involved in the review of these records or had
knowledge of the Columbia ratings score
• Final consensus ratings entered into an Excel database and compared to the final Columbia ratings.
• Discordant ratings between FDA and Columbia discussed with Columbia to better understand the reason for the differences
14
Criteria for Evaluation of Concordant Ratings
• Columbia Definition of Concordance
– Categories 1, 2, 3, 6, and 10 required exact match
– Categories 4, 5 or 11 all describe self-injurious behavior with no suicidal intent and considered equivalent rating
– Categories 7, 8, 9 or 12 all describe “other” (non-suicidal and non-self-injurious behavior) and are considered equivalent rating
15
Results:Concordance among FDA reviewers
• 47 of 64 initial ratings were concordant
• 17 of 64 initial ratings were discordant
– Event narratives discussed during the FDA reviewer’s consensus meeting
– Final consensus ratings reached for all 17 events
16
Results:Comparison of Final FDA and Columbia Ratings
• Concordance on 57 of 64 final event ratings
– Percent agreement of 89% ( k=0.84)
• Discordance analysis (n=7)
– Severity Hierarchy• (1 or 2) > 6 > 3 > (4 or 5 or 11) > 10
– Compared to Columbia, the FDA audit team classified • six events with higher severity (3 of these events were
classified as ‘not enough information’ vs. ‘other’)• one event with lower severity
17
Limitations
• Neither the quality of the narratives nor the clinical source material for the narratives were evaluated
• Validity of the Columbia classification method not assessed
– there is no gold standard
18
Strengths
• Achieved a high level of concordance between the two independent review teams despite
– Differences in expertise and experience
– Short FDA timeline for training and review
– Intentional over-sampling of difficult to classify events for review
19
Conclusion
• The Columbia Suicidality classification methodology is robust and reproducible.
20
Acknowledgments
• OCTAP/CDER: Hari Sachs, ShaAvhree Buckman, Rosemary Addy, Kristin Phucas, Susan Cummins
• DNDP/CDER: Robert Levin, Cara Alfaro, Thomas Laughren, Alice Hughes, Tarek Hammad
• DSI: Robert Stasko
• OND: Armando Oliva
• Columbia University: Kelly Posner
21
Thank you!