Innovative Solutions

Sound Science

The Methodology and Challenges of the Caffeine Systematic Review: An Expert Perspective

Dr. Daniele Wikoff - ToxStrategies

Experimental Biology 2017

Disclosure

2

•Member of the ILSI North America Caffeine Systematic Review

Team

• ILSI North America provided support to attend this meeting

Multidisciplinary Team

3

Daniele Wikoff, Brian T. Welsh, Rayetta Henderson, Gregory P.

Brorby, Janice Britt, Esther Myers, Jeffrey Goldberger, Harris R.

Lieberman, Charles O’Brien, Jennifer Peck, Milton Tennebein,

Connie Weaver, Seneca Harvey, Jonathan Urban, Candace

Doepker

Outline – “Methods” and Challenges

4

•Initiating the review

•Finding and assessing individual studies

•Integration and synthesis

•Challenges and solutions

Initiating the Review

5

IOM: Initiating the Review

6

Topic Formulation: Nawrot et al. 2003

7

• Highly cited reference

• Authors from Health Canada

• Review of multiple endpoints

• Guidance on “safe” level

• 400 mg/day in healthy adults

• Format is easy to follow for the

reader

Topic Formulation: Overall Analytical Framework

8

Populations

Pharmacokinetics/pharmacodynamics included as a contextual topic

Healthy Adolescents (12-19)

Healthy Children (3-11)

Healthy Adults

Healthy Pregnant Women

< 2.5 mg/kg-day

<400 mg/day

<300 mg/day

Acute Toxicity

Bone and Calcium

Cardiovascular

Behavior

Reproduction &

Development

Comparator/Exposure Outcome

PECO Questions

9

Population Healthy

Adults

Pregnant

Women

Adolescents Children

Exposure > 400

mg/day

> 300

mg/day

> 2.5

mg/kg-day

> 2.5

mg/kg-day

Comparator ≤ 400

mg/day

≤ 300

mg/day

> 2.5

mg/kg-day

≤ 2.5

mg/kg-day

Outcome Acute, Reproductive and Developmental,

Cardiovascular, Behavior, Bone and Calcium

For [population], is caffeine intake above [dose/exposure], compared to intakes

[dose/comparator] or less, associated with adverse effects on [outcome]?

Outcomes and Endpoints: Terminology Clarification

10

Endpoint: Specific E.g.:

Mortality Heart Rate Cholesterol

Morbidity Heart Rate VariabilityAortic stiffness/wave

velocity

Blood Pressure Cerebral blood flow Catecholamines

Endothelial function Electrical activityOther hemodynamic

measures

Homocysteine Ventricular function

Outcome: Broad (e.g., Cardiovascular)

Finding Individual Studies

11

IOM: Finding and Assessing Individual Studies

12

Overview of Literature Search Strategy

13

• Three databases searched January 1, 2001 to June 8, 2015

• Outcome- and database-

specific syntax developed

with librarian (concatenated;

in protocols)

• Search results imported into

DistillerSR

DistillerSR

14

Software designed to perform, track, audit, and

report systematic review projects

• Web-based platform (allows for multiple users, team review, coordination, etc.)

• Documentation of inclusion/exclusion (audit log tracks all changes)

• Custom forms and reporting

Screening Approach

15

PilotingConduct Search

Remove Duplicates

Initial Screen (Titles and Abstracts)

QA, Committee

Review

Subject Expert Review

Full-Text Screen

Subject Expert Review

Included Studies

Screening Form Example

Abstract with

keyword highlighting

Dynamic form

and questions

Article title,

journal, and

other info

16

Study Screening and Selection – Inclusion/Exclusion

17

• Quantitate estimate of caffeine exposure• e.g. Coffee, tea, cola, energy drink, supplements

• Must have reported an effect for an Outcome of interest

• Healthy population only• i.e. not having been hospitalized or diagnosed with disease and/or receiving medical treatment for a disease

at the time of the study

• Caffeine must have been studied alone, or in one of the approved forms• Mixtures with alcohol, nicotine, acetaminophen, or other drugs were excluded

• Available in English

• Case reports included for Acute only

• Must present original data; reviews excluded• Exception: Meta-analyses were included

Full Text Screening and Data Extraction

18

• Full text versions obtained

• Study first checked against inclusion/exclusion

criteria

• Generate evidence tables (generated in

DistillerSR; reported in AHRQ-SRDR)

1. Basic information as reported by author (direct

extraction of information)

2. Customized information (some interpretation)

– Exposure data need to be standardized?

– Selection of data used for comparison

» No observed effect level (NOEL)

» Lowest observed effect level (LOEL)

Standardized Caffeine Content (Supplemental Materials)

19

24 24.836.8

47.2

80

95

0

20

40

60

80

100

Soft Drink(cola)

Tea(green)

Soft Drink(citrus)

Tea(black)

EnergyDrink

Coffee

mg

Ca

ffe

ine

pe

r 8

oz

Caffeine Source

Did author conduct analysis or report based on caffeine? If no,

Mitchell et al. 2014; 2015

Assessing Individual Studies

20

IOM: Finding and Assessing Individual Studies

21

Individual study assessment

22

• Internal validity (risk of bias)

Aims to discern if anything in the study design

compromises the credibility between exposure

and outcome

• External validity

Evaluates generalizability; considered

directness

• Level of adversity

Categorizes the level of importance in decision-

making and assists in weighing endpoints

OHAT Risk of

Bias Rating

Tool

OHAT

Handbook

(GRADE)

GRADE (Guyatt

et al., 2011)

OHAT Risk of Bias Tool

23

• "OHAT Risk of Bias Rating Tool for

Human and Animal Studies” (2015)

• Set of 11 questions

• Grouped under 6 types of bias: selection,

confounding, performance,

attrition/exclusion, detections, and selective

reporting

• Applied based on study design

• Questions are rated by selecting

among 4 possible answers

RoB (Internal Validity – “Quality”)

24

+2: Definitely low

+1: Probably low

-1: Probably high

-2: Definitely high

2. For each study and each

question, apply responses based

on potential for risk of bias:

OHAT Handbook (January 9, 2015)

Table 9. Example of a Visual Summary of Risk of Bias Ratings for Animal Studies

Risk of Bias Question Stu

dy

1

Stu

dy

2

Stu

dy

3

Stu

dy

4

Stu

dy

5

Stu

dy

6

Stu

dy

7

Stu

dy

8

Stu

dy

9

Stu

dy

10

Stu

dy

11

Stu

dy

12

Stu

dy

13

Stu

dy

14

Stu

dy

15

Stu

dy

16

Stu

dy

17

Stu

dy

18

Stu

dy

19

Randomization + − ++ ++ − ++ + + ++ − − − + + + − − + ++

Allocation concealment − − − − − − + − − − − − − − − − − − −

Confounding (design/analysis) ++ + ++ ++ ++ + ++ ++ ++ ++ + ++ ++ + − − − − ++

Unintended exposure + + + + + + + + + + + + + + + + + + +

Identical experimental conditions ++ ++ + + ++ ++ ++ ++ ++ + ++ + ++ ++ ++ ++ ++ ++ ++

Adhere to protocol + + + + − + + + + + + + + + + + + + +

Blinding of researchers during study − − − − − − + − − − − − − − − − − − −

Missing outcome data − + ++ ++ −− − + − − + −− − − + ++ + ++ + ++

Assessment of confounding variables + + ++ ++ ++ − + + ++ ++ + + + ++ ++ − + + ++

Exposure characterization ++ − + + − − + + − − − + + + + + + − +

Outcome assessment + + + + + + ++ + + − ++ + + + + + + + +

Blinding of outcome assessors + + + + ++ + + + + + + + −− + ++ + + + +

Outcome reporting + + + ++ −− + + + + − + + −− + + + ++ − +

Key:

Definitely low risk of bias ++

Probably low risk of bias +

Probably high risk of bias −

Definitely high risk of bias −−

Studies are evaluated on all applicable risk of bias questions based on study design. The rating or answer to each risk of bias question is selected on an outcome basis prior to determining the tier from 4 options: definitely low risk of bias (++), probably low risk of bias (+), probably high risk of bias (-), or definitely high risk of bias (--).

52

4. Array the responses in tabular format to

observe trends in the body of evidence

Source: NTP OHAT (2015)

1. Evaluate RoB question based

on study type.

• (Q1) Randomization

• (Q2) Allocation Concealment

• (Q3) Identical experimental conditions

• (Q4) Confounding (design/analysis)

• (Q6) Blinding during study

• (Q7) Missing outcome data

• (Q8) Confidence in exposure

• (Q9) Outcome assessment

• (Q10) Outcome reporting

• (Q11) Other (Exposure/purity)

3. Evaluate responses for individual studies

when considering findings

Integration and Synthesis

25

IOM Framework for Synthesizing the Body of Evidence

26

Qualitative Endpoint and Outcome Syntheses (IOM Standard 4.2)

27

• General characterization of literature identified

• Endpoint characterizations

• Summary of individual studies

– Plots of findings

– Risk of bias figures

• Body of evidence assessment (including confidence) and conclusion (tabular/narrative)

• Outcome characterization

• Body of evidence assessment (including confidence) and conclusion (tabular/narrative)

• Strengths, limitations, uncertainties

• Data gaps

Integration of the OHAT Framework

28

• Handbook for conducting SRs

and evidence integration

• Process for qualitatively

evaluating data

• Accommodated our

anticipated dataset

• Can be tailored/modified by

endpoint as appropriate

Synthesize Evidence and Rate Confidence

29

Confidence in Body of Literature (Endpoint/Outcome) (from OHAT; consistent with IOM)

30

Endpoint

No. of

Studie

s

Initial

ConfidenceOverall RoB Indirectness Magnitude Confounding Consistency

Final Confidence

Rating

Rating

Based on study

type and study

features (OHAT,

2015)

Domain-based

evaluation of

risk of bias per

the OHAT RoB

tool (OHAT,

2015)

Was the study

designed to

evaluate the

PECO?

Strength of

effect

(when effect

observed below

the

comparator)

Were plausible

confounders that

would change

the observed

effect accounted

for?

Were findings

consistent in

demonstrating

effects or lack of

effects at or

below the

comparator?

What is the overall

rating when factors

that increase or

decrease confidence

were considered?

Endpoint #High/ Moderate/

Low/ Very Low↓ / - / ↑ ↓ / - / ↑ ↓ / - / ↑ ↓ / - / ↑ ↓ / - / ↑

High/ Moderate/

Low/ Very Low

↑ increased confidence - no change to confidence ↓ decreased confidence

Key Considerations in Evaluation of WoE and Developing Conclusions

31

• Findings relative to the comparator

• Individual study context (e.g., internal and external validity)

• Confidence in evidence base (e.g., consistency,

magnitude)

• Level of adversity

Reported in a common structure (tabular, narrative)

Endpoints

Outcomes

Overall

Challenges and Solutions

32

PECO for multiple outcomes and endpoints

33

Population Healthy

Adults

Healthy

Pregnant

Women

Healthy

Adolescents

Healthy

Children

Exposure > # mg/day > # mg/day > # mg/kg-

day

> # mg/kg-

day

Comparator ≤ # mg/day ≤ # mg/day ≤ # mg/kg-

day

≤ # mg/kg-

day

Outcome Adverse effect (five outcomes)

• For [population], is caffeine intake above [exposure], compared

to intakes [comparator] or less, associated with adverse effects

on [outcome]?

Population: “healthy”

34

• Issue: Determining inclusion/exclusion based on healthy populations• Compromised physical function; caffeine as

a therapeutic

• Physicians and SR expert played a key role in helping with this determination

1. Define healthy• Subjects who were not specifically described

as hospitalized, diagnosed with disease, and/or receiving medical treatment for a disease at the time of the study

2. Impact on SR• Many abstracts/papers excluded based on

definition (retained if included healthy control arm or similar)

– E.g., diabetes, Parkinson’s

Example:

Outcome: “adverse”

35

• Issue: Nawrot et al. 2003 update to adverse effects

1. Define adverse• Use effects in Nawrot?

• Subjective, difficult to draw a line

• Decision: be conservative and comprehensive

2. Impact on SRs• Pilot searches utilized “adverse” term (and

related) – too restrictive

• Extracted information based on author conclusion as well as analyst conclusion

• Characterization of effects in subgroups (e.g., physiological/clinical, order in progression of effect, etc.)

Outcome: “adverse” – an added layer of complexity

36

• Issue: Caffeine exposure has been associated with benefits and adverse effects

• Adverse effects in studies that are evaluating benefits? (e.g., RCTs)

• Controlled exposure, etc.

• OR lack of adverse effects in studies evaluating benefits?

• Null findings on potentially adverse outcomes (e.g., heart rate)

• Decision

• Include: studies reporting data associated with adverse effects within a benefit/therapy study.

• Exclude: Studies assessing only beneficial or therapeutic endpoints or outcomes following exposure to caffeine.

Outcome: multiple endpoints, multiple outcomes

37

• Issue: Outcome of each review associated with many endpoints

• Decision: Comprehensive and inclusive (consistent w/ characterization of hazards)

• Impact: Frameworks and implementation activities need to be accommodating

Cardiovascular: Endpoint Keywords Identified From Abstracts

acute stiffness (pulse

wave velocity)chest pain and palpitation

stroke volume and

cardiac contractility

arrhythmia cholesterolsupra ventricular

dysrhythmias

atrial fibrillationendothelial

function/performanceventricular function

blood pressure heart rate

cardiovascular disease hypertension

cerebral blood flow myocardial blood flow

Data integration challenges

38

• Accommodation of framework to our question (beyond classification of hazard,

but rather evaluation of a specific dose)

Challenges with reporting

39

1. Length of the paper

• Balance of rigor and transparency with “standard” reporting

• Utilized supplemental materials and AHRQ SRSR

2. Conclusions – developing and reporting

• Clarity within manuscript

• Stakeholders are diverse

• Transparency & breadth of topics made it difficult to be “simple”

Project Flow and Timeline

40

Project Flow

41

PHASE 1. Initiate Systematic Review

1

2

3

4

Define project team & SAB

Determine project objectives

Develop a systematic review protocol

Select appropriate tools for review implementation

PHASE 2. Find & Assess Individual Studies

PHASE 3. Synthesize the Body of Evidence

PHASE 4. Report Systematic Review

1

2

3

4

Conduct comprehensive systematic literature search

Screen & select studies; document data collection

Report findings of literature search

Refine outline (i.e., topic areas)

Develop specific systematic review process

Obtain literature

5

6

1

2

Systematically assess the body of evidence (by topic area)

Conduct a qualitative synthesis (by topic area)

1

2

Prepare draft manuscript

Prepare updated draft for submission (based on SAB comments)

3 Peer-review journal submission and publication

SAB

Meeting

SAB

Meeting

SAB

Meeting

SAB

Meeting

~4 months from Nov 1, 2014 to

March 1, 2015

~11 months from March 1, 2015

to Feb 1, 2016

~8 months from Feb 1, 2016 to

Oct 1, 2016

~5.5 months from Oct 1, 2016 to

March 20, 2017

Questions?

42