weighing evidence and assessing uncertainty in microbiological risk assessment

FE

DE

RA

L IN

ST

ITU

TE

FO

R R

ISK

AS

SE

SS

ME

NT

Weighing evidence and assessing

uncertainty in microbiological risk

assessment

Approaches for preparing appropriate scientific support

for decision making in complex questions

Matthias Greiner

Shaping the Future of Food Safety, Together.

Milan, 14-16 Oct 2015

Shaping the Future of Food Safety, Together. Milan, 14-16 Oct 2015

Contents

What is evidence, uncertainty and WoE all about and

what are the application areas in microbiological food

safety?

Why do we need to apply these concepts more

systematically?

How can we use these concepts to deal with (the

limitations of our) knowledge?

What

Why

How


Credit

To all members of the current Working Group of the EFSA Scientific

Committee “Guidance on The Use of the Weight of Evidence

Approach in Scientific Assessments”

External experts: Maged Younes, Emilio Benfenati, Qasim Chaudhry, Peter Craig, Geoff

Frampton, Matthias Greiner, Anthony Hardy, Andrew Hart, Christer Hogstrand, Gijs

Kleter, Claude Lambre, Robert Luttik, Alonso Siani

EFSA: Djien Liem (AFSCO), Sybren Vos (ALPHA), Elisa Aiassa (AMU), : Marco Binaglia

and Michaela Hempen (BIOCONTAM), Jaime Aguilera (FEEDAP), Anna Castoldi (FIP)

and Camilla Smeraldi, Antonio Fernandez-Dumont (GMO), Silvia Valtuena-Martinez

(NDA), Andrea Terron (PRAS), Roy Kirby (Quality Management), Jean Lou Dorne

(SCER)


Definitions and context

– Working definitions

– Current practices related to evidence and uncertainty in the area of

microbiological risk assessment

– Application contexts in which the weight-of-evidence (WoE) concept

is used

What


Definitions

Evidence

– The available body of facts or information indicating whether a belief

or proposition is true or valid (Oxford Dictionaries)

Uncertainty

– Used in the EFSA Draft Guidance on Uncertainty as a general term

referring to all types of limitations in the knowledge available to

assessors at the time an assessment is conducted and within the

time and resources agreed for the assessment

Weight-of-evidence (WoE)

– Process in which all of the evidence considered relevant for a risk

assessment is evaluated and weighted (WHO/IPCS, 2009)


Context of statements about evidence and

uncertainty

Search in the EFSA Extranet

– Repository of documents of EFSA and collaborating partners

– Search not restricted to publication date or to a particular panel;

last update 10 Oct 2015

– Text snippets as retrieved (rather than the full documents) used as

text corpus

– Extracting sentences with the terms “evidence” or “uncertainty”

evidence AND food AND (microbio* OR outbreak* OR foodborne) in Documents in the Directory, Collaboration Items

uncertain* AND food AND (microbio* OR outbreak* OR foodborne) in Documents in the Directory, Collaboration Items


Context of EFSA statements about evidence and

uncertainty

Evidence: in the conclusion – Uncertainty: in the scientific background?

Data source:

EFSA Extranet


Context of statements about evidence and

uncertainty

Search in ScienceDirect

– Scientific contents source covering articles from over 2,500 journals

and more than 33,000 book titles

– Search covering freely accessible contents (guest access); not

restricted to publication date; last update 10 Oct 2015

– Using title, abstract and keywords as text corpus

– Extracting sentences with the terms “evidence” or “uncertainty”

TITLE-ABSTR-KEY(evidence AND food AND (microbio* OR outbreak* OR foodborne))

TITLE-ABSTR-KEY(uncertain* AND food AND (microbio* OR outbreak* OR foodborne))


Context of other scientific statements about

evidence and uncertainty

Data source:

ScienceDirect

Evidence: part of discussion – Uncertainty: close to regulatory science?


Weight-of-evidence concept

EFSA Extranet

– Citations in all areas: 7 results

– Citations in microbiological food safety: Zero results

ScienceDirect

– Citations in all areas: 1,636 results

– Citations in microbiological food safety: Zero results

("weight-of-evidence" OR "weight of evidence" OR "evidence synthesis") AND AND food AND (microbio* OR outbreak* OR foodborne) in Documents in the Directory, Collaboration Items

TITLE-ABSTR-KEY({weight-of-evidence} OR {weight of evidence} OR {evidence synthesis}) AND TITLE-ABSTR-KEY(food AND (microbio* OR outbreak* OR foodborne))


Weight-of-evidence concept

WHO Microbiological Risk Assessment Series

– WoE for causality inferences. Assessing quantity, quality and nature of the

results available from various study types; pathogen characteristics;

biological mechanisms; extrapolating from animal or in vitro studies to

humans (WHO, 2003)

– Weights proportional to sample size, expert beliefs and uncertainty (WHO,

2008)

– WoE will become increasingly prominent in risk assessments of

microbiological pathogens in food (WHO, 2009)

Advocating the WoE approach for integrating of

evidence from multiple sources


Application of the weight-of-evidence concept in

microbiological food safety and related areas

Foodborne outbreaks:

Support of emergency

measures

Scientific evidence on:

Population, agent,

vehicle, source,

adverse effect

Examples:

Health Canada, 2011

Vik et al., 2014

Food safety

intervention:

Decision support


Problem, intervention,

outcome

Examples:

Fazil et al., 2008

Other applications: a) Dose-response models

b) Water management

c) Risk ranking


Various aspects depending

on the problem

Examples:

a) Moon et al., 2005

b) Olivieri et al., 2014

c) EFSA BIOHAZ, 2015


Why do we need to apply these concepts more

systematically?

– Complexity of questions in microbiological risk assessment

– Dealing with limits of our current knowledge

Why


Complexity in microbiological food safety

Epidemiological triangle Host

– Behaviours

– Acquired or inherent

susceptibility or vulnerability

Agent

– Species /genotype /phenotype

characteristics

– Gene transfer

Environment

– Biotic and abiotic factors for

growth

Environment

Agent

Host

Agent


Complexity in microbiological food safety

Host

Microbiome

Infectome

Adapted from Bogdanos et al., 2013


How can we use these concepts to deal with

(the limitations of our) knowledge?

– Risk assessment – an evidenced-based approach

– Must make limitations and uncertainties explicit

– Use WoE in cases of alternative sources of evidence

How


Conceptual model of a food safety question


Uncertainty assessment

Risk question

Scenario

Model

Model parameters

Data

Calculations/Simulations

– Describe limits of current

knowledge

– Safeguard against over-

interpretation

http://www.bfr.bund.de/cm/350/guidelines-on-

uncertainty-analysis-in-exposure-assessments.pdf

http://www.efsa.europa.eu/sites/default/files/consultati

on/150618.pdf

http://www.bfr.bund.de/cm/350/guidelines-on-uncertainty-analysis-in-exposure-assessments.pdf














http://www.efsa.europa.eu/sites/default/files/consultation/150618.pdf




Combining evidence (I)

Meta-analysis situation

Multiple primary studies on the same question

and comparable outcome metrics (e.g. odds

ratios from epidemiological studies; sensitivity and

specificity of diagnostic tests)

– Be systematic use systematic review

(PRISMA standard*)

– Use an instrument for assessing study quality*

– Use strict inclusion/exclusion criteria

– Investigate and account for heterogeneity

– Use established statistical models for summary

estimates and confidence intervals

*http://www.equator-network.org/ http://www.efsa.europa.eu/sites/de

fault/files/scientific_output/files/mai

n_documents/432.pdf

http://www.equator-network.org/




http://www.efsa.europa.eu/sites/default/files/scientific_output/files/main_documents/432.pdf





Combining evidence (II)

Weight-of-evidence situation

Combining information from primary studies with different types of

observations (e.g. in vivo, in vitro, in silico, epidemiological), different

study organisms (e.g. human and animal)

– Resembles the approach for meta-analysis (use systematic approach

as far as possible)

– ...

– No established statistical tools available for integrating evidences

(currently under review by EFSA working group)


Combining evidence (II)

Weight-of-evidence

Line of

evidence

Internal

validity

Relevance

for question Weight Outcome

in vitro ? ? ? ?

in vivo ? ? ? ?

epidemiology ? ? ? ?


OUTBREAK INVESTIGATIONS AND

DOSE-RESPONSE MODELLING

Alternative weight-of-evidence approaches


Health Canada, 2011 (clipping of p. 12)

Grading

of the

evidence


Binomial dose-response model

Dose

– Number of bacterial cells ingested: n

Response

– Probability of infection at dose n: P(n)

Model

P(n) = 1 – (1 – p)n

p is the shape parameter (probability that a single cell causes infection)

Assumptions

– r is constant in the population of bacterial cells

Use

– If the dose n is known exactly

Using

Evidence in

modelling


Extensions of the binomial model

Exponential

– Use if n is not known precisely; n ~ Poisson(●)

Beta-binomial

– p is not known precisely; p ~ beta(●,●)

Beta-Poisson

– use if p and n

are not known precisely;

p ~ beta(●,●)

n ~ Poisson(●)

● = distribution

parameter

for simulation


Evidence-based choice or combination of models

Classical approach

– Using biological reasoning

about underlying assumptions

– Asymptotic behaviour

– Choosing the model with the

best fit

Binomial Beta-Poisson

WoE approach

– Use all models and combine

results mathematically

– “Evaluate the relative

plausibility of each fitted model

by a weight of evidence

relative to the selected best

model” (Moon et al., 2005.

Model Averaging Using the

Kullback Information Criterion

in Estimating Effective Doses

for Microbial Infection and

Illness. Risk Analysis 25/5,

1147-1159)

0 500 1000 1500 2000

0.0

0.2

0.4

0.6

0.8

1.0

n

P(n)

r=0.003

r=0.002

r=0.001

d

P(d)

1021 10

210

410

610

810

1010

12

0

0.2

0.4

0.6

0.8

1

0.2 0.4

0.2 40

0.8 4000


Conclusions

Internal validity and relevance for the question (external validity)

– Can be part of a uncertainty assessment

– Can be used to derived weights in a WoE approach

`The two methodologies weighing of evidence and uncertainty

assessment are complementary and have a common overall goal,

which is to provide the best possible basis for science-based

decision making

Line of

evidence Internal validity

Relevance for

question Weight Outcome

in vitro ? ? ? ?

in vivo ? ? ? ?

epidemiology ? ? ? ?


References

Bogdanos DP et al. (2013). Infectome: A platform to trace infectious triggers of autoimmunity. Autoimmun Rev. 12:726-40.

EFSA BIOHAZ Panel (2015). Scientific Opinion on the development of a risk ranking toolbox for the EFSA BIOHAZ Panel.

EFSA Journal 2015;13: 3939, 131 pp.

Fazil A et al. (2008). Choices, choices: the application of multi-criteria decision analysis to a food safety decision-making

problem. Journal of Food Protection 71, 2323-2333.

Health Canada (2011). Public Health Agency of Canada; Canadian Food Inspection Agency. Weight of evidence: factors to

consider for appropriate and timely action in a foodborne illness outbreak investigation. Ottawa, ON: Minister of Health.

Moon H et al. (2005). Model averaging using the Kullback information criterion in estimating effective doses for microbial

infection and illness. Risk Analysis 25/5, 1147-1159.

Olivieri AW et al. (2014). Risk-Based Review of California’s Water-Recycling Criteria for Agricultural Irrigation. Journal of

Environmental Engineering, 2014.140.

Vik J et al. (2014). Summary: weight of evidence - factors to consider when investigating a food-borne illness outbreak.

Canada Communicable Disease Report 40, 303.

WHO (2003). Hazard characterization for pathogens in food and water: guidelines. Microbiological Risk Assessment Series,

No. 3. WHO, Geneva, 76 pp.

WHO (2008). Exposure assessment of microbiological hazards in food. Microbiological Risk Assessment Series, No. 7.

WHO, Geneva, 102 pp.

WHO (2009). Risk Characterization of microbiological hazards in food. Microbiological Risk Assessment Series, No. 17

WHO, Geneva, 135 pp.

WHO/IPCS (2009). EHC 240: Principles and methods for the risk assessment of chemicals in food. Annex 1, 45 pp.

FE

DE

RA

L IN

ST

ITU

TE

FO

R R

ISK

AS

SE

SS

ME

NT

FE

DE

RA

L IN

ST

ITU

TE

FO

R R

ISK

AS

SE

SS

ME

NT

Thank you

Matthias Greiner

Federal Institute for Risk Assessment, and

University of Veterinary Medicine Hannover (TiHo),

Foundation

Tel. +49 30 18412 3297 Fax +49 30 18412 2958

matthias.greiner @bfr.bund.de www.bfr.bund.de