grace stakeholder consultation on good review practice in ... good review... · grace stakeholder...

1

GRACE Stakeholder Consultation on Good Review Practice in GMO Impact Assessment

Part 4: Draft Protocols for Systematic Reviews and Evidence Maps

FP7 Collaborative Project GRACE 311957

01/10/2014

2

Contact: Gloria Adduci, [email protected], Armin Spök, [email protected]

“GMO Risk Assessment and Communication of Evidence”, GRACE is a Collaborative Project of the

Seventh Framework Programme of the European Community for Research, Technological

Development and Demonstration Activities.

Grant agreement no: 311957

Project duration: 1st June 2012 – 30

th November 2015GRACE website: www.grace-fp7.eu

mailto:[email protected]

mailto:[email protected]

http://www.grace-fp7.eu/

3

Table of Contents

List of abbreviations ......................................................................................................................................... 5

This Report ......................................................................................................................................................... 7

Chapter 1: Environmental Impacts of GMOs ................................................................................................ 9

General Comments ................................................................................................................................ 9

1) Systematic Review “Does the growing of Bt maize change populations or ecological

functions of non-target animals compared to the growing of conventional non-GM maize? .... 11

2) Systematic Review “Environmental effects of the cultivation of GM herbicide tolerant

crops” ..................................................................................................................................................... 18

3) Systematic Review “Are population abundances of soil invertebrates changed by Bt

crops compared with conventional crops?” ...................................................................................... 28

4) Systematic Review:“Target Insect Resistance Development and Resistance

Management of Bt-Crops – Review Question 2”. Review Question: “Are data on base line

susceptibility of different lepidopteran/coleopteran maize pests to Bt-proteins available?” ..... 32

5) Evidence Map: “Target Insect Resistance Development and Resistance Management of

Bt-Crops – Review Question 4”. Review Question: “Is the inheritance of resistance alleles

fully recessive in populations of lepidopteran/coleopteran maize pest?” .................................... 34

6) Systematic Review “Effects of Bt crops on soil microorganisms” (NTO review question 3).

Review Question: “Are soil microbial endpoints changed by Bt crops compared with

conventional crops?” ............................................................................................................................ 37

Chapter 2: Health Impacts of GMOs ............................................................................................................ 41

General Comments .............................................................................................................................. 41

1) Evidence Map “What are the characteristics of toxicity studies in which animals receiving

newly expressed proteins from GM crops are compared to animals that are administered

appropriate controls without these proteins?” .................................................................................. 41

4

2) Evidence Map “What are the characteristics of comparative studies of changes in the

levels of key chemical crop constituents in GM crops compared to non-GM crops?” ............... 50

3) Evidence Map “What are the characteristics of repeated-dose feeding studies in which

experimental animals receiving whole food or feed products derived from genetically modified

(GM) crops are compared to animals receiving conventional non-GM counterparts?” ............. 58

4) Evidence Map “What are the characteristics of studies on the risk of allergic sensitization

and elicitation in humans and animals exposed to an allergenic plant that has been

genetically modified as compared to individuals exposed to the non-genetically-modified plant

counterpart?” ......................................................................................................................................... 63

Chapter 3: Socioeconomic Impacts of GMOs ............................................................................................ 70

General Comments .............................................................................................................................. 70

1) What is the impact of the introduction of GM crops on the welfare effects in different

countries in comparison to a situation where there are restrictions on GM cultivation? ........... 72

2) What is the impact on GM regulation of different political actors and other drivers in the EU

in comparison to the US? .................................................................................................................... 73

Revisions, peer-review and final versions of the Draft Systematic Review Protocols ....................... 75

Annex ................................................................................................................................................................. 78

Stakeholder comments received ....................................................................................................... 78

5

List of abbreviations

AGRIS International Information System for the Agricultural science and technology

AUS Australia

BCH Biosafety Clearing House

Bt Bacillus thuringiensis

Corg Organic Carbon

CA Competent Authority

CAB Commonwealth Agricultural Bureau

CADIMA Central Access Database for the Impact Assessment of Crop Genetic

Improvement Technologies

CAN Canada

CBD Convention on Biological Diversity

CEE Collaboration for Environmental Evidence

CERA Cyprus Energy Regulatory Authority

CSO Civil Society Organisation

DIGE Difference gel electrophoresis

DNA Deoxyribonucleic acid

EAST Enzyme Allergosorbent Test

EC European Commission

EFSA European Food Safety Authority

ELISA Enzyme-linked immunosorbent assay

EM Evidence Map

ERA Environmental Risk Assessment

EU European Union

FF Food and Feed

FSANZ Food Standards Australia New Zealand

Glu Glutamic acid

Gly Glyphosate

GM Genetically Modified

GMC Genetically Modified Crops

GMHT Genetically Modified Herbicide Tolerance

GMO Genetically Modified Organism

GMP Genetically Modified Plant

GRACE GMO Risk Assessment and Communication of Evidence

HT Herbicide Tolerant

IgE Immunglobulin E

ILSI International Life Sciences Institute

IP Intellectual Property

IR Insect Resistant

IRM Insect Resistance Management

ISAAA International Service for the Acquisition of Agri-biotech Applications

6

IUIS International Union of Immunological Societies

L Line

MS Member States

NB Nota Bene

NTO Non-Target Organism

OECD Organisation for Economic Co-operation and Development

OSR Oil Seed Rape

PICO Population, Intervention, Comparator, Outcome

PMM Post-Market Monitoring

RA Risk Assessment

RAST Radioallergosorbent test

RD Relevance Dimension

RQ Review Question

SD Standard Deviation

SR Systematic Review

TUM Technical University of Munich

US United States

USDA United States Department of Agriculture

WHC Water Holding Capacity

WHO/IUIS Allergen Nomenclature Sub-committee

WTO World Trade Organization

7

This Report This report provides the comments received from stakeholders to the Draft Review Protocols circulated in

the context of the FP7 project GRACE (GMO Risk Assessment and Communication of Evidence,

http://www.grace-fp7.eu/) along with the responses from GRACE team members.

The available evidence of health, environmental and socio-economic impacts of genetically modified

(GM) crops – concerning both risks and benefits – is being reviewed in the course of the European

Commission-funded research project “GMO Risk Assessment and Communication of Evidence”

(GRACE). GRACE reviews exceed previous reviews of GMO evidence and are conducted in a highly

systematic, transparent and inclusive way, based on the concepts of systematic reviews (SRs) and

evidence maps (EMs). Both types of review procedures have been proposed or have been used as

valuable tools to support evidence-based policy-making in many areas including medicine, environmental

studies, social studies, and more recently, food safety. In contrast to narrative reviews, SRs and EMs

require the development of well framed review questions (RQs) and detailed review protocols describing

all steps and methods of the review process before the review itself. The literature searches are typically

extensive and the conduct of the review is documented in detail to allow for later verification as well as for

updating. For SRs, two sets of criteria need to be developed – one for study inclusion and another for

assessing the quality of the evidence. Should sufficient quantitative data be collated, a meta-analysis

could be conducted.

Based on these concepts, GRACE developed a general framework for Good Review Practice in

assessing health, environmental and socio-economic impacts of GM crops. The general framework is

being applied in the specific reviews outlined below. All key steps of these reviews were or will be

subjected to stakeholder scrutiny and feedback. The results of the reviews will be made available via the

open access database CADIMA (www.cadima.info) which will be designed to serve professional users of,

and interested parties in, GMO risk assessment. GRACE reviews will yield two deliverables: the results

and conclusions of the reviews and a proposal for a framework of Good Review Practice. Both

deliverables are expected to support evidence-based policy-making concerning GMO risk assessment.

Stakeholder involvement in key steps of the research process is of particular importance in GRACE. The

overall approach to stakeholder involvement is described at http://www.grace-fp7.eu/content/stakeholder.

Planning stage consultations on the GRACE evidence synthesis tasks were already held on:

overall framework for evidence synthesis used in GRACE

review questions

prioritization of the preview questions

Up to Sept 2014 one report on the overall framework for evidence synthesis and the review topics is

already available on the GRACE website. This report in hand covers the consultation on the draft review

protocols, which represents the final step in the planning stage of the GRACE reviews. For more

information on the consultation reports which are still in the publication process see http://www.grace-

fp7.eu/content/stakeholder-consultation-evidence-synthesis-gmo-impact-research.

This consultation was conducted by inviting 56 stakeholders involved in previous consultation steps via e-

mail and providing access information to pdf versions of the draft review protocols. Overall 14 draft review

protocols were subjected to this consultation step with a commenting period of two weeks (extended on

request of stakeholders to 3 or 4 weeks resp.) between November 2013 and January 2014. Overall 27

sets of comments were received from five organisations /individual experts. The majority of the comments

were received from governmental authorities and industry. No comments were received from CSOs. Only

one set of comments was received on one of the draft protocol on socioeconomic impacts of GMOs (see

Table 1).

http://www.grace-fp7.eu/

http://www.cadima.info/

http://www.grace-fp7.eu/content/stakeholder

8

Table 1: Feed-back received to draft review protocols

Draft protocols Protocols circulated

Protocols receiving

comments

Stakeholder organisations/

individuals commenting

Health impacts of GMOs 4 4 5

Environmental impacts of GMOs 6 6 3

Socio-economic impacts of GMOs 4 1 1

The revised protocols were (will be) submitted for publication to a scientific journal1 and thereby the drafts

revised according to stakeholder comments were subjected to a peer-review procedure. Following

another revision of the protocols based on reviewer’s comments the final versions of the protocols were

then (will be) re-submitted to the journal. By the time of publication of this report (October 2014) 6 review

protocols were published and 5 are in different stages of the publication process.

The overall structure of the report mirrors the thematic areas in GRACE in which systematic reviews and

evidence maps are being conducted: health, environmental and socio-economic impacts of GMOs. The

substructure in each chapter corresponds to the titles of the review protocols. Stakeholder comments

were re-organised and - in general - provided in full-length. Editorial comments were not included in this

report. Stakeholder comments were numbered to facilitate orientation and cross-references between

responses. References to pages and paragraphs provided in the comments in brackets are referring to

the draft review protocols circulated for consultation.

Responses to comments from the GRACE project team are provided. In some cases responses are

referring to several related comments. References to pages or paragraphs are referring to the original

draft review protocols. In the chapter on health impacts a summary response is provided instead of

responses to each or related comments.

The last chapter provides the references to the final review protocols already published or in the process

of publication.

1 Two protocols on socioeconomic impacts were not submitted to the journal for budgetary reasons as the

journal asked for a publication fee.

9

Chapter 1: Environmental Impacts of GMOs

General Comments

This section refers to comments that are not linked to a specific protocol but that are of relevance for most

or all of them.

1) A clear explanation why systematic reviewing vs evidence mapping was chosen as methodology

(would be helpful).

In this respect, I do not understand the rationale for choosing different methodologies for the 2

protocols related to “Insect Resistance Development of Bt-crops”. The protocol related to question 2

refers to a “Systematic Review”, whereas the one related to question 4 refers to an “Evidence Map”. I

would expect these two questions to be addressed according to the same methodology. (CA)

- Response: Systematic reviews are being conducted in order to give a defensible answer to a

question under debate by minimizing the introduction of bias whilst ensuring transparency of the

methods used. They follow a standardized methodology in order to identify, critically appraise,

analyse and if possible synthesis (meta-analysis) data from relevant primary research and are

thus especially valuable for impartially synthesizing evidence relating to contentious topics for

which stakeholders may hold differing views.

Evidence maps aim to characterize the evidence base in order to 1) provide an overview of the

extent, range and nature of research activities in a particular field and 2) decide whether a

specific synthesis of outcomes would be valuable, in one or more SRs on a specific section of

the map. When compared to a systematic review, they employ the same methodological

stringency for study identification and selection but typically provides a less detailed synthesis of

the evidence.

-

2) A better harmonization of some content between the different documents (would be appreciated). I

understand that different authors are responsible for drafting different documents but a certain level of

harmonization would improve consistency between the different SR protocols. This applies for

instance to:

- The background sections, in particular the description of GRACE which is done in different ways or

is sometimes even missing.

- Information related to the input of the stakeholders. This appears in some protocols (e.g. Bt maize

on non-target animals), not in others. I think this is valuable information and that, considering the

importance of stakeholder involvement in GRACE, it is important to show how the opinions of the

stakeholders has been considered.

- The Databases used for the searches. It is unclear why some general Databases (e.g. BCH, CERA,

SCIRUS) are mentioned in some protocols and not in others. In my opinion, the same basic set of

Databases should be used for all protocols with additional specific DBases when needed.

- Specialized sources of information: The sources from which additional data might be available differ

between documents (people having conducted field trials, experts from the consultation network...).

- Non-english searches. (CA)

- Response: some harmonisation of the drafts has taken place but it must be considered that

each group have approached their reviews independently, each review is considering a different

10

scientific question and have constructed their reviews in ways that they feel will obtain

meaningful outputs. In relation to languages: this is for each group to consider whether valuable

information is likely to arise from publications in different languages. In the case of GMHT crops

it was considered that several S American agronomy journals were likely to have published

papers on the effects of GMHT crops on weed control that might be relevant to the study.

3) There are no explicit criteria listed on how quality of each reviewed paper was assessed. EFSA

published a methodology on how to conduct literature review in a transparent, although burdensome,

way (http://www.efsa.europa.eu/de/efsajournal/doc/2092.pdf). The basis of the quality assessment

was published by Klimisch et al. (1997) in their publication "A Systematic Approach for Evaluating the

Quality of Experimental Toxicological and Ecotoxicological Data"2. The matter is that most scientific

literature in the academic world is biased towards finding effects, since absence of effects usually

leads to rejection of the paper when submitted to a journal. Therefore the quality of the work

published needs to be assessed very thoroughly, in order to counter-balance the bias. Industry work

can be considered as biased by some stakeholders, but study protocols are pre-defined, validated,

documented and reported in a transparent way, although confidential. Academic work very often is

exploratory, which can lead to identification of effects being the priority compared to thorough

hypothesis testing...We would strongly suggest a revision of the literature review and to use a

thorough, reproducible, transparent and robust reviewing methodology, in line with what is listed

above. (industry)

- Response: Study quality will be assessed according to the criteria listed at the end of paragraph

3.3. Each study will be assigned low, medium, high, or unclear risk to bias. This procedure is

conforming to the CEE guidelines that are the current state of the art regarding systematic

reviews in the field of environmental sciences. The systematic review guidelines developed

within the GRACE project is more detailed than the CEE guideline and include aspects of CEE,

Campbell Collaboration, Cochrane Collaboration, and the EFSA methodology, of which we are

aware. Therefore, we are confident that a thorough, reproducible, transparent and robust

reviewing methodology as demanded by industry is provided.

Related to bias: We are aware that there might be bias away from studies showing no effects

and there may be selection of studies for positive or negative effects. As discussed in each

protocol – these issues will be taken into account and we will indicate whether there is a risk of

bias in individual papers or an imbalance of studies reporting particular effects which is not fully

supported by the weight of evidence. We will place emphasis on the quality of the studies not

their outcomes.

2 Klimisch et al., A Systematic Approach for Evaluating the Quality of Experimental Toxicological and Ecotoxicological Data Regul

Toxicol Pharmacol. 1997 Feb;25(1):1-5.

http://www.ncbi.nlm.nih.gov/pubmed/9056496

http://www.ncbi.nlm.nih.gov/pubmed/9056496

11

1) Systematic Review “Does the growing of Bt maize change populations or

ecological functions of non-target animals compared to the growing of

conventional non-GM maize?

General or Overarching Comments

1) The title is too broad to describe the objective of this review. (industry)

- Response: The circulated title “Effects of Bt maize on non-target animals” was a short version of

the review question, which is “Does the growing of Bt maize change populations or ecological

functions of non-target animals compared to the growing of conventional non-GM maize?”. The

long version is the title for publication.

2) It is important to acknowledge that only one kind of science exists regardless of the funding of the

scientific studies. (industry)

- Response. We agree that science should be universal regardless of funding in an ideal world.

However, our analysis whether the outcome of studies is correlated with funding will show if a

potential bias exists.

Comments on Background

3) (Ref: p 4, last paragraph): The authors identified only in field (or does in field include field margins?)

direct and indirect effects of the Bt toxin and unintended changes in plant composition of nutrients or

toxicants as pathways to harm. Off field or field margins seem not to be included. Impacts of the

specific cultivation and/or management were not discussed.

Bt maize supports monocropping of maize, which might have a negative effect on biodiversity.

Furthermore, in the background text and question 1, it was not clearly stated that only in field

biodiversity will be investigated. Maize pollen might affect the biodiversity of habitats adjacent to the

field. Nevertheless, the next chapter describes the selection of studies only that focus on in field

biodiversity and only on effects of Bt toxin.

The review question should be amended to direct and indirect effects of Bt maize or their toxins on

biodiversity in field and off field –field margins and to include indirect effects of cultivation and/or

management changes on biodiversity in field as well as adjacent habitats. The restriction to one

season will exclude long-term and cumulative effects. It would be preferable to include such studies

(if they exist), otherwise this gap should be clearly stated in the review question and discussion

chapter of the report (as a knowledge and data gap). (CA)

- Response: Field margins: studies on field margins will be included in the review if they fulfil the

inclusion criteria (comparative studies on arthropods in field margins of Bt maize fields compared

to non-Bt maize fields).

Off-field: not considered as it is impossible to link potential changes in off-field habitats to one

particular maize field. It is also impossible to compare off-field effects of Bt maize to conventional

maize and we are not aware that such studies exist. The focus of the systematic review will be

on field level. We added a sentence explaining this.

Cultivation/management effects: The most likely changes in management will be that the

insecticide regime changes with the cultivation of Bt maize. Therefore, we include comparisons

of Bt maize to untreated as well as treated conventional maize. Apart from that, the review

focuses on the interaction of the Bt plant with non-target animals and we will not cover changes

in crop rotation, changes in the frequency of maize cultivation, etc. We added a sentence stating

those limitations. The amount of available data on management practices other than pesticides

12

would be very limited anyway.

Long-term effects: The nature of the meta-analyses statistics as we will apply them requires that

datasets are on a yearly basis. Studies covering several years will be included in the review, but

each year will be analyzed individually. We added a field (years_transgenic) for data extraction

stating the number of years a plot/field is known to be under Bt maize cultivation. This allows

conducting meta-analyses for first year Bt-maize fields in comparison to long-term Bt maize field

(if the available datasets allow such analyses).

4) (Ref: p 4, last paragraph): Harm can also occur if populations of NTOs increase. This can also result

in disruption of agro-ecological functions. (CA)

- Response: We agree, we now use “change”.

Comments on Objective of the review

5) (Ref: p 5, “Population”): In the background text of review question 1, it was not clearly defined that

only in field/in crop biodiversity will be investigated. It is strongly recommended to include off field

effects. Accordingly the study selection should be adjusted to include adjacent habitats. (CA)

- Response: We added information that populations in field margins will be included in the review.

6) (Ref: p 5, “Intervention”): Another intervention are the cultivation and management practices used

with Bt maize. (CA)

- Response: Insecticides are included as one comparator. Other management practices are not

focus of the review. We make this explicit in the revised protocol.

7) (Ref: p 6, “Outcome”): The here utilized definition of biodiversity should be explained. What is

included and what excluded? (CA)

- Response: Biodiversity was replaced by species richness to be more precise. (Remark: The

term biodiversity was also used in your remarks without having a definition for it.)

8) (Ref: p 6, bullet points): Add the bullet point “What is the concentration of the expressed Bt toxin in

different plant parts?”

Different traits can produce wildly varying amounts of Bt toxin and there can be big differences in

different plant parts being very relevant regarding impacts. Pooling studies with different Bt toxin

concentration in the plant material has to be avoided in the meta analysis. (CA)

- Response: Bt protein concentrations in different plant parts were not measured in most of the

published field studies. Therefore, previous meta-analyses used the values published in the EPA

documents. In addition, ELISA values “wildly” vary with the applied method (extraction method,

batches of kits, supplier of kits, source of used standards, etc.) and are difficult to compare

between studies or labs. Therefore, to adequately capture the range of concentrations in

different plant parts assessed with different methods would require many extra fields with the

result that no information is available for the majority of the studies. We therefore suggest to

address the issue of uncertainties in concentrations in the bias-assessment (performance bias).

Differences in effects of different transformation events can be analyzed in sensitivity analyses.

Those analyses can be discussed with respect to concentrations. Respective information was

added to the bias assessment and sensitivity analyses sections.

9) (Ref: p 6, last paragraph): Effects on landscape level through changes in cultivation and management

practice will not be analysed in this review. This knowledge gap und limitation of the results must be

discussed in the review. (CA)

13

- Response: See reply for comment 3.

Comments on Methods

10) Review team: The review team should be supplemented by an external expert/review team, which

reflects the discussion on GMO effects on non-target organism out of different perspectives. Experts

could be selected from the CAs commenting applications for cultivation in the EU. (CA)

- Response: External expertise is well covered within the review process in the GRACE project:

a) all review protocols and reviews are commented by the consortium members, which have

different expertise. b) Stakeholders are consulted in each critical step of the process and

comments will be addressed. c) Review protocols as well as draft manuscripts of the

environment work package will be peer reviewed by the Collaboration on Environmental

Evidence (CEE).

Comments on Search Strategy

11) (Ref: p 8, “Databases to be searched”): (There is the) need to harmonize the list of databases

between all SR protocols. (CA)

- Response: We tried to harmonize the different protocols as possible. However, the review

teams working on those protocols are composed of people from different institutions with

different resources. Access to literature databases is often restricted and depending on

subscription contracts. While all the reviews follow the same methodological guidelines that were

agreed between all partners, a certain individuality of each review team should be allowed.

12) (Ref: p 10, last paragraph): With the current wording of this section, it might be questioned why a SR

is needed, since with the scoping exercise you do not retrieve additional papers as compared to

classical reviews. I would therefore add one or two sentences explaining the added-value of a SR.

(CA)

- Response: We reworded the section to make its purpose clearer. The aim of the scoping was to

determine if all relevant papers that were already known will be found with the keywords and the

databases we plan to use. The purpose of this exercise was not to find additional papers (which

will clearly be the purpose of the actual review).

Comments on Study Inclusion Criteria

13) (Ref: p 11, first line): Why "evidence map"? It is a SR protocol. (CA) (Note: this is likely to be an

editorial comment and will not be included in the final report)

- Response: We changed to systematic review throughout the manuscript.

14) (Ref: p 11, general): Does that mean that only papers containing the original data are considered

instead of the reviews? The second part of the first sentence is somehow confusing. Does that mean

only peer reviewed publications and each data set is included only once? Or what is meant with not

been published elsewhere? (CA)

- Response. We agree that this statement was confusing and changed the text accordingly. Peer

reviewed papers are preferred, but not the exclusive source of information. Each dataset will be

included only once.

15) (Ref: p 11, “Population”): It is correct to exclude studies that deal only with the target organism. But

information on the effectiveness of the Bt toxin on the target organism is also relevant for the selected

14

non-target studies. If there are no or positive effects on non-target organisms, it should be assured

that the target organism is affected by the GMO, as it is possible that the trait has failed to express

enough toxin (positive control). (CA)

- Response. This issue is part of the concentration discussion (see reply for comment 8, Ref: p 6,

Bullet points). Information on targets will be considered for this discussion if available.

16) (Ref: p 11, “Outcome”): The here utilized definition of biodiversity should be explained. What is

included and what excluded? (CA)

- Response: Replaced by species richness.

Comments on Study Quality Assessment

17) (Ref: p 12, first paragraph): It is not clear how the authors will deal with the bias and quality of studies.

Is a low quality or uncertainty an excluding criteria or only criteria of the sensitivity analysis?

The text seems to be drafted in a biased way itself. Many criteria to recognize bias of the studies are

only useful, if the aim of the review is to show effects of the Bt toxin. Studies which deal with

cultivation und management effects and unintended effects were excluded.

The criteria are not well defined. (CA)

- Response: Studies with a high risk to bias will not be excluded, but their influence will be

analyzed in sensitivity analyses. The review focuses on the effects of the Bt maize plant on a

field-scale in comparison with non-Bt maize plants. This includes unintended effects, if they

influence non-target animal populations. Cultivation and management effects other than

insecticide treatments are not in the scope of the review. It is difficult to define criteria for bias

assessment if the data are not known beforehand. The assessment as suggested is fully in line

with the guidance provided by CEE. Therefore, we reject the accusation that the text seems to

be drafted in a biased way itself.

18) (Ref: p 12, “Plot Size”): How will it be determined if the plot size is smaller than the common

movement range of a non-target organism? (CA)

- Response: The text was reworded to acknowledge that a direct relationship between plot size

and movement of animals is very difficult to assess.

19) (Ref: p 13, “Application of Pesticides”): The pesticide use before sampling is in some studies a

potential source of bias. If the pesticide use is very high and the biodiversity in the field is low,

differences due to Bt toxin might be covered by effects due to pesticide use and not be measurable.

(CA)

- Response: This point was added to selection bias.

20) (Ref: p 13, “Type of Methods”): The authors address the type of method as a source of bias. But not

the type of method is the source of bias but rather the question whether the method is used in an

adequate way. The visual control is in ecosystem field research an accepted method, if the method is

used with standardized and proofed criteria. Traps are often not applicable to show effects on

immobile stages of non-target organisms. (CA)

- Response: sentence on the need for adequate use of methods was added

21) (Ref: p 13, “Reporting Bias”): Study funding is not the only way to influence the results of the study. If

not provided, scientists should disclose the source of the used GM material and any related contracts,

agreements or memorandums of understanding. Big research projects are often connected to

steering committees. The composition of these committees should be provided. (CA)

15

- Response: Information on study funding can usually be obtained from the acknowledgement

section of a study. The requested information, however, is not available for the majority of the

studies and can thus not be used for analyses. Furthermore, the aim of the review is to evaluate

the existing (published) evidence, not to analyze the reliability, motivation and contractual

relationships of the study authors with third parties.

22) (Ref: p 13, after paragraph “Reporting Bias”): Please add the bullet point “general plausibility check”.

It is difficult to assess studies only by the above mentioned bias criteria. The experimental design,

statistic and method are assessed with regard to the hypothesis at the start of the experiment.

Studies, which have the aim to show differences, could have a complete different design from studies

with the aim of showing equivalence and different alpha and beta errors. In the risk assessment the

evidence of risk are very important, so a minority of reports could have more gravity than the majority

of reports. Pooling of studies with different aim is not always appropriate. (CA)

- Response: The suggested meta-analyses is based on the characteristics (e.g. abundance) of

populations or ecological functions on a field level. Original data will be statistically meta-

analysed, which is independent from the statistics applied by the study authors. Therefore, the

hypothesis of the study is not important as long as the measurements are done and presented in

a way that they can be used in the meta-analyses. Adding an extra point for general plausibility

check is thus not necessary. Data suitability for meta-analyses will be checked with the inclusion

criteria.

23) (Ref: p 13, “Selection of field sites/plots”): Please add “baseline of biodiversity and base abundance

of the non-target organism”. (CA)

- Response: Baseline data are not provided in published studies and generally not available.

However, Bt maize is compared to control treatments (without insecticides or with insecticides),

which provides two baselines in the studies themselves.

24) (Ref: p 13, “Plot Size”): Please add “duration of study”, “type of hypothesis”, “statistics”. (CA)

- Response: “duration of study”: The years under Bt maize cultivation will be used as a parameter

to analyse the data (comparison of short-term versus long-term data), but not as a source of

bias. “type of hypothesis” and “statistics” are not a factor in our assessment of bias because the

statistics of the original studies are irrelevant for the conducted meta-analysis.

25) (Ref: p 14, first paragraph): The screening process should be done by an external expert team, which

reflects the discussion on GMO effects on non-target organism. Experts could be selected from the

CAs commenting different applications for cultivation in the EU. (CA)

- Response: See reply for comment 10).

Comments on Data Extraction Strategy

26) (Ref: p 14, third paragraph “Observations”): To focus on only one season will exclude all long-term

and cumulative effects which should be mentioned in the review question and the discussion chapter

of the report. (CA)

- Response: Long term cultivation will be compared to short term cultivation in our analysis.

27) (Ref: Table 2, p 15): Please add “pip-conc: concentration of the Bt toxin” and “pip-measure: Are the

concentrations measured or estimated?” (CA)


16

28) (Ref: Table 2, p 16): (Parameter) “cultivation” is too broad. This parameter should be split into

different parameters. (CA)

- Response: Cultivation practices are likely to be very variable among studies. Therefore, we

provide only one text-field and no predefined terms for this aspect. Depending on the available

information, the field might be split at a later stage.

29) (Ref: Table 2, p 16): (Parameter “control_mean”) Please split into the parameters “Mean for the

control treatment with pesticide application” and “Mean for the control treatment with non-pesticide

application”. (CA)

- Response: The specification of the comparison is included in the field “comparison_type”. The

meta-analyses we plan to perform require this format.

30) (Ref: Table 2, p 16): (Parameter “warning 1”) Please add “full site description: including description of

the soil, weed infestation, disease infestation, target infestation (treatment and without treatment),

cropping history, pesticide application history and abiotic conditions”. (CA)

- Response: Text field for site characteristics other than cultivation practices is added.

Comments on Data Analysis

31) (Ref: p 18, “Meta-analysis”): The amount of acceptable missing data should be fixed before the meta

analyses. (CA)

- Response: Meta-analyses might not make sense if less than 5 observations are available for a

certain parameter and taxonomic group. We added this information to the protocol. Studies with

many blank fields will result in a high risk to bias. However, they should not be excluded as long

as the information specified in the inclusion criteria is available.

32) (Ref: p 18, “Meta-analysis/ taxonomic group”): If a taxonomic group or ecological function is identified,

it has to be checked whether these groups are really measured by the methods in the study and

whether differences in important subgroups can be determined with this parameter and method. (CA)

- Response: Meta-analysis can be performed on different taxonomic levels (e.g. species, genus,

family, order). The appropriateness of the applied method to measure the respective taxon or

function is assessed in the risk to bias assessment.

33) (Ref: p 18, “Meta-analysis/ means over plots”): Before building means over plots and fields, the

variance within plots and fields has to be analyzed. It could be an evidence of genotype x

environment interactions. (CA)

- Response: The unit of replication for most field studies is the plot within one field or the field in

multi-field studies. Therefore, means of those plots or fields reflect the appropriate unit of

replication. Potential genotype x environment interactions might be reflected in the variance of

the treatments. We do not see the necessity of evaluating within-plot variance.

Comments on Measure of treatments effect

34) (Ref: p 18, “Different Bt Proteins”): Please add “Bt toxin concentrations”. (CA)


17

Comments on Dealing with missing data

35) (Ref: p 19): The amount of acceptable missing data should be fixed before the meta analyses. The

definition of “relevant data” should be done by an external expert/review team. (CA)

- Response: Data need to fulfil the inclusion criteria (see 3.2. and 3.5.) to be included in the study.

The text was amended accordingly.

Comments on Quantitaive Synthesis

36) (Ref: p 19, second paragraph): Please add: Different results between taxa species level should be

mentioned und discussed in the report. The results of special taxa or functional groups should be

carefully interpreted. (CA)

- Response: Different functional groups and main taxonomic groups will be compared and

therefore discussed and interpreted.

37) (Ref: p 20, “Assessment of publication bias”): The statement is not correct. Results with no effect or

positive effect on biodiversity will be published by the industry, because the industry has to show in

the risk assessment that the GMO has no negative effect in comparison to conventional plants. (CA)

- Response: The sentence this comment refers to generally describes the interpretation of Begg’s

funnel plots and does speculate on the outcome of our particular review. For Bt maize, there

might indeed be an interest to publish studies no matter what the outcome is. Thus the funnel

plot would not indicate bias.

18

2) Systematic Review “Environmental effects of the cultivation of GM

herbicide tolerant crops”


1) It is not clear how the outcome of this document can help risk assessors/managers in Europe as the

cropping characteristics of Europe are quite different from other parts where GMHT crop are

cultivated (e.g. small sized fields in Europe compared to big size in the US). A link about what the

outcome of this document really means for European agriculture is missing. (industry)

- Response: the conceptual model clearly shows the potential for GMHT cropping systems to

influence botanical diversity and that studies in different countries (including Europe) have shown

different impacts on botanical diversity of GMHT systems. A compilation of the data in a

systematic review and analyses of this data will show impacts of GMHT systems and also what

factors influence changes in botanical diversity. This will inform risk assessors trying to form

judgements on the likelihood of environmental harm and inform risk managers on how GMHT

crops can be managed in order to minimise or avoid adverse impacts and in some cases to

improve on impacts of current cropping systems.

2) The title is too broad to describe the objective of this review. The general language in the document is

biased. Examples are provided in the specific comments below. (industry)

- Response: The title has been changed to “what are the effects of the cultivation of GM herbicide

tolerant crops on botanical diversity? “ This was chosen since plants form the first tier of food

chains and therefore influences other components of biodiversity in agro-ecosystems. The

protocol describes the primary review question on the influence of GMHT cropping on plant

populations and a secondary one the development of resistant weeds. The introduction clearly

indicates the conflicting results obtained from studies of GMHT crops which underlie the need for

this study. The protocol also recognises the tensions between managing weed competition with

crops and the desires to maintain biodiversity in agro-ecosystems. The specific comments are

addressed below.


scientific studies. In the document the text suggests that results depend more on the funding rather

than on the protocol used and the quality assurance methods. This is certainly not the case and is a

biased approach. (industry)

- Response: Unfortunately scientific studies and reports come in numerous forms of very variable

quality and therefore the data abstraction and systematic review will carefully examine the quality

of all studies and publication. Meta-analysis of the data will hopefully show trends and patterns

that allow shortcomings in the quality of the data to be overcome and more solid conclusions to

be made. A range of factors which influence study design, data collection, analyses and

outcomes will be examined, including funding, political positioning, interest in outcomes etc. This

is described in the protocol.

Comments on the Abstract

4) (Ref: p 1, “Background”): Shifts in faunistic diversity (effects along the food chain) are equally

important, but no adequately considered here, which they should (cf. Table 1). (CA)

19

- Response: Addresses this and impacts on other biota were not included because of limited time

and resources for this SR.

5) (Ref: p 1, “Background”): In the sentence ‘There are concerns that GMHT crops treated with broad

spectrum herbicides will cause declines in biodiversity.’ replace ‘will’ by ‘might’. (industry)

- Response: The wording is in line with independent opinions on GMHT crops published by

ACRE in UK, EFSA and others, that GMHT crops, if not managed appropriately, WILL cause

declines in biodiversity within crops.

6) (Ref: p 1, “Background”): ‘Research results show a range of effects and the priority is to determine

whether research studies show shifts in botanical diversity and/or declines in plant populations in

GMHT compared with conventionally managed crops.’

This does not include e.g. earth worms’ populations and soil erosion, which are both major issues for

the EU. As indicated in the General Comments, the title of this document is too broad. (industry)

- Response: see responses 1),2) & 4). The SR will also consider factors such as minimal tillage

which will influence soil moisture retention, soil erosion, and also impact botanical diversity.

7) (Ref: p 1, “Methods”): ‘We will perform a rigorous review of studies of plant populations in fields of

GMHT crops by complying with CEE (www.environmentalevidence.org) requirements for Systematic

Reviews (SR).’

Why not with the EFSA guideline for systematic reviews? (industry)

- Response: The EFSA guidelines are also now mentioned in the protocol but it should be noted

that they are largely based on the CEE and other related SR strategies.

8) (Ref: p 1, “Methods”): ‘A RP is presented for the SR of data from field studies of GMHT crops,

comparing the effects of GM crop, herbicide regimes and associated management applied to HT

crops with conventional crops and their weed management for impacts on plant populations in

fields as assessment end points or indicators of impacts on botanical diversity and

associated food chain and ecosystem services effects.’

The endpoint selection is not justified; in fact the literature review should try to establish first the right

endpoints.

This term needs clarification. What is understood as food chain? (industry)

- Response: details of the conceptual model have been added to clarify why this endpoint was

selected for the SR. See also responses 1) & 2).

9) (Ref: p 1, Methods): ‘They include appropriate study designs and comparators. The RP describes the

collation and integration of the studies and outlines the type of analyses that will be performed to

assess bias of the selected studies and if covariables describing the heterogeneity of the studies

introduces biases.’

Will studies repeated by several scientific groups get preference treatment over single group studies?

(industry)

- Response: No preferential treatments will be given. However if similar studies have been

independently conducted then the data and outcomes can more readily be subjected to meta-

analysis and overall outcomes considered.

10) (Ref: p 2, first line): Not only plant species, but also animals should be analysed separately. (CA)

- Response: See comments above.

11) (Ref: p 2, second line): Please consider the following endpoints as well:

20

- i.total number of weed species

- ii.individual number of weed species

- iii. seed bank populations

- iv. time of emergence

- v. time of flowering

- vi. abundance of invertebrates and vertebrates (CA)

- Response: The i-iii characters are listed in the table of endpoints/data collected. iv. Will be

derived from early plant count data. v. will be concluded from late plant count data since high

late counts will assume flowering, seed set and seed return. Soil inverts are being considered in

the SR by P H Krogh: for vertebrates and other fauna: see comments in 1)-3).

12) (Ref: p 2, “Background”): ‘HT crops are widely grown in N and S America (1,2) and are being

developed for many other regions, including Europe. Since use of herbicides has been associated

with declines in farmland biodiversity in some regions, there are concerns and some evidence that

GMHT crops treated with broad spectrum herbicides will also cause declines in biodiversity’.

This was also addressed in our last round of comments. Although HT crops have been submitted in

Europe, no companies have developed GMHT crops specifically for Europe. (industry)

- Response: Applications to cultivate GMHT crops have been submitted to EFSA and GMHT soya

was extensively grown in Romania. European plant breeders (eg KWS) are developing GMHT

crops.

13) (Ref: p 2, “Background”, first paragraph): How does the text define biodiversity compared to farmland

biodiversity? What is included and what excluded? (CA)

- Response: No definitions of biodiversity are supplied. Farmland biodiversity refers to land under

particular management and the biodiversity found there. This follows the thread described in the

conceptual model and the literature particularly in relation to FSE studies.

14) (Ref: p 2, “Background”): ‘Therefore the range of impacts of HT systems requires study in order to

determine GM HT cropping effects on farmland biodiversity. There are also concerns that repeated

use of the herbicides used on HT crops will promote herbicide resistance development (8) and further

inappropriate use of herbicides leading to reductions in biodiversity’.

This is not HT crop specific. Will the study take into account cropping history from several years

before each study, as that will have an effect? (industry)

- Response: All farmland management activities are likely to influence biodiversity. That is why

there is a comparator. Previous cultivations will be considered where there is data on weed

populations at different stages of rotations. Continuous or repeated cropping of GMHT crops and

seed banks in subsequent years will be considered.

15) (Ref: p 2, “Background”, second paragraph): Soil related aspects are hardly covered in this document

(cf. comment (Ref: p 3, Primary Review Question).

GMHT crop cultivation might affect the quality and composition of the soil, its microflora and function

as well as crop health. When assessing the latter for GMHT and conventional crops, differences in

agronomic practices (e.g. fertilization) should be considered. (CA)

- Response: Comparative studies where a range of other factors are changed in addition to the

GMHT specific herbicide management will not be considered unless the compounding influences

of these other factors can be separated from the herbicide management effects. Minimal tillage

studies will be included if good comparisons can be made.

21

16) (Ref: p 2, “Concept”): In the sentence ‘The main concern is that the cultivation of GMHT crops will

reduce biodiversity and ecosystem services in farmland regions’, replace ‘will’ by ‘might’.

This sentence looks like a hypothesis but lacks comparison to something. (industry)

- Response: see response to 5) above. The comparator and the hypothesis are clearly described

in the conceptual model and the review question.

17) (Ref: p 3, “Stakeholder Concern”): How many stakeholders have defined this as a high priority? With

regards to raised public concern, which is the difference to other cultivated HT plants? In which region

is this topic considered scientifically controversial, in Europe, in the US? Farmers and their concerns

should also be consulted. (industry)

- Response: Stakeholders were consulted and gave this review question a high priority and high

marks for concern and controversy. Farmers have considerable interest in this topic because of

the potential and real advantages and pitfalls.

18) (Ref: p 3, “Stakeholder Concern”): ‘The EC is currently considering applications to commercialize

GMHT crops in Europe and this study will inform them of the current state of knowledge on this topic’.

Does this sentence imply that the risk assessment done up to now by EFSA did not take into account

current state of knowledge? (industry)

- Response: EFSA is not part of the European Commission or the decision making process of

GMOs and so No this is not the implication. EFSA is aware of current state of knowledge on

GMOs. Unfortunately this cannot be said for many EU, MS and NGO institutions or

organisations.


19) (Ref: p 3, first paragraph): Soil organisms might be directly affected as well (cf. comment on primary

questions). (CA)

- Response: See comments above on the scope of the SR.

20) (Ref: p 3, first paragraph): Please add references to examples of conventional cropping systems that

take this into account when applying herbicides.

Decline = change, this suggest the study is biased in the direction of decline. Will the study focus on

In field, where the herbicides are used or off field? (industry)

- Response: Reference added. The concern is about decline in biodiversity. The review will

assess changes in weed/plant populations as indicators of shifts in biodiversity. The later

methodology makes clear that the data will be assessed for changes in any direction.

21) (Ref: p 3, second paragraph): At this stage, we cannot conclude yet that a SR is required. One should

rather say that a SR can bring some added-value in comparison with a narrative review. (CA)

- Response: Text changed accordingly to indicate desirability for an SR and not need.

22) (Ref: p 3, second paragraph): Contextualization is missing: a shift must not be bad "per se". It is

important to explain this.

This also suggests a bias towards decline. Will it be focusing on numbers of different plants or total

number of plants? (industry)

Response: See response to comment 20) above. Species, family and total population data will be

examined where it is available. This is described later in the methods.

22

23) (Ref: p 3, “Primary Review Question”): Another primary review question should relate to soil

organisms:

Is the soil microflora (P) changed (O) by management regimes applied to GMHT crops or associated

agronomic practices (I) compared with conventional crop management (C)?

Studies investigating this question should also include experimental studies performed in the

laboratory to assess the repeated application of glyphosate or glufosinate on microbial soil

communities. (CA)

- Response: See comments 1) & 2) about the scope of this SR.

24) (Ref: p 3, “Primary Review Question”): Changes and Shifts in faunistic diversity should also be

considered (cf. comment to the abstract (Ref: p 1, Background). (CA)


25) (Ref: p 3, “Primary Review Question”, first paragraph): If there is a change, it would not inherently be

bad.

How is conventional defined, does that include organic? Does that include management with the

same herbicides but just without the GMHT? (industry)

- Response: The review will assess changes in weed/plant populations as indicators of shifts in

biodiversity. The later methodology makes clear that the data will be assessed for changes in

any direction. The GMHT weed management systems will be compared with alternative systems.

These will mainly be conventional weed management systems but may include changes in

cultivation and organic systems.

26) (Ref: p 3, “Primary Review Question”, second paragraph): The stressor is the GM plant. The

herbicide applied is already assessed through the CP regulation and the associated management

applied. (industry)

- Response: The stressors are the GM plant and its management. This is clearly described in the

conceptual model, is in line with EC 2001/18 and is the reason for the extensive literature

referred to.

27) (Ref: p 4, secondary question): Please consider the following question as well:

Is biodiversity indirectly and further affected through adaption of the management regimes applied to

GM HT crops because of resistant weeds? (CA)

- Response: Data on weed population shifts will examine closely the associated managements

and will comment on changes in management systems in response to weed resistance if it can

be associated with population changes.

28) (Ref: p 4, secondary question): Weed resistance development is already happening in conventional

agriculture. Will this be compared to conventional agriculture with the same Herbicides regime just

without the GM crop grown? (industry)

- Response: Data on weed resistance to the herbicides used in GMHT systems will be compared

to any baseline information on levels before the introduction of these systems.

Comments on Methods

29) (Ref: p 4, “Search Strategy”): Because of the strong and only focus on the effects of botanical

diversity and weed populations in farmland, effects on soil functions and faunistic biodiversity are

neglected. (CA)

23


30) (Ref: p 4, “Search Strategy”): ‘The aim of the literature search is to be comprehensive in revealing all

available scientific and technical publications and reports which compare the environmental effects of

GMHT and conventionally managed crops, with a specific focus on the effects on the botanical

diversity and weed populations in farmland receiving the two comparable managements’

GM HT crops have no effect on weeds. The applied herbicides have. (industry)

- Response: The GMHT weed management systems will be compared with alternative

management systems as stated.

31) (Ref: p 4, “Search Strategy”): ‘The search will be indiscriminate and unbiased and include all the

main GMHT crops being grown and studied including maize, soya, oilseed rape, sugar beet, cotton

and rice.’

An unbiased search is only possible if the question is unbiased. Comparison of two GMHT to all kinds

of different conventional treatments does not sound unbiased.

Will the study include the field history as this will have an effect as well? (industry)

- Response: GMHT systems are being introduced to replace or complement existing systems. I

do not understand why a comparison of the newer systems with the old systems is considered

biased per se. Comparisons will take into account other stressors and factors which may also

cause differences between management effects.

32) (Ref: p 4, “Search Strategy”): ‘From existing literature reviews it can be anticipated that there is

considerable literature on this topic.’

It should be explained how the available literature applies to Europe. (industry)

- Response: Some of the data is from field studies in Europe. Other data will be examined for its

relevance to EU.

33) (Ref: p 4, “Search Strategy”): ‘Because keywords are not standardized in environmental sciences, a

high-sensitivity and low-specificity approach will be applied to capture a high proportion of existing

datasets.’

Not all data sets have equal value. This can be measured by the impact factor of the journal they are

published in. Also, if a study is proven to be not scientifically correct later on, will the data set be

removed again? If the data comes from a group with a non-scientific reputation, will the data set be

removed on this reputation? (industry)

- Response: Study selection criteria and quality evaluation are described in section 3.2 and 3.3 of

the protocol.

34) (Ref: p 4, “Search terms”): Soil organisms, functions and animals are not considered.

Please add the terms: Soil biodiversity; Soil function; Arthropod abundance; Arthropod diversity. (CA)


35) (Ref: p 5, “Others”, 29.”Field margin”): Why are field margins included? The objective is to evaluate

effects on botanical diversity and weed populations in farmland. There is no reference to margins.

(industry)

- Response: Field margins are considered part of farmland as they are managed in conjunction

with the management of the cropped land area.

36) (Ref: p 5, “Search Strings”): Search terms and strings need to be complemented to cover for our

proposed extension of the review questions. E.g. studies investigating the effects of changed weed

management on the soil (i.e. community composition and function) by repeated application of

24

glyphosate or glufosinate do not necessarily require growing the HT crop. The present draft, which

limits the query to HT crops (Part 1), needs to be adapted accordingly. (CA)


37) (Ref: p 6, first paragraph, part 1): I was wondering whether other herbicides such 2,4-D, dicamba or

sulfonylurea should not be included in the search criteria. If not, it may be useful to specify

somewhere that only glyphosate and glufosinate will be considered. (CA)

- Response: All GMHT systems will be examined but because these other systems are

comparatively recent there is likely to be little data for analysis. In addition management of

GMHT systems containing stacked HT genes will be examined but again it is anticipated that

data will be limited for analysis.

38) (Ref: p 6): ‘Additional searches in other languages (e.g. Spanish, French, German) will also be

conducted’

Which languages are selected? Will these be peer reviewed by the international scientific

community? If not, how can they be treated equal to peer reviewed data published in a highly rated

journal? (industry)

- Response: The examination of Spanish and Portuguese language papers is now better

described and will be supported by Ramon Albajes. Inclusion and quality criteria will be equally

applied.

39) (Ref: p 7, first line): There is the need to harmonize the list of databases between all SR protocols.

(CA)

- Response: this has been discussed and we may all use the CADIMA based system, but

different databases may be preferential for different types of data sets.

40) (Ref: p 7, “CAB/AGRIS”): Are these peer-reviewed by the international scientific community? Is the

data quality checked? (industry)

- Response: Inclusion and quality criteria will be equally applied to all papers examined.

41) (Ref: p 8, “Direct Contacts”): What is the selection criteria who will and will not be asked? Will this

data be peer reviewed? This will be biased data. How can this data be treated equally to data

presented in peer reviewed well regarded journals? (industry)

- Response: Direct contacts will be made for details of data already published and examined for

Inclusion and quality criteria.

42) (Ref: p 8, “Non-English searches”, second paragraph): This is biased to these parts of the world.

(industry)

- Response: Most scientific studies are limited in their geographical and environmental scope.

Comments on Study Inclusion Criteria

43) (Ref: p 9, “Outcome”): Data on plants other than weeds, soil organisms, fauna and data along the

food chain should also be collected. (CA)

- Response: Data on plants in fields and field margins will be examined; this may include crop

volunteers and ferals. See comments 1) & 2) about the scope of this SR.

25

44) (Ref: p 9, “Study Design/conventional weed management”): Similar to the terms population,

intervention, comparator and outcomes on page 9, it needs to be described which conventional

cultivation and weed management techniques are included (possibly it may be helpful to build

categories). (CA)

- Response: The details of the comparator treatments and management will be included in the

data base.

45) (Ref: p 9, “Study Design/reviewed papers”): Does that mean that the papers, containing the original

data, are considered instead of the reviews? (CA)

- Response: Yes. Where possible we will always seek and extract the original reports and data.

46) (Ref: p 9, “Study Design/reviewed papers”): This data is not reviewed, therefore not suited to be

added. (industry)

- Response: Literature and reports not peer reviewed will be identified but included in the review

if quality/inclusion criteria are met.

47) (Ref: p 9, “Study Design/data”): Time of year, time into the season, how is this defined? (industry)

- Response: Season refers to the crop development phase and cropping season.

48) (Ref: p 10, “GMHT systems”): The text is not entirely clear. Glu or gly rates used at higher rates than

recommended should be definitely included. Recommended rewording for first and second bullet

point: "Glufosinate (glu) or glyphosate (gly) as recommended by the Manufacturer or at reduced or

higher rate or at different application timings.” (CA)

- Response: this text has been altered.

49) (Ref: p 10, “GMHT systems”): The sentence ‘glu or gly used in programmes with other herbicides or

treatments (eg hoeing) at various rates and timings‘is unclear. (industry)

- Response: This has been clarified.

50) (Ref: p 10, “GMHT systems”): ‘Unusual practices or situations with extreme or unusual weed

management issues will be considered separately’.

How will this be decided, what is normal and what is unusual? (industry)

- Response: this has been deleted.

51) (Ref: p 11, “Applying inclusion criteria”): Not all science and all scientific data is equal, as can be seen

by the quality of the paper it is published in. (industry)

- Response: Inclusion and quality criteria will be equally applied to all paper/reports examined:

weaknesses in study design etc will be identified.

52) (Ref: p 11, “Quality assurance process”): Please specify the team members.

The review team should be supplemented by an external expert team, which reflects the discussion

of the environmental effects of the cultivation of GM herbicide resistant crops. Experts could be

selected from the CAs commenting applications for cultivation in the EU. (CA)

- Response: The team members will be selected from GRACE and the scientific community by

the authors. This will be fully described in the SR.

53) (Ref: p 11, “Quality assurance process”): Studies excluded by both should also be documented. (CA)

- Response: All inclusions and exclusions will be recorded.

26

Comments on Study Quality Assessments

54) (Ref: p 11): Will the impact factor of the journal the study is published in will be taken into account?

Will the reputation of the group that has published it will be taken into account? What will happen if

later studies discredit previous results and will you actively look for this? Will results that have been

verified by other groups be weight heavier than non-verified results? (industry)

- Response: Inclusion and quality criteria will be equally applied to all paper/reports examined .

The meta analysis of the data will show the distribution and frequency of results and any outliers.

55) (Ref: p 12, point 1): This paragraph seems to be drafted in a biased way itself. It assumes that only

with GMHT crops, management will be different for experimental and field studies. The same can

hold true for conventional crops.

Another difference could be that experimental studies and field studies select for different sites or

soils in terms of weed infestation, yield, climate etc. so that studies do not or not equally consider the

full range of receiving environments suited for cultivation. (CA)

- Response: It is based on the author’s experiences when studying designs and conduct of some

field studies with herbicides, and comments from some stakeholders to such studies.

56) (Ref: p 12, point 2): What about weed resistance development in conventional agriculture? (industry)

- Response: Levels of resistance to Glu and Gly in agricultural systems prior to the introduction of

GMHT systems will be a comparator.

57) (Ref: p 12, point 3): Same for non-conventional and rotational crops. How far back will be taken into

account? (industry)

- Response: See response to question 56).

58) (Ref: p 12, point 4): The circumstances described herein might not only determine the selection of

recorded parameters, but also weed management and other agronomic practices. e.g. fertilization.

The persons performing a SR should be aware of these mutual relations.

- Response: Differences between treatments, additional to herbicide and weed management

practices, will be recorded as shown in Table 1.

59) Study funding is not the only way to influence the results of the study. If not provided, scientists

should disclose the source of the used GM material and any related contracts, agreements or

memorandums of understanding. Big research projects are often connected to steering committees.

The composition of these committees should be provided as well. (CA)

- Response: Unless otherwise indicated it will be assumed that all experimental GMHT crop

material was derived from Biotech companies. Independence of research consortia will be

examined where considered relevant. Implementation of GLP and appropriate research

protocols will also be examined.

60) (Ref: p 12, point 4): ‘Experimental and Reporting bias is likely as some studies will be sponsored or

conducted by interested organizations (eg: agrochem Cos, biotech Cos) or by organizations

antagonistic to GMOs or modern agricultural practices, and therefore there may be some selection of

recorded parameters, data, results or reports to show a particular effect or show a ‘‘desired“ effect or

result.’

This is not acceptable for industry. Industry performs studies according to international protocols.

There is only one type of science independently of the sponsor. (industry)

27

- Response: See response to 59 above.

61) (Ref: p 12, point 4): ‘For example v. low or zero weeds would be considered a desirable result by an

Agchem company but an adverse effect (reduction in botanical diversity) by an environmental

organization.’

Farmers and Agchem companies want the same. That is why this is relevant. There is no knowledge

about how many weeds and which are needed to ensure botanical diversity. (industry)

- Response: there is a difference between efficacy studies and optimum requirements for

sustainable weed management according to IPM requirements.

Comments on Data extraction strategy

62) (Ref: p 13): The unique identification is linked to the publication. It is unclear to me how publications

involving several GM events will be managed in the database. It might be interesting to report

separately the data obtained for each events, which will be difficult if the event is not the "record unit".

(CA)

- Response: Subsets of data will be established relating to events, herbicides and crops.

63) (Ref: Table 1, p 13): Please add: a) "full site description" including description of the soil, weed

infestation, disease infestation, cropping history, herbicide history and abiotic conditions; b) sampling

of faunistic species. (CA)

- Response: Since the data collected will be comparative, a check will be made that the plots x

treatments are exposed to the same biophysical and biotic stresses and treatments within a site.

Differences will be noted. Bio-geographical influences and site to site variation will be examined.

Comments on Data Analysis

64) (Ref: p 16, second paragraph): Please add and clarify that endpoint measurements (cf. comment p 2,

second line) will be considered and assessed for cropped areas as well as for field margins. (CA)

- Response: Yes, this stated in the text.

65) (Ref: p 16, “Assessment of statistical power”): If it is published data I assume that by now everything

will be ex-post calculation.

Regarding the different detection levels for the impact (10-50%) is not justified. The inclusion criteria

refer to 50% level. What brings separating into 10, 20 and 50? (industry)

- Response: The design and power of studies will be examined and those with very low power

built into their design will be considered for exclusion as described. If a meta-analysis can be

performed, studies with a lower power may be included.

28

3) Systematic Review “Are population abundances of soil invertebrates

changed by Bt crops compared with conventional crops?”


1) Changes are not bad per se. The title is too broad to describe the objective of this review: On page 7,

reference is made to: "...their concomitant farming practice through the soil environment and in the

rhizosphere... "(industry)

- Response: The title is intended to be impartial by using the neutral term “changes” and it

reflects the exact objective of the review to collect data irrespective of whether a particular study

reports a change or no change. Likewise for the analysis that will provide statistics on the

presence and level of changes, without implicitly and explicitly providing a value judgement on

whether a change is good or bad. No changes made.

2) The general language in the document is biased. Examples are provided in the specific comments

below. (industry)

- Response: We have taken action according to the specific comments below.





- Response: We have excluded our indication that authors could introduce biases in the outcome

of their studies.


4) (Ref: p 5): ‘The only GM plant which is cultivated commercially at large scale in EU is maize cultivar

designed to produce Bt toxin - Cry1Ab, which provides protection against corn borers (Lepidoptera).’

There are several varieties containing the MON810 gene. Suggest changing to: .....is a maize

designed to produce a Bt toxin-... (industry)

- Response: Accepted.

5) (Ref: p 5): ‘In recent years, numbers of field and laboratory studies have been conducted to establish

the effects of the Bt crops on non-target organisms living above the ground and in the soil.’

Replace with ‘evaluate the potential adverse effects’. (industry)


6) (Ref: p 5): ‘There are two possible ways that the Bt crop can influence soil biota: directly through root

feeding or root exudates and litter that contain Cry toxin and/or other unknown compounds, as a

result of unintended effects of the genetic modification’

This is speculation. When a GM plant is approved for cultivation it has been thoroughly assessed.

What are the unknown compounds? (industry)

- Response: We consider it a well-established fact that unintended effects occur due to the

various steps in the process of modifying the plant genome. Industry may consult the scientific

literature and retrieve the evidence for unintended genomic changes. The sentences have been

amended for clarification of the statement.

29

7) (Ref: p 5): ‘The major concern is that the Bt crop cultivation may have an effect on in the population

abundance of soil invertebrates and their communities, which can cause changes in the soil

ecosystems, disrupting their functioning.’

No reference is made about if changes are always bad. It should be put into context. What happens if

the population changes but the function is maintained? (industry)

- Response: We agree that changes may not always lead to disruption of ecosystem functioning,

but it is implicitly given that a certain change has surpassed a threshold beyond which damage is

done to the ecosystem functioning. We have further specified “…effect on the population

abundance of soil invertebrates and their communities, which could attain a magnitude that

would cause undesirable changes in the soil ecosystem functioning according to thresholds set

by legislation or environmental authorities.” Present risk assessment would consider changes in

biodiversity and changes in ecosystem functioning in their own right, hence a change only in

biodiversity could trigger a concern.


8) (Ref: p 6, first sentence): Suggest changing to ‘potential adverse effects of’. (industry)

- Response: The review will collect results of all types of outcomes from comparative studies. The

meta-analysis will reveal the magnitiude of potential changes and the discussion will address the

question if the effect level could lead to adversity. Hence, adversity will not be concluded from

the total body of data but “Net changes in the population abundances or biomass” will be

assessed.

9) (Ref: p 6, “Review Question”): Are population abundances of soil invertebrates changed by Bt crops

compared with conventional crops?

If an effect is detected there is no information about how this difference will be put into context.

(industry)

- Response: The possible difference will be put into context in a technical statistical manner. Only

in the final discussion of the meta-analysis will the level of differences be put into perspective,

but the systematic review is not a risk assessment analysis.

10) (Ref: p 6, “Outcome”): It is not clear what we want to protect. Only if this is clear it is possible to put

things into context. Do we want to protect soil function? Therefore a change in population abundance

is not enough to claim an effect. (industry)

- Response: As obvious from the protocol introduction and objectives the review exercise only

pertains to population changes. Apparently, Industry will give no attention to

population/biodiversity changes and only to ecological functions. This position does not agree

with present ecological effects assessment and European protection goals. Moreover, as links

between biodiversity and functioning is not fully understood, biodiversity per se needs protection.

The review protocol does not cover the selection of protection goals, which has been selected as

a prerequisite to the protocol.

Comments on Methods

11) (Ref: p 7, 4.1 “Search Strategy”): ‘The aim of the search is to find studies containing data from field

experiments assessing the effect of Bt crop cultivation to soil invertebrates’

How will be discriminated between different sources of data? (industry)

30

- Response: There are quality criteria and design criteria that have to be met by any study. These

are described in 4.1.3 to 4.1.5.

Comments on Search Strategy

12) (Ref: p 10, 4.1.2 “Web Databases”): (There is) the need to harmonize the list of databases between

all SR protocols. (CA)

- Response: There is no formal requirement for a certain Review Protocol to comply with a

general list of databases.

13) (Ref: p 10, 4.1.2 “Web Databases”): ‘For the further analysis studies which language of the original

full text is either English or German will be included’.

If the info would be in Spanish or Catalan, it would not be considered relevant. Spain is the most

important country cultivating Bt maize in Europe and you might be missing important bibliography.

(industry)

- Response: Studies in say Spanish or Catalan, will be considered relevant when they include an

English abstract, but we will not be able to include the study due to lack of language skills in the

review team.

14) (Ref: p 11, 4.1.2.2 “Personal Datasets”): ‘The complete list of articles received by personal

communication and the source will be recorded’.

Relevance of personal communication vs peer reviewed articles or published books? (industry)

- Response: The relevance will be assessed when employing the quality criteria of any study

either peer reviewed and non-peer reviewed.

15) (Ref: p 11, 4.1.3 “Study inclusion criteria”): ‘Relevant exposure(s): Field exposure to genetically

modified Bt crops and their concomitant farming practice through the soil environment and in the

rhizosphere’.

Definition of concomitant farming practice? (industry)

- Response: We substituted “concomitant” with “associated”. Any agricultural crop has an

associated farming practice.

16) (Ref: p 12, 4.1.3 “Study inclusion criteria”): ‘Relevant comparator(s): e Non-Bt near-isogenic crop or

another non-Bt variety of the same crop species in an experimental design allowing for any of the

comparisons’.

Comparison should be made only between GM and its non-GM conventional counterpart. Otherwise

there could be an effect of the variety. (industry)

- Response: We maintain to include studies with any variety of the comparator. However, we

suggest to perform an analysis that assesses the differences obtained with the group of

comparators of near-isogenic varieties vs. the differences obtained with non-isogenic varieties.

Hence in the section “4.3.4.1. Quantitative synthesis” we have further specified the intention of

including effect modifiers: “When the final dataset allows for assessing the effect of the effect

modifers on the outcome of the meta-analysis, such biases will be reported.”

17) (Ref: p 12, 4.1.5 “Study quality assessment”): ‘As specified in section 4.1.3 a study must have a

comparator’.

The comparator should be the conventional counterpart to exclude potential variety effects. (industry)

31

- Response: See previous reply.

18) (Ref: p 13, 4.1.5.1 “Selection bias”): ‘replications - low risk if there is at least five replicates per

treatment. High risk if there are no replications’. Why five replicates? (industry)

- Response: Five replicates is, from our practical experience, a common level of replication that is

able to detect acceptable magnitudes of differences. Below this number of replications detection

levels are dramatically increased.

19) (Ref: p 14, 4.1.5.5 “Reporting Bias”): ‘Reporting bias may occur - as a result of author's interest for

publishing negative or positive outcomes because of commercial or ideological reasons’.

Putting at the same level investments worth at least hundreds of thousands of euros with ideological

published documents is not fair. In addition companies produce studies according to international

agreed guidelines. (industry)

- Response: We agree that implying an influence on a study by authors with an interest in a

certain outcome is impossible. Hence, we can only select studies on the basis of the range of

quality criteria and not on suspicions of conflicts of interests.

Comments on Data extraction strategy

20) (Ref: Table 2, p 16): This table is different from the tables mentioned in the other SR protocols on

environmental effects. There no article_id, no variable referring to author's names, citation,... This

should be checked and improved accordingly to ensure a certain level of consistency. (CA)

- Response: Please note that article ID’s etc. are covered by the bibliographic reference code as

found in the forthcoming Endnote GRACE-WP5 library and database holding all the bibligraphic

information about the data source studies.

We have retained Table 2, as this has been developed specifically for the soil invertebrate

domain, with its particular needs and terms.

Comments on Data analysis

21) (Ref: p 18, first sentence): Please add: potential adverse effects. (industry)

- Response: See previous reply. Adversity is not dealt with in the collection and analysis of the

results.

22) (Ref: p 22, 4.3.4.8 “Sensitivity analysis”): ‘study funding’.

The funding of a study should not determine the outcome. There is only one science. Protocols, study

design, etc... can influence the study but not funding. (industry)

- Response: We fully agree, and have deleted this from the sensitivity analysis.

32

4) Systematic Review:“Target Insect Resistance Development and

Resistance Management of Bt-Crops – Review Question 2”.

Review Question: “Are data on base line susceptibility of different

lepidopteran/coleopteran maize pests to Bt-proteins available?”


1) The title is too broad to describe the objective of this review. (industry)

- Response: Accepted, title was changed.


below. (industry)

- Response: see below.





- Response: We do not agree with the stakeholder. It has been shown by studies in different

research areas that funding can influence the results of studies (e.g. Vartanian et al 2007

American J. Pub. Health, 97 pp667-375.)


4) (Ref: p 2, first sentence): ‘However, one concern with growing Bt-maize is the rapid evolution of

resistance’.

Replace by ‘potential for evolution of resistance’. Rapid needs to be defined. Is 10 years, rapid? Is an

insecticide supposed to avoid resistance development for more than 10 years? (industry)

- Response: Accepted, wording was changed.


5) (Ref: p 3, “Population”): ‘Additionally lepidopteran or coleopteran pest species should be considered,

which might be of economic relevance in maize cultivation.’

If they are not target species they would need to be controlled anyway using the common practices.

(industry)

- Response: The authors followed the recommendation of EFSA. EFSA is of the opinion that not

only the target pests defined by the applicant are relevant, but also other regionally economically

important pests which are sensitive to Bt toxins, They might have a potential to develop

resistance to Bt protein and therefore they should be considered in the risk assessment.

33

Comments on Methods

6) (Ref: p 3): ‘This review is part of the EU funded project GRACE to collate available evidence on

evolution of resistance to Bt-crops relevant in Europe.’

Looking for evidence already indicates a clear goal. (industry)

- Response: Comment is unclear!

Comments on Search strategy

7) (Ref: p 3, “Search terms”): Is this search aiming to address specific European questions or is it

supposed to be more a worldwide question? It is important to know as therefore the search should be

limited to a certain region. (industry)

- Response: The primary focus will be on European data. However the data outside Europe might

also be of interest and will therefore be considered.

Comments on Data analysis

8) (Ref: p 8, “Sensitivity analysis”): ‘Influence of funding sources for the overall results’.

Mentioning that funding can influence the outcome of a study is accepting that there are different kind

of science. (industry)

- Response: See above.

34

5) Evidence Map: “Target Insect Resistance Development and Resistance

Management of Bt-Crops – Review Question 4”.

Review Question: “Is the inheritance of resistance alleles fully recessive

in populations of lepidopteran/coleopteran maize pest?”


1) The title is too broad to describe the objective of this review. This document focuses exclusively on

maize. This should be acknowledged in the title. (industry)



below. (industry)








4) (Ref: p 2): ‘However, one concern with growing Bt-maize is the rapid evolution of resistance.’

Definition of rapid may differ depending on the stakeholder; therefore a definition would be helpful.

Resistance evolution against agricultural control measures are a well-known problem for more than

100 years. (industry)

- Response: Accepted, wording was changed

5) (Ref: p 2): ‘In consequence Bt-products might lose their effectiveness against pests, both as

conventional spray application and as transgenic trait of GM crops.’

Which kind of resistance management control is being done for the Bt sprayed products? Are they

also susceptible to trigger the development of resistance? What is the difference between GM and Bt-

spray? (industry)

- Response: We did not understand the question. The management of Bt sprays is not subject of

the evidence map.

6) (Ref: p 2): ‘Within the risk assessment of Bt-crops not the general risk for resistance evolution is

assessed, but the efficacy of strategies suggested by applicants slowing down the resistance

evolution’

There is a reason for this. Resistance development is not an environmental concern as this happens

in nature. (industry)


35

7) (Ref: p 2): ‘The goal of the SR will be to collect as much data as possible to have a broad data basis

to model and assess the potential for evolution of resistance of different crop-Bt-protein-species

combinations.’

SR is not defined earlier in the document. In addition the goal is not clearly defined (e.g. the

geographical area is not defined). Once there is a lot of data gathered which is the final objective?

(industry)

- Response: The goal of the protocol was changed from a systematic review to an evidence map.

Therefore the comment is not relevant any longer.


8) (Ref: p 3): “One key assumption for the high-dose/refuge strategy is that the inheritance of resistance

is fully recessive. Therefore data on the inheritance of resistance is needed to be sure about the

recessiveness of the inheritance in the target organism”.

The 5 assumptions of the high-dose refuge strategy are interdependent. None are all-or-nothing.

Moreover, it is illusory to believe that it is possible to predict with accuracy the impact that any

resistance will have on field-relevant resistance, prior to when field problems are experienced in

practice. (industry)

- Response: We agree with the stakeholder. Therefore we use the wording “one key

assumption…”.This should express that there are other assumptions which are also relevant for

the high dose refuge strategy. However this is stated also clearly in the text.

9) (Ref: p 3): ‘Additionally lepidopteran or coleopteran pest species should be considered, which might

be of economic relevance in maize cultivation’.

Resistance development to the Bt product is only of importance for the target organism. If the product

is not aiming at controlling other pest farmers should still treat. (industry)


Comments on Methods

10) (Ref: p 4): ‘This review is part of the EU funded project GRACE to collate available evidence on

evolution of resistance to Bt-crops relevant in Europe.’

Looking for evidence of evolution of resistance already puts some bias in the question. (industry)


11) (Ref: p 4, “Search Strategy”): ‘Both published and unpublished data should be collated’.

Unbiased: refer to previous comment. (industry)


12) (Ref: p 4, “Search Strategy”): ‘Different sources of information will be searched in order to maximize

the coverage of the search’.

How will the unpublished data be assessed? (industry)

- Response: The goal of the protocol is an evidence map. Therefore an intensive assessment of

the data will not be conducted. Only available information will be mapped.

13) (Ref: p 4, “Search Terms”, “Population”): This shows that the focus is maize specific. This should be

included in the title. If not Industry suggests to include experiences in cotton, rice and soya. (industry)


36

14) (Ref: p 4, “Search Terms”): ‘Non-English searches will be restricted to search engines that contain

non-English information (e.g. Google Scholar). One example might be German’.

Suggest to include Spanish as well. (industry)

- Response: We would like to include as much information as possible. However the research

team is small and linguistic knowledge is limited to few languages. At the end it is matter of

available resources.

15) (Ref: p 6, “Specialist Sources”): “Additional grey literature or other useful data will be provided by the

stakeholder group”

How will you differentiate between fact and replicated innuendo/fiction? Between referred papers

versus unreviewed conclusions? (industry)

- Response: The goal of the protocol is an evidence map. Therefore an intensive assessment of

the data will not be conducted. Only available information will be mapped.

16) (Ref: p 6, “Study inclusion criteria”): “that if there is any doubt about the relevance of the information it

will be retained.”

Industry suggests distinguishing between 'relevant' versus 'credible' versus 'science/data-based'

publications. (industry)

- Response: We will map available information. An assessment whether the data are credible is

not foreseen, relevant means here related to the research question. Criteria of relevance are

defined in the text.

37

6) Systematic Review “Effects of Bt crops on soil microorganisms” (NTO

review question 3).

Review Question: “Are soil microbial endpoints changed by Bt crops

compared with conventional crops?”


1) Changes are not bad per se. (industry)

- Response: The title is intended to be impartial by using the neutral term “changes” and it

reflects the exact objective of the review to collect data irrespective of whether a particular study

reports a change or no change. Likewise for the analysis that will provide statistics on the

presence and level of changes, without implicitly and explicitly providing a value judgement on

whether a change is good or bad. No changes made.


below. (industry)

- Response: We have taken action according to the specific comments below





- Response: We have excluded our indication that authors could introduce biases in the outcome

of their studies.


4) (Ref: p 5, Line 133): Suggest changing to “establish the potential adverse effects…” (industry)


5) (Ref: p 5, Line 137): Suggest changing “…establish the effects…” by “…establish the potential

adverse effects…” (industry)


6) (Ref: p 5, Line 147): “unknown compounds, as a result of unintended effects of the genetic

modification”

This is speculation. When a GM plant is approved for cultivation it has been thoroughly assessed.

What are the unknown compounds? (industry)

- Response: We consider it a well-established fact that unintended effects might occur due to the

various steps in the process of modifying the plant genome. Industry may consult the scientific

literature and retrieve the evidence for unintended changes due to genetic modifications. The

sentences have been rewritten to clarify of the statement.

7) (Ref: p 6, Line 162-164): It should be made very clear that the planned SR focuses on

microorganisms. And out of our perspective it remains open if there are enough data and knowledge

about soil microbial ecology, the diversity of microorganisms and soil and management conditions

etc. to conduct a SR with such a broad claim. If only comparable data will be included then the

38

conclusions will be restricted to these situations. Overall conclusions on soil ecology in general, given

the diverse taxa and conditions, will not be possible based on this approach. (CA)

- Response: The sentence has been modified to clarify that it SR is on microorganisms and

microbila ecology. It is not the intention to make general conclusions on soil ecology as such.


8) (Ref: p 6, Line 174-175): If a change is identified there is no information about how this difference will

be put into context. (industry)

- Response: This SR aims to make a synthesis of the empirical evidence on the effects of Bt

plants on microorganisms and their activities. So the major objective of the SR is to analyse

whether systematic statistical significant changes can be identified. From the result of this

analysis it will be assessed whether such changes are of functional significance.

9) (Ref: p 6, Line 184, “Outcome”): It is not clear what we want to protect. Only if this is clear it is

possible to put things into context. Do we want to protect soil function? Therefore a change in

abundance is not enough to claim an effect. As the function could be maintained. (industry)

- Response: We agree that changes may not always lead to disruption of ecosystem functioning,

however it is implicitly given that, if a change has surpassed a certain threshold, damage to

ecosystem functioning is likely to occur. Present risk assessment would consider changes in

biodiversity and changes in ecosystem functioning in their own right, hence a change only in

biodiversity could trigger a concern. The SR is not restricted to abundances of microorganisms.

10) (Ref: p 6, Line 184, “Outcome”): It may be worth assessing the heterogenity / variability of endpoints

without GMO. This may be informative for the selection of GMO-endpoints. (CA)

- Response: The establishment of baseline data are not within the scope of this project.

Comments on Methods

11) (Ref: p 7, Line 192): How will be discriminated between different sources of data? (industry)

- Response: There is quality criteria and design criteria that has to be met by any study. These

are describes in 4.2.

12) (Ref: p 7, Line 214): OR Mycorrhiza should be added. (CA)

- Response: The chosen search terms have been identified by an experimental procedure, which

ensures that relevant publications are identified without an overload with not relevant

publications. This means that recommended changes have to be based on thoroughful analysis.

13) (Ref: p 7, Line 215): Is there any reason why Cyt and Vip proteins from Bt are not searched for? (CA)

- Response: We have decided to restrict our efforts to cry toxins.

14) (Ref: p 7, Line 215): Industry suggests including Bacillus in the search. (industry)

- Response; The chosen search terms have been identified by an experimental procedure, which



15) (Ref: p 8, Line 217): Better *nitrific*(instead of nitrogen)? That would include denitrification and

nitrification but not ammonification. (CA)

39

- Response; The chosen search terms have been identified by an experimental procedure, which



16) (Ref: p 8, Line 218): This document includes other crops that are not in the application pipeline in the

EU for cultivation (e.g. Bt cotton, Bt rice, Bt soybean). How will these results apply to Europe?

(industry)

- Response: We intend to look for general patterns and therefore we have not restricted our

efforts to Europe. General patterns are indeed also applicable in an European context.

17) (Ref: p 10, Line 261-262): ‘For the further analysis studies which language of the original full text is

either English or German will be included’.

If the info would be in Spanish or Catalan, it would not be considered relevant. Spain is the most

important country cultivating Bt maize in Europe and you might be missing important bibliography.

(industry)

- Response: Studies in say Spanish or Catalan, will be considered relevant when they include an

English abstract, but we will not be able to include the study due to lack of language skills in the

review team.

18) (Ref: p 11, Line 283-285): ‘The complete list of articles received by personal communication and the

source will be recorded’.

Relevance of personal communication vs peer reviewed articles or published books? (industry)

- Response: The relevance will be assessed when employing the quality criteria of any study

irrespective of whether it is peer reviewed or non-peer reviewed.

19) (Ref: p 11, Line 296): Industry suggests including a definition of concomitant farming practice.

(industry)

- Response: We substituted “concomitant” with “associated”. Any agricultural crop has an

associated farming practice.

20) (Ref: p 11, Line 298): Comparison should be made only between GM and its non-GM conventional

counterpart. Otherwise there could be an effect of the variety. (industry)

- Response: We maintain to include studies with any variety of the comparator. However, we

suggest performing an analysis that assesses the differences obtained with the group of

comparators of near-isogenic varieties vs. the differences obtained with non-isogenic varieties.

Hence in the section “4.5.4.1. Quantitative synthesis” we have further specified the intention of

including effect modifiers.

21) (Ref: p 11, Line 308-310): Industry suggests specifying the kind of study design that are covered by

this case by case approach. (industry)

- Response: The text has been modified to “if the design is explained in details and makes

statistical analysis valid”

22) (Ref: p 12, Line 336): An appropriate comparator to evaluate the potential effect of a Bt variety should

be the conventional counterpart. Otherwise there is the possibility that any detected effect would be

due to the variety chosen. (industry)

- Response: See above, comment 20).

23) (Ref: p 12, Line 338): It may be good to express this more precisely. As we understand you will

assess different types of bias and therefore cannot exclude it a priori. (CA)

40

- Response: As we do not know which kinds of experimental procedures the SR will have to

assess is it hard to define this more precisely.

24) (Ref: p 13, Line 344): Does it mean that there is only one reviewer for the quality assessment and a

second reviewer only in case of doubt. We would propose to have the same procedure as for the

inclusion criteria as bias can especially be a problem in defining the quality of a study. (CA)

- Response: This has now been clarified in the text.” Studies which fulfil the inclusion criteria will

be assessed for possible bias of one reviewer and a second reviewer and studies that have been

excluded will be reassessed by a second reviewer and a conclusion will be reached.

25) (Ref: p 13, Line 344, “Inclusion criteria”): Should it be quality criteria? (CA)

- Response: No it is inclusion criteria.

26) (Ref: p 13, Line 374): Use of pesticides may be a critical issue; not only fungicides but also other

pesticides may disturb the microbial community in soils. (CA)


27) (Ref: p 14, Line 395): “Reporting bias”.

Putting at the same level investments worth at least hundreds of thousands of euros with ideological

published documents is not fair. In addition companies produce studies according to international

agreed guidelines. (industry)

- Response: We agree that implying an influence on a study by authors with an interest in a

certain outcome is impossible. Hence, we can only select studies on the basis of the range of

quality criteria and not on suspicions of conflicts of interests.

28) (Ref: p 16, Table 2 – “Pesticide treatment”): Please differentiate pesticides: fungicides, herbicides,

insecticides, nematicides… (CA)

- Response: Accepted – and now specified in the table.

29) (Ref: p 16, Table 2 – “Crop rotation”): Should be specified in more detail. Do you want a yes/no

answer? (CA)

- Response: accepted – and now specified in the table.

30) (Ref: p 17, Table 2 – “Comments”): Physico-chemical (WHC, Corg, etc.) data of the soil and data

about the weather (temperature, rainfall etc.) during the experiment are missing. (CA)

- Response: Now added as an additional point in the table.

31) (Ref: p 20, Line 525): The funding of a study should not determine the outcome nor the sensitivity.

There is only one science. Protocols, study design,... can influence the study but not funding.

(industry)

- Response: We fully agree, and have deleted this from the sensitivity.

41

Chapter 2: Health Impacts of GMOs

General Comments

This section refers to comments that are not linked to a specific protocol but that could be of relevance for

most or all of them.

1) Bias is evident throughout the language of the document.(industry, on Draft Protocols described in

Chapter 1-3).

2) It is not clear how the outcome of this document can help risk assessors/managers in Europe.

(industry on Draft Protocols described in Chapter 1-3)

3) [..] I am still not certain how these information maps will help regulators or public scientists better

judge the safety and adequacy of the many GM crops that have been developed and are available on

the market in at least some countries. It seems very important that there be some critical review,

rather than face-value acceptance of publications and data as equivalent in rigor. I think that is a

major hurdle facing the GRACE project. (Academia/Scientific community)

4) Potential conflict (competing) of interest of the GRACE panelist: [..] there is no declared competing

interest in these documents. However, some of the authors are being funded by the EU or EC to

perform research on animal models, composition and other parameters. Isn’t that a completing

interest? Since I have studies funded by biotechnology companies and developers, it seems I could

not participate as an advisor to EFSA or the EC. Thus it seems a bit disingenuous to not declare the

interests of the authors, and for the EC/EU and others to not recognize that everyone has some

competing interest. A declaration would help add transparency. (Academia/Scientific community)

1) Evidence Map “What are the characteristics of toxicity studies in which

animals receiving newly expressed proteins from GM crops are

compared to animals that are administered appropriate controls

without these proteins?”


1) Whole objective of this study is not clear. Review publications on feeding studies and toxicity are

already available. Also, applicants have to do the same exercise through literature reviews. Over

many applications, EFSA will have a nearly complete database to refer too. So the purpose of the

proposal is unclear.( Industry)

2) The toxicity DRAFT report is also very general and not very useful. In addition, the assumption that a

repeat dose toxicity study is needed and an acute dose test is not adequate or helpful seems to be a

hypothetical leap of faith. (Academia/Scientific community)

3) Where is the evidence that either assay is good or bad? Where is the data to demonstrate the kind of

toxins that can be detected by either? Which test would identify something like a ricin or a botulinum

42

with the minimum amount of animals, the minimum uncertainty? Where is the data to demonstrate

that a major anti-nutrient is introduced into food? And where is the description of processing to mimic

the normal processing of food or feed based on the type of crop? The need for a map to uncover

important information was needed when EFSA recommendations were made that led to the EC

regulation. Now, to assume that the EFSA and EC have plotted an appropriate test strategy for

uncovering potential toxins is a bit late. (Academia/Scientific community)

Comments on Abstract

4) (Ref: p 2, Line 50): In combination with the previous sentence it is implied that the term “newly

expressed proteins” only encompass proteins that are intended to be expressed in the transgenic

organism and are encoded in the transgenic construct intended for insertion. In the explicatory note,

however, newly expressed proteins are described to also arise from unintended coding sequences in

the junction regions between inserted and native host DNA. It would clarify the scope of the protocol if

sources for newly expressed proteins are clearly stated in the beginning. (CA)

5) (Ref: p 2, Line 61): It is implied here that only studies with purified protein are investigated. Later in

the text it is not unambiguously stated if not also GM plant material containing newly expressed

proteins is considered. (CA)

6) (Ref: p 2, Line 66): (cf.comment Line 61). It should be clarified that only feeding studies with purified

protein are considered and that such proteins can be purified from the GM plant in question or that an

analogous protein produced by microorganism is used. (CA)

7) (Ref: p 2, Line 74-76): This part of the sentence is confusing. Is it not relevant to assess if effects are

observed and if theses observed effects are considered adverse by the authors? Why is it important

to assess if authors describe adverse effects without evaluating the reported effects per se? Or was

the intention to state that the assessor of the studies will not evaluate the described effects? In this

case not only adverse effects should be considered but all observed effects. It can be noted that the

authors of the study consider some/none/all observed effects to be adverse. (CA)


8) (Ref: p 4, Line 118-121): According to James (2012) more than 95% of the GM crops planted

presently and in the past are herbicide and/or insecticide resistant. In this respect, the elaboration

giving here is misleading, because it pretends that relevant areas are grown with GM crops of the

other mentioned traits. (CA)

9) (Ref: p 4, Line 123): The description in this para is misleading in several respects: Insecticidal Cry

proteins in GMOs are already activated and possibly further modified. Therefore, they cannot be

directly compared to Cry proteins in B. thuringiensis with respect to food and feed safety. The

digestive system of humans and livestock is different from the one in insects. Also, specificity of Cry

proteins expressed in GMO and B. thuringiensis to NTO might differ. (CA)

10) (Ref: p 4, Line 125): (cf. comment Line 61); the explicatory note also includes unintended protein

coding sequences in the junction regions (CA)

11) (Ref: p 4, Line 126): Sugest to change ”are insecticidal Cry proteins into “are modified insecticidal Cry

proteins”. Please clarify that the Cry proteins expressed in the GMO are not identical to the ones

expressed in B. thuringiensis, but already activated and possibly further modified. (CA)

43

12) (Ref: p 4, Line 136): Cross out “residues” here, because it complicates the picture and does not help

with understanding. (CA)

13) (Ref: p 4, Line 137-138): This example is much more important in terms of acreage planted with GM

crops. Therefore it should come first. (CA)

14) (Ref: p 5, Line 172-176): Please put the emphasis on the newly expressed protein into perspective,

because there are several other important issues which should determine the need for further studies:

(i) GM crops with herbicide resistance allow to increasingly apply complementary herbicides; (ii) some

traits do not involve expression of a new protein; (iii) interactions in stacked events can impact a

plant’s metabolism. (CA)

15) (Ref: p 5, Line 178): EFSA guidance foresees that basic characteristics of a newly expressed protein

should be considered, i.e. conditions for optimum activity, substrate specificity etc. Please mention

these aspects as well. (CA)

Comments on Objective

16) As commented upon with the evidence map for testing whole GM food/feed, the PICO concept is of

no added value in the risk assessment of GMO derived foods/feed. (Academia/Research

Community)

17) The draft protocol does not cover the characterization of the newly expressed protein, i.e. history of

safe use, which may limit the necessity of animal testing. (Academia/Research Community)

18) The draft protocol does not cover other testing/evaluation strategies like in-vitro testing and sequence

homology evaluation. (Academia/Research Community)

19) The proposed draft protocol does not provide guidance on how the results from the available studies

should be interpreted and characterized regarding their impact on human and/or animal health (see

comments on whole GM food/feed testing). (Academia/Research Community)

20) (Ref: p 6, Line 214): Please add “purified” [“purified newly expressed proteins”] as clarification (CA)

21) (Ref: p 6, Line 233): Shouldn’t all changes be considered at first, before they are ranked as adverse?

Which change is considered adverse might be subjective and the study authors interpretation of the

results should be noted separately. (CA)

22) (Ref: p 7, Line 248): Please consider a) whether the applied proteins were identical (microbial or GM

crop derived) and b) in case the applied protein was not derived from the GM crop, in what way they

were different. (CA)

Comments on Methods (with potential relevance for the Appendix)

23) The assessment and selection of the data package (peer-reviewed, non peer-reviewed, grey

literature) is not well described and there is a lack of clarity on what will be included and

excluded.(industry)

24) No explicit criteria listed on how quality of each reviewed paper will be assessed.(industry)

25) In order to use data provided by applicants in dossiers, there should be agreement from the

applicant.(industry)

26) The proposed approach of reviewing summary information will not be sufficient, full access to the

study reports will be needed. Therefore it is not clear whether the goal of the intended studies can be

completed.(industry)

44

27) Contains only EU links. Other authority reviews such as CAN, US, AUS and ILSI links should be

considered.(industry)

28) The protocol contains some useful information on search strategies for information available on

animal testing of newly expressed proteins in GM crops and derived food/feed It is useful to

standardize the available information on performed studies in order to judge on the quality of

generated data and whether comparison of the results of different studies is possible.

(Academia/Research Community)

29) For toxicity, it is not clear how publications from researchers who are not developers of a product will

be evaluated or judged for inclusion. Or how they might be weighted. For instance, there are many

studies that are questionable in terms of controls, test materials, control materials or methods. Are

these to be included with equal weight of prominence as controlled studies that are performed

following OECD guidelines and with well described and control test substances? (Academia/Scientific

community)

30) Are these only going to be studies in which purified proteins are included? Or will it include “toxicity”

studies that are really whole food feeding studies (e.g. 90 rat)? And even longer studies.

(Academia/Scientific community)

31) If pure protein only, then what control proteins are used and how are they characterized? How many

studies have used positive and negative control proteins in acute or repeat dose testing? What

characteristics would be reasonable to select control proteins? Mechanism of action? It would seem

that toxicity tests should have a hypothesis that is reasonable and specific for the type of

protein/event. It is not clear to me that the MAP has outlined such criteria in sufficient detail to ensure

an adequate review of data. (Academia/Scientific community)

32) At what level will the data / studies be evaluated? And how will they be evaluated in terms of judging

the adequacy and accuracy of results? It seems there is no value in simply listing all studies that you

can find. The value would be in providing some expert opinion about the validity of the studies or at

least some critical review. Even in “peer reviewed” literature, there are many studies that have been

soundly questioned (e.g. Seralini’s 2012 2 year rat study, which has now been retracted from the

journal). If poorly controlled studies are presented with equal weight to studies that have sound

characterization of test materials, transparent protocols (whether OECD or not), with sound scientific

hypotheses, then there will be no value in the GRACE database. (Academia/Scientific community)

33) (Ref: p 9, 3.2.4) – Databases – incomplete list. (industry)

34) (Ref: p 9, 3.2.5) – Not sure Google Scholar is the best search engine. (industry)

35) (Ref: p 9, 3.2.6)– What is meant by grey literature? – either it is scientific or not. (industry)

36) (Ref: p 12, Line 360): It should be clarified that purified protein from the GMO or microbially produced

is meant. And that proteins encoded in the junction regions are also considered. (CA)

37) (Ref: p 15, Table 3); Newly Expressed Proteins,”traits introduced“:It should be noted if the protein is in

any way modified from its original source (codon optimization, truncation etc.). (CA)

38) (Ref: p 16, Table 3), Animal Experimentation Set Up, “mixing with diet”: It should be noted whether

the diets contain other GM crop derived components, if pesticide residues are present and if

medication was part of the diet (e.g. salinomycin, a regular feed supplement in industrial poultry

farming, is not only effective against protozoa and Gram-positive bacteria, but also kills or inhibits

45

human cancer stem cells and breast cancer cells in mice (Naujokat et al. 20103). This compound may

well suppress the development of cancer in broiler chicks and cover any related adverse effects).

(CA)

Comments on Appendix

39) (Ref: p 19, 8.1): Sample data extraction sheet, “recombinant micro-organisms”: In this case, it should

also be evaluated if data on the suitability of the analogue is provided (N-terminal sequencing;

glycosylation; molecular weight; enzymatic activity, substrate specificity, etc. in comparison to the

plant derived protein). (CA)

40) (Ref: p 20, Ibid): “characterisation”: t should be noted if the protein is in any way modified from its

source (codon optimization, truncation etc.). (CA)

41) (Ref: p 21, Ibid): “diatary protein constituent”: The diet should be defined and it should be noted if the

diet contains other GM crop derived components, if pesticide residues are present and if medication

was part of the diet (e.g. salinomycin, a regular feed supplement in industrial poultry farming, is not

only effective against protozoa and Gram-positive bacteria, but also kills or inhibits human cancer

stem cells and breast cancer cells in mice (Naujokat et al. 2010). This compound may well suppress

the development of cancer in broiler chicks and cover any related adverse effects.) (CA)

42) (Ref: p 27, Line 143-149): “Route of administration” is explained twice with different texts. (CA) (Note:

this is an editorial comment and will not be present in the final version).

43) (Ref: p 28, Line 158): At what degree of modification is a protein to be considered of coming from

“nosource”. Does truncated, codon optimized protein already qualify? (CA)

GRACE team sumary response

The general and detailed comments made by the stakeholders were received in good order and carefully

considered with a view on possible modifications that had to be made to the draft protocol so as to

address the points raised. Various additional edits of an editorial nature were made, without impacting on

the document’s contents.

A summary of the stakeholders’ comments is provided below, grouped together under headings of the

topics addressed by them:

1) “Abstract”, “Background”, “Objective”, “Methods”: Several respondents submitted various comments

pertaining to the description of newly expressed proteins, both as subject of the evidence map

(encoded by target genes or also by other newly created reading frames, tested in purified form or

not, only plant-produced proteins or also analogues produced using recombinant microorganisms,).

One respondent also commented to the example of such proteins occurring in commercialized crops

(asking for more detailed clarification that these proteins are not exactly the same as in the gene

donor Bacillus thuringiensis).

- Response: Apparently a more detailed definition of “newly expressed protein” is warranted, yet

the abstract cannot be the place for a great level of detail, nor has it background text been

intended to narrow the focus of the research (i.e. to exclude newly expressed proteins that would

not fall into a narrow definition). In various places in abstract, background, objective, and

3 Naujokat C.,Fuchs D. and G. Opelz. (2010): Salinomycin in cancer: A new mission for an old agent.

Molecular Medicine Reports 3: 555-559. http://www.spandidos-publications.com/mmr/3/4/555

http://www.spandidos-publications.com/mmr/3/4/555

46

methods text has been added so as to clarify that the term “newly expressed protein” as used in

this protocol pertains not only to the sequence being part of the intended modification but also

any other modified sequences (truncated, fusion proteins, products from open reading frames)

expressed by genetic material formed upon genetic transformation of the host plant. A number

of features such as differences as compared to the naturally occurring protein and the gene

encoding it (e.g. codon optimization, truncation) were also referred to as details to be included in

the data extraction yet, while this information per se can in some instances be useful as it helps

to identify the exact nature of the protein used (as part of its characterization) and also affects

the comparability between studies with a given protein (depending on whether the protein used

was the same or similar except for a few differences), the primary focus is on the characteristics

of the studies employing these proteins thus also whether such information was included

instead of the data per se. Data on the characterization of the administered newly expressed

protein and other test materials is part of the items to be collected as summarized in the data

extraction sheet (see protocol appendix).

2) “Background”: One commenter provided multiple comments about the presentation of context and

additional features of the newly expressed proteins, apparently perceiving that the importance of

other traits besides insect and herbicide resistance was overrated in the background description

(based on the fact that the majority of the acreage of GM crops expressed either or both of these

traits), while also the fact that the expression of novel proteins may not always be the target of

modification plus that the introduction of herbicide resistance might give rise to altered metabolite

profiles and that stacking of genes could also interfere with a plant’s metabolism. Another commenter

argued that also other characteristics, such as history of safe use, should be given attention to, as

this would be an important factor in deciding whether an animal study is needed at all or not.

- Response: The focus of the current paper is on newly expressed proteins; issues of interest yet

secondary importance regarding plant intrinsic metabolism were within the scope of several

other proposed questions, one of which has also been prioritized for evidence mapping(i.e. the

map on compositional studies). While the metabolism of pesticides does not pertain to the

potential toxicity of the newly expressed protein per se (i.e., it is a common part of pesticide

safety assessment it is clarified now, under “concern/problem” in the background section, that,

although important, they are not within the scope of the envisaged map. Considering the

purpose of comprehensively mapping evidence on the characteristics of safety studies with

newly expressed proteins, it does not seem warranted to limit the scope to newly expressed

proteins from only the major GM crops (insect-resistant, herbicide-resistant) planted around the

globe. The experience with (safe) use of the protein is now being referred to in several places.

3) “Background”: Regarding the nature of the report, one of the respondents argued that the protocol

was too general and apparently perceived there to be an underlying assumption that repeated dose

studies were considered more useful than single-dose acute toxicity assays (by citing EFSA), while

arguing in a separate comment that there is no evidence to judge one assay to be better that the

other, while also going into the sensitivity and suitability of animal models to detect changes in GM

crops.

- Response: One of the issues to be mapped is which types of studies (both acute and repeated-

dose) are performed with the newly expressed proteins. Repeated-dose studies are referred to

in section 3.5.1, but in a different context, namely the fact that these different types of studies

may lead to heterogeneity in the selected records. The sensitivity to detect other changes in GM

foods appear to refer to whole-food feeding studies, which are out of scope. The outcomes of

the studies (e.g. sensitivity towards positive controls included in the same study, if applicable)

47

are not evaluated in an evidence map but might be the subject of a further systematic review

based on the outcomes of the evidence map.

4) “Objective”: One of the commenters argued that reviews on toxicity studies were already

available and that, in the frame of applications for regulatory approval, such reviews are carried out

by applicants, as well as the fact that risk assessors (EFSA) should already have a record of

experience with this topic based on the many applications already assessed.

- Response: We are not aware of an evidence map previously published on the same topic. The

proposed evidence map is distinct based on the following considerations:

The evidence map will focus particularly on features of the selected studies, such

as design, test materials, and reporting formats

The outcomes are to become publicly available and to inform different groups of

stakeholders (contrary to dossier information) in line with the overall objectives of

the GRACE project

The methodology is to be described in such detail such that it can be readily

performed and updated by others

5) “Objective”: Several comments pertained to the type of animal study to be focused on, with one

commenter recommending narrowing this to feeding studies (also pointing to the potential impact of

other contaminants in feed in a separate comment), while others asked for more clarification on the

kinds of toxicity studies to be included, and what the comparators were and how they were

characterized . In a broader sense, some comments also referred to other types of studies (e.g.,

bioinformatics and digestibility) and considerations (e.g. known experience of safe use) besides

animal studies performed to assess the potential toxicity of newly expressed proteins.

- Response: While it is realized that the safety assessment of a newly expressed protein is based

on a number of additional considerations besides animal toxicity studies with these proteins (if

carried out), the questions to be addressed should be sufficiently targeted so as to be able to

adequately define search criteria and carry out these searches A number of these issues will

actually be among the details to be mapped, i.e. they are not used as selection criteria, although

they might be so in a follow-up systematic review based on this map. The focus will be on

studies in which the newly expressed protein is administered to the animal in a purified or semi-

purified form, and is not intended to be narrowed down on specific types of study. The latter will

actually be the subject of investigation, i.e. to map out, among others, the various types of

studies done, in particular the route of administration (e.g. oral, intraperitoneal), the form in which

materials are administered (e.g., oral gavage, included as feed ingredient), what the comparators

were (e.g. empty vehicle, protein solution in vehicle), and the origins and characterization of the

newly expressed proteins (e.g. purified from GM plant tissues or analogous protein from

recombinant microorganisms; amino acid mutations/extensions as compared to plant-expressed

protein). Additional text has been inserted in the various sections (including background,

objectives and methods) to clarify that the focus is on experiments with newly expressed proteins

in purified and semi-purified forms.

6) “Objectives”: According to one commenter, it should be clarified whether the tested protein was

derived from the GM plant itself or whether an analogue produced in recombinant microorganisms,

and, in the latter case, what the differences were between the latter and the plant-expressed protein

actually being replaced by it.

- Response: The issue is indeed relevant yet the objectives section may not be the most

appropriate place to deal with the issue. In Table 3 (factors and examples of characteristics to be

48

mapped), the use of analogues and the characterization of these analogues is paid attention to.

An additional item concerning this is now mentioned in the data extraction table.

7) “Objectives and Methods”: Several commenters directly or implicitly referred to the evaluation of

outcomes of studies, e.g. by referring to lack of criteria for this purpose or by raising concerns that the

outcomes of studies of varying design and rigor (e.g. with test materials of different quality; use of

appropriate control proteins) were to be evaluated as if they were equals.

- Response: The purpose of the map is exactly to highlight these differences in design and other

methodological items of the studies with newly expressed proteins. Evaluation of the outcomes

of these studies will not be part of the evidence map, while this could be the purpose of a follow-

up systematic review based on the map (no such review has been scheduled within the GRACE

project, though).

8) “Methods”: One of the comments raised the issue that studies not performed by the developers of

the actual GM crop expressing the newly expressed protein being investigated, as it implicitly was

considered that these external authors might not dispose of the appropriate test and control materials

as well as non-adherence to e.g. OECD guidelines. Another commenter advised to standardize

methods/data based on the outcomes of the map so as to ensure that valid comparisons between

different studies can be made.

- Response: the evidence map is envisaged to highlight these and other differences in test

methodology applied. Identified differences and similarities between studies can help to assess

the comparability of these studies for further reviewing purposes (although the reviewing per se

is not part of this activity).

9) “Methods”: There was a comment regarding the perceived lack of utility of summaries of risk

assessment data, stating that full access to the original study reports is needed. In addition, it was

stated that for use of application dossier data, permission from the applicant is needed.

- Response: There is a varying level of detail in the summaries of the experiments as provided by

the regulatory authorities of the various countries on their websites (for example, the summaries

prepared by Food Standards Australia New Zealand are relatively detailed), while some

authorities also provide access to original submission documents (USDA, India, FSANZ) in some

cases. It is indeed intended to use only data that are sufficiently detailed for this purpose. It is

intended to obtain these data from accessible sources (hence not from confidential dossier data),

thus ensuring that the outcomes can be reproduced by third parties wishing to repeat or update

the map.

10) “Methods”: Various comments referred to the data sources mentioned, for example pertaining to the

perceived lack of non-EU links, the suitability of Google Scholar outputs for inclusion in search

results, and the definition of grey literature sources.

- Response: Google search as information source has been removed from the methods section

list of resources. Several of the links listed refer to international databases that contain (or link to)

the risk assessments carried out by non-EU authorities as well (CERA-ILSI, CBD-BCH, ISAAA,

Euginius), while grey literature sources are defined in 3.2.6 (these also have to comply with the

same inclusion criteria as reports from peer-reviewed literature).

11) “Methods”: A comment pertained to potential presence of other bioactive substances (antibiotics

possibly used in poultry diets) in the diets if proteins were to be incorporated into them as route of

administration.

49

- Response: While the experimental setup for all animals is supposed to be the same (including

diets) except for the newly expressed protein and its comparators being administered, the

question may also remind us of a more generic issue, namely the potential presence of other

constituents in the test materials besides the tested protein. In the data extraction sheet, this

item has therefore been added as descriptor under “test material”.

12) “Methods”: One commenter argued that the PICO approach would be of little value to risk

assessment.

- Response: It is not the purpose to perform risk assessments per se, while the map is envisaged

to provide an overview of the characteristics of the various studies performed with newly

expressed proteins per se, which may also inform risk assessment professionals in a more

general sense about the similarities and differences encountered and possible opportunities for

harmonization.

13) “Conflict of Interests”: One of the comments related to all protocols pertained to perceived interests of

team members, in the topic of investigation, based on their involvement in EU projects and risk

assessment activities (CA).

- Response: The section on “competing interests and sources of support” has been extended to

accommodate the issue.

50

2) Evidence Map “What are the characteristics of comparative studies of

changes in the levels of key chemical crop constituents in GM crops

compared to non-GM crops?”


1) It is not clear to me if these roadmaps and the investigations or searches are intended to only include

studies that were performed with GM and nearest isogenic lines or if it is intended to provide more

useful information about the composition and or toxicity of the broad spectrum of crops of the same

type that have been modified. The broader studies are really needed to understand the spectrum of

acceptable foods and ingredients in the various food crops and tissues consumed. Otherwise it would

be quite helpful if this is really narrowly focused, to explain that in the beginning. Are you looking for

real risks and natural variability, or are you simply looking to look at “published” data on GM crops. In

addition, if it is only to review the data on GM, then why doesn’t the map include a list of GM products

that have been evaluated and approved somewhere in the world? (Academia/Scientific community)

2) In addition the documents do not provide any guidance on the level of difference or the characteristics

(components) that are important to focus on that would indicate a potential risk. Those should of

course be related to the species of the crop. (Academia/Scientific community)


(industry)

4) One of the most important erroneous assumptions made in documents describing comparative

evaluation of GM crops relative to safety is the assumption that any measurable difference in

composition of a GM crop relative to measurements made of non-GM crop of the same species, and

specifically to a near-genetic comparator is that a difference indicates: 1) any change is caused by

insertion of the gene, or action of the products and 2) that any change might indicate a risk for either

human or animal consumers. It seems from these documents that it is the position the GRACE group

has taken. There are thousands of measurements that could be made on any food crop/material and

compared between a GM and a “near-genetic” comparator. That does not mean that any difference

indicates potential harm or that the difference is due to insertion of the gene. PLEASE [emphasis

added by commentator] consider modifying the statements in the documents to make that clear. The

way they are written in such a vague way, one might assume any difference is of concern. In addition,

there has not been establishment of any boundaries of how much difference is needed to be

considered important. Just because it is statistically significant (which isn’t clear), does not mean it is

biologically relevant. (Academia/Scientific community)

5) In the case of the GRACE documents it seems that the authors are not recognizing that there are

many genetic differences between the GM and any non-GM variety or line for any sexually

reproduced organism. Even “near-isogenic” crops are often more than 3% different in genetics due to

breeding. For instance, wheat transformation occurs in a relatively old line of wheat that is really not

used much in cultivation for production. It is used as it is easy to transform relatively to the more elite

germplasm. That means that the GM wheat has to be crossed and backcrossed around 6 times to get

to about 97% homozygousity with an elite line. Three percent is a lot of difference (potentially) in a

crop with ~ 10,000 to 20,000 genes. That has nothing to do with the GM event. So, somehow the

“map” or comparative data should be taking into account the important genetic diversity, in addition to

environmental diversity. Then of course there is a failure discuss the environmental interactions. Each

plant is an individual and the exact response is an interaction of genes and environment.


51

6) It seems to me that the focus of the assessment should begin with an evaluation of the type of

components for a given crop species (non-GM) that have been identified as presenting some health

risk: e.g. pharmacological affect, toxic effect, anti-nutrient effect. For many of the crops used for food

and feed, there are already characteristics that are known for the given crop type that have a health

impact. In addition, the variation in all components of any crop will not be known and the impact of the

range of environmental differences that may exist during the cultivation of the crops and the impact of

the differences will not be fully known. (Academia/Scientific community)

7) It is not clear from the examples that are provided in the document that it is clear what is being looked

for. The etc., etc., etc., are just confusing. Why not some real examples? (Academia/Scientific

community)

8) In order to use data provided by applicants in dossiers, there should be agreement from the applicant.

(industry)

9) Why are only government and agency websites mentioned, which for a great part do not contain

information on compositional characteristics, while other databases are available (ILSI)?


10) […] where is a discussion about finding differences of significance in terms of health risks?


11) There is also a failure to discuss the components that are important for individual crops. Only certain

metabolites are important for maize, others for soybean, others for apples, wheat and so on.


12) There is a lack of discussion regarding the proper tissues to sample, which will of course be plant

specific. Yes there is a question of which tissues, but it makes sense to test only samples that are

consumed by humans or agriculturally important farm animals to evaluate food and feed safety.


13) This protocol contains useful information on gathering data on the comparative compositional

analysis of GM crops and their respective non-GM counterparts. In particular Appendix 8 is useful for

data reporting. Compositional analysis is not only targeted at unintended effects but also at intended

changes (line 77). (Academia/Research Community)

14) The role of non-targeted analysis (metabolomics) in compositional analysis remains unclear in the

document (validation, recognized methods, targets, variability of patterns, statistical handling).


Comments on Title

15) Title „…comparative studies of changes in the levels of key chemical crop constituents…“: The title

suggests that changes in key chemical crop constituents between GM crops and non-GM crops are

observed in any event. Suggest to change to „…comparative studies of assessment of levels of key

chemical crop constituents…“ or alike. (industry)


16) (Ref: p 2, Line 48-49): „which considers the differences between the GM crop and a conventional,

non-GM counterpart with a history of safe use.“ p.3, lines 52-53 „Based upon the differences

identified between the GM crop and its counterpart.“ p.4, lines 69-72 „The differences thus identified

between the GM crop and its counterpart“: see [comment on title]. Differences between the GM crop

and the conventional counterpart should not be generally postulated. (industry)

52

17) (Ref: p 2, Line 59-61): „The proposed evidence map will help to gain insight into these different

methods used to carry out compositional studies with GM crops“: Suggest to evaluating also the

suitability and applicability of these methods for risk assessment of GM crops. Otherwise it will be an

academic exercise only without any further impact. (industry)

18) (Ref: p 2, Line 69-72): „Information on the compositional characteristics of a given GM crop and its

counterpart will be obtained from summaries of the data package that is part of applications for

regulatory approval of GM products submitted by companies to regulatory authorities“: Data in

dossiers are solely submitted to regulatory authorities in the frame of risk assessment to obtain an

approval for the respective product. The data cannot be used for any other purposes without the

explicit agreement from the applicant. (industry)


19) (Ref: p 4, Line 129-130): „Information on the safety of GM crops can be found in a range of

information sources, including scientific literature“ and lines 137-139 „reports in scientific literature by

authors from a more diverse range of backgrounds, carried out for different purposes, focusing on

different aspects“: Suggest changing lines 129-130 to „Information on the composition of GM crops

can be found in“. As indicated in lines 137-139, the purpose of many scientific studies might not be a

safety assessment, therefore conclusions on the safety of a GM crops cannot necessarily be

deduced. (industry)


20) It seems that the focus of the GRACE documents are too narrow in not looking for compositional data

from the broad spectrum of plant varieties and lines that are acceptable for consumption. An example

of statements that lead me to those conclusions are in section 2 of the Draft Protocol Evidence Map

Compositional Analysis, showing 2.1, Primary question and 2.2, Secondary considerations. When the

basic assumption underlying the document are so general and erroneously focused, the scientific

value of the document should is in question. (Academia/Scientific community)


literature) is not well described and there is a lack of clarity on what will be included and excluded.

(industry)

22) No explicit criteria listed on how quality of each reviewed paper will be assessed. (industry)

23) There is no information on quality assessment of available studies and further analysis/evaluation of

potentially identified compositional differences (ranges of variability, available databases,

difference/equivalence testing). (Academia/Scientific community)

24) (Ref: p 6, Line 221-222): „What are the characteristics of comparative studies of changes in the levels

of key chemical crop constituents in GM crops compared to non-GM crops“: The relevance of the

objective is questionable. It is not clear how assessment of study characteristics alone should help

stakeholders. See also [comment on lines 59-61]. (industry)

25) (Ref: p 6, Line 231): How is the history of safe use defined? It is defined for the crop or the variety?.

(CA)

26) (Ref: p 6, Line 232): „any changes in the key compositional parameters of the GM crop“: See

[comment on title]. Will only those studies/analytes be considered where differences in composition

were observed? (industry)

53

27) (Ref: p 7, Line 246): „Is the non-GM counterparts relatedness to the GM crop described?“: The nature

of the non-GM-counterpart is listed here as a secondary question, whereas in line 229-231 clear

requirements about the nature of the control are predefined. (industry)

28) (Ref: p 7, Line 252): The bulleted considerations outlined on p 8 of the document are generally quite

important. However, the statement on line 252 is not quite correct. The plant has not been subjected

to herbicide tolerance, stress resistance etc., rather the purpose of the modification would be to

provide herbicide tolerance, resistance to insects or draught and other intended modifications.


29) (Ref: p 7, Line 269): The quantity of analyzed parameter, missing data, the statistical power and the

rate of occurrence of statistic significant differences are important characteristics to evaluate the

conclusion of the authors and should be added here. (CA)


30) (Ref: p 8, Table 1): [..] analytical chemistry is not an outcome as such. (Academia/Scientific

community)

31) (Ref: p 10, Line 341-344): “Assessment of grey literature, e.g. personal communication”: How will this

kind of information be weighted compared to peer-reviewed articles? (industry)

32) (Ref: p 10, Table 2): “Organizational websites/ Government and agency web sites and libraries”: It is

unclear what kind of compositional data can be retrieved from the indicated websites. (industry)

33) (Ref: p 10, Line 356 ff): „Study inclusion criteria“: We recognize that scientific quality is not

considered in the criteria to include a study or not. (industry)

34) (Ref: p 14, Table 3): Crop plant material, “How many plants were sampled?”: Please add: “Are the

effects of different environmental conditions assessed in the studies?” (CA)

35) (Ref: p 15, Table 3): Crop compositional analyses, “Statistical treatments, if applicable?”: Add the

bullet points “The quantity of analyzed parameter”, “the statistical power”, “the quantity of missing

data” and “the rate of occurrence of statistic significant differences”. (CA)

(Ref: p 16, Table 3): Reporting of results and discussion, “What conclusion do the authors draw on

possible compositional changes observed, if applicable?”: The quantity of analyzed parameter,

missing data, the statistical power and the rate of occurrence of statistic significant differences are

important characteristics to evaluate the conclusion of the authors and to assess whether the

methods are appropriate to show differences between GMO and comparator. (CA)

Competing interests and sources of support

36) (Ref: p 17, Line 455): „Competing interests and sources of support“: To declare that no competing

interest exists seems questionable in light of the review team (page 9) that comprises one member of

the EFSA GMO Panel and one member of the FF Working Group of the GMO Panel. (industry)


The comments submitted by stakeholders were well-received and proved useful in identifying points that

needed further clarification, elaboration and adjustment in the proposed protocol on compositional studies

on GM crops versus their non-GM counterparts. A number of additional, minor editorial changes (some

54

indicated by the commenters, such as removal of “etcetera”) were made as well, without affecting the

core contents of the document.

In the response below, the comments from the different stakeholders who responded have been

summarized and grouped according to the main topics addressed by them:

1) “Title”: A comment featured a suggestion for a change in the title as the latter, if not changed, would

appear to suggest that changes are always observed.

- Response: The proposed change was persecuted.

2) Purpose: One commenter raised doubt about the usefulness of evidence maps containing onl data on

characteristics of studies.

- Response: The various comments received, particularly on this topic, highlight the concerns that

commenters have over the selection of parameters, the choice of comparator, the tissues

analyzed and the parameters chosen. This implies that these choices are important for the

validity of the study outcomes and therefore the authors consider that this also underscores the

importance of an evidence map to get an idea of how studies are performed.

3) “Background (1)”: A commenter referred to the mention of omics (which is done in the abstract of the

document, for example) while pointing out that it is not clear how these omics methods are included

and how specifically the differences identified are dealt with, particularly with an eye on the statistics

used for this purpose.

- Response: While the purpose of evidence maps is not to evaluate the outcomes of the selected

studies, the mention of omics in the protocol will be modified so as to be better highlight that it is

the endpoints about which the studies to be selected report rather than the actual methodology

used to measure the endpoint (also so-called targeted analyses can actually involve different

methods for the same endpoint such as microbiological and high-performance chromatography

for certain vitamins). A further exploration of the findings of each of the selected studies could

well fit into a systematic review or even meta-analysis expanding on the evidence map, but this

is not yet foreseen to be done within this project (the documentation with the evidence map

should enable such studies also to be performed by others). The issue is also explained in

section 2.2 (not modified, same as in previous version).

4) “Background (2)”: In the response from one commenter, it was emphatically recommended to better

clarify in the document that compositional changes do not necessarily constitute safety concerns, and

to distinguish statistically significant concerns from biologically relevant ones. Another commenter

argues that the document does not provide guidance (e.g. for thresholds) on what differences should

be considered relevant or not. From a slightly different angle, an additional comment queried whether

the study would look into natural variability among varieties from the same crop, and focus on

changes between the GM crop and its comparator constituting risks. Vice versa, another comment

actually recommended focusing on those compositional parameters that are known to have a health

impact while asserting that for all compositional parameters in any crop the natural variability may not

be exactly known although the wide-ranging environmental conditions were considered to have a

great impact on these endpoinds.

- Response: It is not the purpose of the evidence mapping exercise itself to assess and interpret

the outcomes of the selected studies (also in terms of the magnitude of observed changes being

relevant for safety or not, such as for example by comparing these changes with background

variability). The background should nonetheless provide the appropriate context of the mapping

exercise and avoid misunderstandings on this crucial issue, for which reason the introduction in

55

the background section has been modified with an additional clarifying sentence under the

heading of “concern/problem” and under “evidence synthesis”. A subsequent systematic review

based on the outcomes of the evidence map (not foreseen for this particular project yet) could

further explore the outcomes of the studies selected through the proposed search and selection

methods. The comment on compositional endpoints to be included into the mapping exercise

and related comments are further discussed in the section on methods.

5) “Background (3)”: It was suggested by one commenter to also evaluate the suitability and applicability

of the analytical methods for risk assessment of GMOs.

- Response: Answering such a question appears to better fit into a systematic review question

that also considers the outcomes of the studies (the proposed evidence map aim to provide a

comprehensive overview of the methods used but will not assess the methods per se). In

addition, a similar question on the robustness of the analytical methods used for comparative

compositional analysis of GM crops was actually one of the four initially proposed questions on

composition, yet this was not included in the final selection of questions based on stakeholder

feedback and prioritization.

6) Methodology for performing the evidence mapping (1): With regard to the information sources to be

used, various comments pointed to the fact that dossier information referred to may not be accessible

and that permission from the applicant be needed to publish the findings of the exploration of these

sources. Others pointed to the fact that the grey literature sources were not well-defined, while some

commented that information on crop composition could also be obtained from databases not

mentioned (e.g. ILSI Crop Composition Database) in contrast with some of the sources mentioned

that appeared to be less relevant for this particular topic.

- Response: It is only the intention to use summaries and original reports of dossier data that

have been made publicly available through trustworthy sources (e.g. government agency’s

websites) and not to use and publish confidential data, this also with a view on the need for

reproducibility of the searches (in the abstract, the terms “publicly available” have been added).

Grey literature sources will be subject to the same selection criteria as data from other sources

and the sources of grey literature are defined in section 3.2.6 of the protocol. The Internet links

provided in 3.2.6 in most cases link to websites with information on regulatory safety

assessments of GM crops including summaries of the findings of the comparative compositional

analytical studies performed on particular GM crops. The authors are aware of the fact that

some authorities do publish relatively extensive opinions and also may provide access to several

dossier reports (e.g. Australia-New Zealand, EFSA, India). The ILSI Crop Composition database

is extensive and quality-controlled yet the data contained by this database can (purposefully) not

be linked to particular varieties (although the list of varieties is given). Moreover, the ILSI

database do not provide study descriptions of comparative compositional experiments from

which the data on non-GM commercial varieties included in the same field might have been

derived.

7) Methodology for performing the evidence mapping (2): various comments were received as regards

the parameters to be reported and the focus of the analysis, for example by focusing only on the

measurements of the edible parts of the crops (i.e. the parts consumed by humans and animals).

- Response: A decision often has to be made whether such features are something to be mapped

or whether they should be used as a selection criterion. For some crops, it is obvious which the

edible parts are while for others, some particular parts may still be edible although not usually

used in isolation for that purpose (green tissues of maize versus forage/silage used as feed).

Although experience teaches that in most cases, it is such edible parts that are analyzed, It is

56

proposed to keep this as a feature to be mapped rather than to list the various tissues to be

included or excluded from the exercise. Composition of non-edible tissues may still have

relevance for environmental safety (feed source for non-target organisms), as a secondary

consideration.

8) Methodology for performing the evidence mapping (3): Some commenters advocated a clear

definition of “key chemical components” while one of them recommended limiting the search to those

parameters that known health impacts.

- Response: Similar as for the previous question, this could be used as a selection criterion (in

line with the commenters’ recommendations) or as an item to be mapped. It is realized that the

OECD Task Force for the Safety of Novel Foods and Feeds has developed consensus

documents on key compositional parameters for certain crops but not for others (the latter would

then have to be defined by the researchers themselves). In the factors to be mapped (table 3 in

section 3.4 “mapping strategy”), it is requested to provide details on whether the reported

compositional parameters are in line with OECD consensus documents, if such documents exist

for the given crop species.

9) Methodology for performing the evidence mapping (4): One commenters suggested to include

additional details of the statistics applied to the compositional analysis.

- Answer: The request for details on statistical treatment is open-ended. The purpose of evidence

mapping is not to provide the actual outcomes of such treatments, but details on the

methodology and the way in which the outcomes are presented. Some of these were already

covered in Appendix 8 with further details on sample data extraction, and several more details on

statistics have been included in that sheet.

10) Methodology for performing the evidence mapping (5): Various comments pointed to the difficulty of

obtaining a near-isogenic counterpart for the GM crop.

- Response: As for several other considerations, the nature of the comparator for the GM crop

could be used as an selection/exclusion criterion for the studies to be mapped, yet it is felt that

information on the comparator used in the various studies would actually yield insight into what

kinds of comparators are used in the published studies and whether a description is provided

about the genetic identicalness. Of course these considerations are important in exploring the

validity of any potential findings of differences between the GM and its comparators, yet such

questions lend themselves for a systematic review. It is noted that in the data extraction sheet,

details are requested on the description of the comparator used for the GM crop.

11) Methodology for performing the evidence mapping (6): Several comments concerned the conclusions

or effects observed (e.g. environmental quality) and the scientific quality of the article:

- Response: Evidence mapping does not foresee an evaluation of the findings of the different

studies; the exploration of findings in this case is limited to the presentation of data rather than

the contents; a systematic review extending on the findings of the map could further go into this

but is not foreseen to be done at the moment. Objective verifiable and reproducible criteria are

used for inclusion of studies.

12) “Conflicts of interest”: There was a comment similar to that received for parallel evidence map

protocols, namely that membership of some of the researchers of publicly funded research projects

and risk assessment bodies would constitute an interest.

- Response: As for the parallel protocols, this was been mentioned based on the consideration

that this would not be a conflicting interest. Under the heading where authors’ interests are

57

declared, there is now an additional statement on some researchers’ membership of research

project teams and risk assessment institutions.

58

3) Evidence Map “What are the characteristics of repeated-dose feeding

studies in which experimental animals receiving whole food or feed

products derived from genetically modified (GM) crops are compared to

animals receiving conventional non-GM counterparts?”



literature) is not well described and there is a lack of clarity on what will be included and excluded.

(industry)

2) No explicit criteria listed on how quality of each reviewed paper will be assessed. (industry)

3) In order to use data provided by applicants in dossiers, there should be agreement from the applicant.

(industry)


(industry)

5) (Ref: p 2, Line 58): ‘The aim of the presented project is the elaboration of systematized document

containing information on the characteristics of whole-GM food feeding trials and its subsequent

analysis‘.(industry)

6) (Ref: p 7, Line 249): ‘The objective of the present/proposed evidence map is to combine all possible

relevant studies from peer-reviewed, non-peer-reviewed and grey literature sources through a

transparent and unbiased search and extraction and systematization of the data on the

characteristics, particularly the design and the methodology used, of studies on the health of animals

fed with GM-plant-containing diets and with an appropriate control‘.(industry)

7) (Ref: p 7, Line 249): ‘The The objective of the evidence map is to map out the characteristics of the

studies conducted with whole GM products upon animals that would be a source of information for

risk assessors and other involved stakeholders about specific items of these studies‘.

From the above paragraphs, it is evident that the main purpose or objective of the exercise is unclear

or completely undefined. Further, it is not clear how assessment of these study characteristics shall

support stakeholders. (industry)

8) The document clearly points out the complexity of whole food and feed animal trials with complex test

and control materials. However, it seems to me that the GRACE review/map is not giving enough

attention to the genetic diversity of even the GM and nearest genetic neighbor. It is the same as for

compositional studies. The comparator is often only about 97 genetically identical to the GM. The

characterization should include some remarks or evaluation of the knowledge of the genetic

differences. (Academia/Scientific community)

9) Importantly, the environmental conditions under which the crops were grown and harvesting

conditions and characterization of test materials are often overlooked by many of the published

studies, especially those performed by other non-developer scientists. (Academia/Scientific

community)

10) As for the other maps, it seems there needs to be some critical review of all studies included in the

mapping process and it is not clear to me that sufficient detail has been provided to understand how

the maps and content will be judged. (Academia/Scientific community)

59

11) The protocol contains some useful information on search strategies for information available on

animal feeding trials with whole GM food/feed, which is of interest for risk assessors, policy makers

and other interested stakeholders. It is useful to standardize the available information on performed

studies in order to judge on the quality of generated data and whether comparison of the results of

different studies is possible. (Academia/Scientific community)

12) The PICO concept may be useful in certain areas like human epidemiology, but is of no added value

in the risk assessment of GMO derived foods/feed, and is in fact never used in this field. It is more

pertinent to apply the classical risk assessment paradigm, i.e. hazard identification, hazard

characterization, exposure assessment and risk characterization, and check whether the results of

available studies are in compliance with the risk assessment strategy developed for GM derived

food/feed. (Academia/Scientific community)

13) The draft protocol unfortunately does not cover the important issue that studies performed with whole

GM food/feed always need adaptation [emphasis added by commentator] of existing

protocols/guidelines (for example OECD test guidelines for chemicals). Providing detailed information

on adaptation of existing test protocols is essential in order to enable the risk assessor to interpret the

data properly (i.e. balancing test diets, proper choice of dose levels in order to avoid nutritional

imbalances etc). In many cases the results obtained from animal feeding studies illustrate serious

deficiencies in this respect and are thus of limited value. This type of information is essential in order

to be able to compare and select studies for further risk assessment. (Academia/Scientific

community)

14) The proposed draft protocol does not provide guidance on how the results from the available studies

should be interpreted and characterized regarding their impact on human and/or animal health.

Issues like a) possible dose-related trends, b) possible interrelationships between tested endpoints, c)

occurrence of effects in both sexes, reproducibility and animal specificity, d) background ranges of

variability in test parameters (obtained from historical data of reference groups?) and e) description of

identified uncertainties in the applied model, statistical analysis and use of biological test parameters

(see also EFSA opinion of the EFSA Scientific Committee, Guidance on conducting repeated-dose

90-days oral toxicity study in rodents on whole food/feed, EFSA Journal 2011:9(12):2438). This type

of information is essential in order to be able to compare and select studies for further risk

assessment. (Academia/Scientific community)

Comments on Title

15) Title ‘What are the characteristics of repeated-dose feeding studies in which experimental animals

receiving whole food or feed products derived from genetically modified (GM) crops are compared to

animals receiving conventional non-GM counterparts?” The title is unclear on what is the purpose of

the review. The title is too broad to describe the objective of this review. (industry)


16) (Ref: p 2, Line 58): ‘…particular, it would be useful to obtain insight into how many of such studies

have been performed on which GM crops, and how these studies compare to each other with regard

to the test performed,….’It is not clear what the main objective of the systematic review is. There are

already a number of review articles that looked into this. It is not clear how different this exercise

would be from the already publicly available review articles on the subject. (e.g. Ricroch, 2013)

(industry)

60

17) (Ref: p 2, Line 67): ‘The proposed evidence map will represent a collection of relevant information

concerning the characteristics of whole-GM-product animal feeding trials‘.The relevance of the

information should be clearly indicated. (industry)

18) (Ref: p 2, Line 69): ‘The searches will be performed in several databases and search engines, and

evidence will be collected from accessible peer reviewed, non peer-reviewed and grey literature

sources‘.“Non peer-reviewed” and “grey literature” are not well described in the document. How will

the non peer-reviewed and grey literature be assessed in relation to peer-reviewed? (industry)

19) (Ref: p 2, Line 76): ‘The analysis of the collected and described evidence will be presented in a

narrative format ‘.The analysis of the retrieved information has to be better described. Is the intention

to publish the analysis in a scientific publication? (industry)


20) (Ref: p 4, Line 148): ‘…inventory of the various animal studies involving whole-food administration of

GM crops, the methodologies followed in these cases, and what the impact of the results of the study

has been on the respective risk assessment...‘.It is not clear how assessment of study characteristics

would be done. (industry)

21) (Ref: p 6, Line 200): ‘…it can be envisaged that animal feeding tests within the regulatory framework

are bound to certain protocols and standardized working procedures, this may not hold true for animal

feeding tests that are carried out for academic and other than regulatory purposes”.

Comparisons between animal feeding studies performed for regulatory purposes and other studies

carried-out by the academia or for other purposes should be careful done. (industry)


22) (Ref: p 7, Line 280): Please add in brackets: “Including food and water consumption, body weight,

organ weight and histopathology” in order to give more detailed information what typical physiological

test parameters are. (CA)

23) (Ref: p 7, Line 280): Please add “clinical observations and signs of toxicity” as test parameter. (CA)

24) (Ref: p 8, Line 288): ‘…final outcome (O, any effects)... ‘“Any” effects should be better characterized.

Will only effects statistically significant be included? (industry)

25) (Ref: p 8, Line 297): It has to be pointed out that OECD Test Guidelines have been developed for

toxicity testing of chemicals. So far no OECD Test Guidelines has been adopted and validated for

toxicity testing of whole GM food/feed. (CA)


26) (Ref: Table 1, p 9 – “Intervention”): “HT”. I recommend using the abbreviation “HR” for herbicide

resistance as search term too. (CA)

27) (Ref: Table 1, p 9 – “Outcome”): “repeated dose toxicity stud”. I suggest adding the search terms:

”long-term stud*, chronic stud*” in order to find as many as possible relevant studies. (CA)

28) (Ref: p 10, Line 382): ‘…Search engines ... Google Scholar... ‘It is not mentioned how the retrieved

information will be assessed and evaluated. (industry)

29) (Ref: p 10, Line 391): ‘Table 2 Organizational websites/ Government and agency web sites and

libraries‘. The method to retrieved information from these websites is unclear. (industry)

61

30) (Ref: p 12, Line 399): ‘Study inclusion criteria‘. No criteria listed on how quality of retrieved articles will

be assessed. There is uncertainly on what would be excluded from this exercise. (industry)

31) (Ref: Table 3, p 15 – “Compared plants and other comparators”): “Confirmation of non-GM nature?”

This should include the analytical verification, that the control is free from any GM material not only

free from test material. (CA)

32) (Ref: Table 3, p 16 – “Hypothesis formulation and conclusions”): “If this is indeed the case” Please

delete “indeed”. It implies bias. (CA)

Comments on Competing interests and sources of support

(Ref: p 17, Line 514): ‘None declared’. To declare that no competing interest exists seems questionable in

light of the review team (page 8) that comprises one member of the EFSA GMO Panel and one

member of the FF Working Group of the GMO Panel. It is interesting to note that there is no declared

competing interest in these documents. However, some of the authors are being funded by the EU or

EC to perform research on animal models, composition and other parameters. Isn’t that a completing

interest? Since I have studies funded by biotechnology companies and developers, it seems I could

not participate as an advisor to EFSA or the EC. Thus it seems a bit disingenuous to not declare the

interests of the authors, and for the EC/EU and others to not recognize that everyone has some

competing interest. A declaration would help add transparency. (Academia/Scientific community)


The stakeholders’ comments received to the draft protocols covered a range of items within the draft

protocol which had been put up for consultation, including the purpose and context described in the

document, as well as the process and details of the evidence mapping exercise itself. Various comments

for textual changes and editorial comments have been incorporated, while also additional text has been

incorporated to address various of the comments raised, e.g. regarding purpose of the mapping exercise.

Below, a summary of these comments will be provided, addressing each category of issues raised within

these comments from the various stakeholders:

1) Purpose (1): Some commenters indicated that the purpose of the exercise was not sufficiently clear,

while another casted doubt as to whether the outcomes could provide useful information and whether

this was not a duplication of already performed reviews published elsewhere. One commenter also

noted that a publication of the map in scientific literature should be pursued and mentioned.

- Response: In order to better highlight the objectives, these have been put upfront as

introductory statements in the abstract and background sections. Also the aim to publish the

map in scientific literature is mentioned now.

The evidence map is intended to obtain both an overview (“landscape”) of the evidence that

exists, and to map the characteristics of the feeding experiments performed. This evidence map

can serve various purposes, such as to provide risk managers background on the studies that

have already been carried out, to furnish insights about how feeding trials with complex

food/feed products are set up (including shared and non-shared features).

The map will have a number of unique features (avoiding duplication of previously published

research) such as the focus on details of the methods and design of the studies, as well as the

inclusion of certain grey literature resources (NB all of the data will be evaluated according to

inclusion criteria). Moreover, the methodology is to be reproducible by providing the methods

and data used.

62

Under the objectives, various examples of how the outcomes could have value (e.g. for

background when media reports or discussions on the outcomes of specific studies arise; for

harmonization of guidance on how to perform feeding studies with complex products such as

foods or feeds; as a basis for a further systematic review).

2) Purpose (2): Various commenters recommended evaluating various details of the studies (e.g.

protocols followed, statistics, feeding materials used, animal strains, feeds) to be evaluated, and/or

expressed concern about the quality of the reports/studies to be included.

- Response: While the evidence map will actually list these features of the studies retrieved and

selected, some of the commenters actually referred to a “review” to be performed. This

apparently is a misunderstanding as the purpose is to develop an evidence map and not a

systematic review, with the latter indeed to evaluate the outcomes of studies included (not the

aim of the evidence map). The proposed evidence mapping exercise is a stage prior and

amenable to systematic review in that it will map out the various characteristics of the studies

available from scientific literature and other sources through the search methods applied,

including the details that the commenters referred to. No selection of studies is to be made

based on these details yet as this is not the purpose of the evidence map.

3) Methodology for performing the evidence mapping: One commenter stated that details of the sources

and inclusion criteria for grey literature data were missing; moreover, this commenter noted that there

was a lack of evaluation criteria as well. It was also noted that a comparison between different types

of animal studies with the same GM crop could be made only carefully. Another commenter argued

that the specific nature of each specific GM crop should be taken into account. Another commenter

more generally criticized the PICO approach and recommended an alternative strategy with

successive steps in line with risk characterization.

- Response: The PICO approach is in line with review/mapping practices, allowing in this case to

disentangle the various components of the studies and to systematically search for literature

data bringing together these components in a reproducible and tangible way. The analysis of the

studies retrieved will, for example, include a description of the GM crop used, its comparator,

and further details relating to the design and execution of the study. The search strategy in

bibliographies includes terms related to the population (animals used for feeding studies on the

safety of GM crops) and the intervention (GM crops, which are used as dietary ingredients).

It is not the purpose of the mapping exercise to evaluate the outcomes of these studies and

therefore neither to reach conclusions about the lack of presence of potential risks of GM

foods/feeds. As stated above, the map could provide a basis for a further systematic review into

specific features of animal feeding studies with GM-crop-derived foods and feeds.

4) Interpretation of the evidence to be found: As stated above, some commenters referred to the

interpretation and selection of studies.

- Response: Evidence maps are intended to provide the landscape of evidence and are not

intended to assess the outcomes provided in the studies; this could be done in a next step of

systematic review (or even meta-analysis, if feasible).

5) “Conflicts of interest”: Some commenters argued that membership of some of the researchers of

publicly funded research projects and risk assessment bodies would constitute an interest.

- Response: This has not been mentioned as this was considered to be no conflicting interest.

Under the pertinent heading, an additional statement has been inserted referring to the various

researchers’ membership of research project consortia and risk assessment bodies.

63

4) Evidence Map “What are the characteristics of studies on the risk of

allergic sensitization and elicitation in humans and animals exposed to

an allergenic plant that has been genetically modified as compared to

individuals exposed to the non-genetically-modified plant counterpart?”


1) I am a bit confused about the purpose of the Evidence Map Protocol. The title indicates it is to

investigate the types (characteristics) of studies on the risk of allergic sensitization and elicitation in

humans and animals exposed to GM as compared to non-GM plant counter-parts. However, the

background seems as if you are asking for what evidence is available right now regarding the

potential differences in allergen content/activity of GM vs non-GM plants. Are you suggesting that

GRACE will look for evidence of increased allergenic activity of existing GM vs non-GM or for study

methods that might be useful to test for differences? (Academia/Scientific community)

2) Additionally it is important to note that we do not know what it is in any plant that might be important in

SENSITIZING [emphasis added by commentator] individuals, except for the presence of allergenic

proteins that have already sensitized someone, or bound IgE due to cross-reactivity.


3) And we (collectively, scientists who study allergy), do not know whether high or low concentrations of

proteins are more conducive to sensitization. Only regarding elicitation, that substantially higher

amounts of allergenic proteins are more likely to stimulate an allergic response (elicit a reaction).


4) In the introduction it would be useful to describe HOW [emphasis added by commentator] a GM event

might create a difference in the allergen content or concentration. For instance, one might speculate

that insertion of a transcription factor MIGHT [emphasis added by commentator] increase expression

of some proteins including allergens. But that is based on positional affects or functional affects. Also

if a fusion protein was created by introduction of the gene/DNA. Otherwise there is almost no way to

introduce a new allergen. What is proposed? (Academia/Scientific community)

5) The overall purpose and deliverables for the review are not clear. (industry)

6) The basis for evaluating potential allergenicity in case of GM plants (other than the point that the

allergy assessment is part of the GM pre-market safety assessment for some regulatory agencies) is

not clearly described. Bias is evident in the document and the paper seems to pre-assume that the

genetic modification is bound to cause changes in allergenicity though no direct evidence is

presented. (industry)

7) The selection of the data package (peer-reviewed, non-peer-reviewed, grey literature) is not clearly

described. Justification to include/exclude non-peer reviewed as well as grey literature is not made. If

included, a scoring matrix to rank the publication by quality of peer-reviewed vs non-peer

reviewed/grey literature should be made. An apparent risk of misinterpretation of the obtained results

due to the inclusion of non-peer reviewed literature cannot be negated. (industry)

8) There is no mention about the discredited or retracted publications which might still be cited in the

non-peer reviewed or grey literature affecting the conclusions. (industry)

9) The effects of food processing on the allergenicity of the foods should be considered. For example,

highly refined oil from GM or non-GM soybean, canola, etc. should be equally safe in terms of

64

allergenicity, since it is devoid of any proteins. Similarly, allergenicity of soy sauce (compared to raw

soy) is diminished due to fermentation. (industry)

10) This draft protocol raises a number of questions regarding its objectives, feasibility, outcome and

added value. (Academia/Scientific community)

11) The document should clearly state that the comparative evaluation of the allergen repertoire of a GM

plant versus a non-GM counterpart is a typical EFSA approach, not suggested by other international

risk assessment for a like Codex Alimentarius and OECD. (Academia/Scientific community)

12) The fundamental problem with this protocol is the lack of any guidance on how to assess potential

differences in the allergen repertoire between the GM and non-GM plant regarding their impact on

human and animal safety. (Academia/Scientific community)

13) The draft protocol does not cover other testing/evaluation strategies like in-vitro testing and sequence

homology evaluation. (Academia/Scientific community)

14) At least in Europe, I am not aware of any GM crop approved to be released on the market where its

endogenous allergenicity has been modified. However, I do not know what the picture is in other parts

of the world. In the case that examples of GM plants with a modified endogenous allergenicity

(intended or unintended) that are released on the market and with specific monitoring programs in

place [emphasis added by commentator] are not available, I am not sure that I understand the

relevance of the question of the project. (CA)

15) In Europe and according to the EFSA Guidance Document for the food/feed risk assessment of GM

plants (2011), if a GM plant is considered to be as safe and as nutritious as the non-GM comparator

then a post-market monitoring is not required. This is the case for all current GMOs approved in

Europe for which obviously no post-market monitoring [emphasis added by commentator] is required.

Again, I am not sure that any relevant publication will be found in the public literature that will meet

the expectations of the title of your question. (CA)


16) (Ref: p 3, Line 43): According to Table 3 (Ref: p 15), all plant species are considered for which

allergens are listed in the IUIS (International Union of Immunological Societies) allergen

nomenclature database. This interpretation would reach way beyond what is usually considered

“commonly allergenic”. We would strongly suggest using the University of Nebraska as a source of

peer reviewed allergenic species listings. In addition, it is worth highlighting that at the “Workshop on

key allergens and compositional analysis in the allergenicity assessment of genetically modified

plants” held in Paris in 2012, it was stated that even soybean is likely not to be defined any longer as

“commonly allergenic”. (industry)

17) (Ref: p 3, Line 56): if one only considered potential differences in the allergenic content of a GM

relative to the nearest non-GM relative, the measure of a difference is meaningless. It needs to take

into account differences across varieties of the crop used in commerce today. In the case of soybean,

there may be 200 varieties grown in the US in a given year, in 10 or so different maturity groups. We

do not have an estimate of the natural variation of allergen expression and certainly not across

environmental conditions that may cause significant differences in allergen expression. Furthermore,

various allergenic proteins may be more variable in expression than others. Yet we do NOT have an

evidence of the level of difference in expression or abundance of allergens in soybean that are

biologically important in elicitation, let alone sensitization to soybean. (Academia/Scientific

community)

65

18) (Ref: p 3, Line 56): We suggest considering also natural variability with respect to allergen abundance

and allergen isoform distribution in the host plant. (industry)


19) (Ref: p 5, Line 123): This proposal seems to be biased towards a specific outcome. The way it is

written presupposes that increases in allergens are an existing risk per se, which has not been

demonstrated so far. (industry)

20) (Ref: p 5, Line 129 ff): Have you found any evidence that we understand the quantitative difference in

expression or abundance of any allergenic protein that might make abiologically significant difference

in allergic reactions to a food (e.g. foods derivedfrom soybean). I am not aware of any conclusive

data that has demonstrated a single case of an allergic reaction that occurred which would not have

occurred if the amount of soybean protein was half as much or one-fifth as much. Or a reaction that

occurred due to an increase of two fold or five fold. The closest evidence is through thresholdstudies

that use five-fold increases in allergen dose between challenge doses. (Academia/Scientific

community)

21) (Ref: p 6, Line 144-157): Page 6 already discusses the types (characteristics) of studies used to

evaluate potential differences in allergen expression in endogenous allergens. It could have cited

studies from my laboratory. We have used individual sera with immunoblots and then pooled sera for

ELISAs and ELISA inhibition (as would be done to validate the equivalence of allergen in

pharmaceutical grade allergenic extracts (as per the US and EU Pharmacopeaiea), to judge

diagnostic or therapeutic extracts as equivalent. That level in pharmaceutical regulation is 2 fold,

being equivalent. Individual serum IgE differences are generally more readily compared to indicate

qualitative differences (see Goodman et al. J. Agric Food Chem 2013, 61(35):8317-8332, Evaluation

of endogenous allergens for the safety evaluation of genetically engineered food crops: review of

potential risks, test methods, examples and relevance). (Academia/Scientific community)

22) (Ref: p 6, Line 150): When describing the methods to measure endogenous allergens, we suggest

not only to include IgE 2D immunoblots and mass spectrometry. There are further well established

methods that can be used to measure allergen expression levels (e.g. gel-based comparisons, DIGE,

individual protein ELISAs, etc.). (industry)

23) (Ref: p 6, Line 170): While it is often the case in a new area of study that evidence maps might be

useful in understanding what information is already available and what avenues of research might be

useful, it is hard to understand how the proposed evidence map is likely to uncover additional relevant

published information that allows a better understanding of risks or test methods that might be useful.

It is essential to acknowledge that we do not know have good estimates of the frequency of

sensitization and elicitation for even the most allergenic of foods. That should lead to a question of

how many subjects would be needed to do a relevant comparison between one GM crop and another

non-GM but nearest related variety of the same crop. (Academia/Scientific community)

24) (Ref: p 7, Line 180): The use of grey literature is questionable, and there are no criteria for adequate

selection of such type of documents. The document appears to set up a literature review that

considers all literature, even web-based articles that have had no peer review or any review for that

matter, as having the same value, weight, and quality regardless of how it was performed or by

whom. (industry)

66


25) (Ref: p 7, Line 189): Regarding the “control population”, it should be considered that GM crops may

enter the food supply globally through international trade in food/feed commodities, as stated also on

page 5 of the Draft protocol. Accordingly, it might be difficult to define a population that is purely

exposed to non-GM plants. This point becomes vital specifically in case of publications describing

epidemiological studies with populations consuming GM vs non-GM foods. (industry)

26) (Ref: p 7, Line 190): How realistic is it to map out the characteristics of the studies performed on

changes in the levels of endogenous allergens in GM plants and the associated changes in

allergenicity of these plants to humans? (Academia/Scientific community)

27) (Ref: p 8, Line 215): How realistic is it given the extensive available literature to expect any changes

in the frequency of sensitization and elicitation and the severity of elicitation reactions, caused by

exposure to the GM plant and derived products in humans (and animals that could serve as model)

as compared to reactions in subjects exposed to a non-GM control plant and derived products?


28) (Ref: p 8, Line 220): This section seems like a very naive set of questions. The information asked for

in thebullets is not known for any of the major (common) allergens. For instance, while there are a

few studies claiming that Ara h 2 and/or Ara h 6 of peanut are the most potent allergenic proteins for

food allergy, there are conflicting reports from different populations and scientists/clinicians that Ara h

1 or Ara h 3 are as important or more so. Due to the low prevalence of allergy to any given food, and

the heterogeneity of responses, it is not clear that we have a definitive answer for the most allergenic

of food such as peanut, cow’s milk, or other tree nuts or grains. Furthermore, we do not have good

scientific data on the differences in expression/accumulation differences in these “major allergens”

within multiple samples of the crop, such as peanut. One could/should know those “facts” without

doing an evidence map or an expensive study. It simply takes a few days of searching with PubMed

and reading/reviewing papers related to a few of the most commonly allergenic foods such as peanut,

soybean, tree nuts. (Academia/Scientific community)


29) (Ref: Table 1, p 9): If the purpose of Grace is to show unequivocally that there is little or no data to

support the statement that food allergies to soybean (or any other food) have increased due to the

introduction of GM soybeans, then it might be useful to use the multitude of search terms begun on

page 9. But that should not be necessary. The list of suggested search terms seems really over-the-

top. One should be able to simply search for soybean AND allergy (=1011 references), adding

“genetic brings that to 37, adding “sensitization brings it to 3 (that I am very familiar with). None of

them demonstrate any quantitative difference across any soybean varieties. Similar results (with

fewer returns) are found with maize instead of soybean. (Academia/Scientific community)

30) (Ref: p 14, Line 348): There is no clear rationale of why animals should be considered as being an at

risk population. In our opinion, there is no place for this in the safety assessment per ‘endogenous

allergenicity’ assessment. In addition, the text might be interpreted as if the entire population is at risk

for potential harm by changes in endogenous allergens. That is not the case, since by definition the

only population at risk of e.g. soybean allergy, are soybean patients. It must be emphasized that

within the human population not everyone is allergic to any particular crop. (industry)

31) (Ref: p 14, Line 357): We would like to reemphasize that in our view a non-GM comparator is not the

most important control when assessing endogenous allergens. Due to the considerable variation of

endogenous allergens between different plant varieties and between plants grown under different

67

conditions, inclusion of an adequate number of controls would ensure that any variation observed can

be put into perspective regarding any potential change in allergens observed in a GM plant. (industry)

32) (Ref: p 17, Line 402): “Assessment of heterogeneity” seems to be highly speculative. I would be very

surprised if you can find clearly definitive information on any food crop in terms of being able to

measure and benchmark levels of allergen expression differences with increases in risks for already

allergic subjects. I will be even more surprised if you can find evidence of increased risks of

sensitization tied to specific increases in the abundance of proteins in a quantitative sense. There

have been a number of studies (including our own) that have attempted to evaluate potential

differences in expression of endogenous allergens in GM vs Non- GM varieties either by IgE binding

(immunoblot, ELISA, RAST, EAST, or inhibition assays), but as far as I am aware none of these have

been able to demonstrate a biological difference associated with differences in expression of any

endogenous allergen due to transformation or selection. There are clearly no identifiable

epidemiological studies that have identified a change in sensitization to a food crop (or organism)

associated with a genetic variant of the crop or species. (Academia/Scientific community)


The comments received straddled various features of the proposed protocol, with regard to both the

background information and rationale provided in the document, and the evidence mapping approach per

se.

In summary, the comments received could be categorized as follows and answers are duly provided:

1) Purpose: General comments related to the purpose and the focus of the proposed evidence mapping

exercise, i.e. whether it was on changed allergen repertoires in GM plants or methods to study

increased allergenicity.

- Response: The purpose of the evidence map is both to get an impression of the evidence that

has already been gathered on the specific issue, and to provide details of the characteristics of

these studies, such as the kind of publication, the population (human, animal) tested, the

methods used to measure changes in plant allergen repertoire and in the allergic response to the

plant, the allergenic plant to which the population has been exposed, etcetera. The studies to be

selected should report on changes in allergic response to plants in which the allergen repertoire

has been changed following genetic modification.The question is formulated so as to be able to

distinguish population, intervention, control, and outcome.

2) Regulatory requirement: Various stakeholders pointed to the fact that the comparison of allergen

repertoires in GM crops versus conventional counterparts is something typically requested by EFSA

according to its guidance (recently enshrined into regulation), yet this is not commonplace yet outside

Europe.

- Response: In line with the comment, text has been added to the background text to explain this

3) Avoidance of bias towards outcomes: Reference to changes, in particular increases, in the content of

specific endogenous allergenic proteins, was considered to reflect bias towards thinking that GM per

se has an impact on allergen repertoires. It was also noted that while the comparison GM versus

direct counterpart might reveal changes, these should still be interpreted in the light of background

variability.

- Response: Reference to increases was initially made so as to exclude hypoallergenic items

from the original, more broadly defined question. It is realized that this may be perceived as too

focused on one of the possible outcomes. The wording in the pertinent sentence in the

68

introduction has been changed. Moreover, the search strategy description refers to “changes”

(more neutral). Interpretation of outcomes is not part of the evidence mapping (unlike systematic

reviews). In Table 3, factors to be considered during evidence mapping, additional lines have

been included so as to ensure that the inclusion of additional comparators (e.g. for background

variability) is described.

4) Methodology for measuring the allergen repertoire: Reference was made to the lack of guidance on

how to assess changes in endogenous allergenicity. In addition, it was criticized that certain

methodologies such as in-vitro assays and bioinformatics were not covered by the protocol. Another

commenter devised to exclude certain methods for allergen measurement, while a second

commenter referred to additional methods besides the ones already mentioned.

- Response: While the EFSA guidance provides instructions on how to measure the allergen

repertoire, the potential changes in allergic response in humans and animals as might be due to

differences in these repertoires may become evident indeed in various ways. The search terms

used in the search strategy cover a range of conditions related to clinical reactions as well as

clinical-biochemical responses (e.g. nausea, oedema, erythema versus Th2 cell response).

In-vitro assays and bioinformatics are primarily linked with the allergenicity assessment of newly

expressed proteins, not with that of the whole food/feed plant (the latter being the focus of the

evidence mapping question). It is conceivable that under certain circumstances, specific

components of the allergen repertoire have been identified with the aid of bioinformatics or that

ex-vivo allergic responses have been measured in-vitro (e.g. with isolated basophil cells). If

such studies are found, they can indeed provide useful insights.

Methods for using allergen repertoire: It is actually one of the tasks for the evidence mappers to

verify which methods have been used for determining the difference in allergen repertoires, and

as such this will provide useful information.

5) Plant species: A comment pertained to the plant species included: These included allergens listed by

WHO-IUIS. It was recommended instead to resort to another peer-reviewed source.

- Response: It is realized for many of these allergenic plant species, no GM variants may yet exist

that are to enter the market or that have already been brought to the market. It was therefore

decided to replace this list of plant species with those that are known to have been genetically

modified and to have been introduced into the market place (e.g. based on the list in the CERA-

GMC database).

6) Feasibility of studying changes in allergic responses to GM crops: Doubts were raised as to whether

data on changes in allergic responses to GM crops might be available, particularly in humans given

the lack of monitoring programs for GM food allergenicity. Moreover, the difference needed in

expression levels of endogenous allergenic proteins for eliciting a different response was conceived

to be high and an explanation for what might cause such changes in the GM plant was recommended

to be included. Also the difficulty of finding humans not exposed to GM foods was highlighted.

Moreover, it was noted that highly refined foods from GM crops, such as oils, might be devoid of any

allergenic proteins. The term “at risk” used for both humans and animal raise a comment about the

reason why animals could be considered risk populations for allergies.

- Response: It is exactly the purpose of the evidence map to find out if there is any such report

describing changes in allergic response and what the characteristics are of these studies. A

short survey of the first 100 hits in the searches of CAB Abstract and Web of Science shows that

various studies have recently been undertaken to measure the immunological impact of certain

GM crops in laboratory rodents and livestock animals, plus several reviews on the topic. We are

aware of the difficulty with defining the exposure of human consumers to GM foods, yet we’d like

69

to refer to the fact that the occurrence of GM ingredients in foods is rare in the EU, that other

routes of exposure could also be of interest (e.g. respiratory allergies), and that there are

initiatives towards monitoring for allergies related to GM plants (E.g. French initiative for

respiratory allergies). As previously stated, the purpose is to explore what evidence is available,

yet we are aware of the fact that the reasons provided by the stakeholders may account for the

possible rare or no occurrence of such studies (this would also provide us with an answer of the

available evidence). The quoted reference to derived products also pertains to, for example,

protein-rich crop products used in food and feed. The term “at risk” was replaced so as to avoid

the misperception that also animals were considered to be a risk population.

7) Interpretation of evidence and data sources: Various comments related to the interpretation of the

evidence to be mapped, including whether or not indeed changes in response to the challenge with

GM plants have been found to occur by investigators or not, or whether the inclusion of grey sources

of literature may lead to other conclusions based on un-refereed, low-quality data.

- Response: the evidence map will not evaluate the outcomes of the studies (unlike a true

systematic review). It is inherent to this kind of mapping to also take into account grey literature

sources, while the specific categories to be looked for (such as risk assessment summaries) are

defined by the protocol. Criteria are applied for inclusion of studies into the evidence map, while

also descriptors are included, such as the publication type and whether publications pertain to

materials also published elsewhere.

70

Chapter 3: Socioeconomic Impacts of GMOs

General Comments

This section refers to comments that are not linked to a specific protocol but that could be of relevance for

most or all of them.

1) There are generalities in the preamble that include the statement that the science around GM is

controversial (surely the overwhelming body of scientific evidence points to the opposite). This is

certainly the case in Europe, where GM crops are not widely grown. But in other parts of the world

GM crops have become an integral part of mainstream commercial agriculture, much less shrouded

in debate and controversy. The context of the European market is not then necessarily the most

representative starting point for this exercise. (industry)

- Response: It is indeed correct that there is more controversy on the cultivation of GM crops in

Europe as compared to, for example Latin America, North America, China and India. However,

in several other parts of the world, such as for example several African countries, there is also

controversy on the cultivation of GM crops. In addition, in the major grain producing countries,

such as Russia, Ukraine and Kazakhstan, GM crops are not commercially cultivated and there is

substantial discussion on their use. In fact, in Russia, a recent survey has shown that almost

80% of the Russian citizens support the bill that is currently under preparation and that would

heavily restrict the import of genetically modified agricultural produce, as well as stop it

altogether from being produced domestically.

2) There seems to be no real methodology for capturing either the direct or indirect benefits of GM which

is at odds with farmer adoption rates around the world and various studies showing positive

environmental benefits. (industry)

- Response: This statement is rather vague and we do not completely understand what is meant

by this the direct and indirect effects (effects on what?). As already indicated by TUM, studies

have used a wide range of methodologies to quantify the impact of GM production on certain

socio-economic factors, such as income, well-being, health, etc.

3) Potential agronomic factors such as pollen flow and coexistence are framed as a specific issue to GM

rather than Good Agricultural Practices (including much more serious pollen flows where known

toxins are dealt with on farm using planting times and prevailing winds such as diesel OSR). The

same positioning is erroneously used for potential impact to non-GM growers who choose to go

beyond labelling thresholds (and should cover cost of voluntary schemes). (industry)

- Response: The comment is unclear. Pollen flow from GM crops could cause socio-economic

impacts on non-GM farms. Good Agricultural Practices and known toxins are beyond the scope

of our socio-economic work.

4) Other socio-economic questions are being asked (such as gender issues) but don’t have any

explanation / basis for a GM impact assessment. Preliminary research would need to point to a link

before going into these questions. (industry)

- Response: Specific link(s) will be added in the conceptual model(s).

5) There are some more assertions about trade not being impacted by a de facto moratorium, without

any proper argumentation. We would suggest the addition of consequences of non-GM in terms of

pesticide use/exposure etc. (industry)

71

- Response: It is unclear which assertions are meant as we do expect that trade is likely to be

affected by a de facto moratorium on GM products. In fact, the de facto moratorium of the EU

was exactly the reason behind the trade dispute (WTO case) between the US, Canada and

Argentina on the one hand and the EU on the other hand. This dispute lead in 2006 to the

condemnation of the EU and several authors (e.g. Disdier and Fontagné, 2010) find indeed that

the de facto moratorium had a negative effect on trade.

The suggestion of adding the consequences of non-GM in terms of pesticide use/exposure is an

interesting suggestion and the economic effect of pesticide use is already being dealt with in the

SR that deals with the welfare effects.

6) There are generalities in the preamble that include the statement that the science around GM is

controversial (surely the overwhelming body of scientific evidence points to the opposite). This is

certainly the case in Europe, where GM crops are not widely grown. But in other parts of the world

GM crops have become an integral part of mainstream commercial agriculture, much less shrouded

in debate and controversy. The context of the European market is not then necessarily the most

representative starting point for this exercise. (industry)

- Response: Unfortunately this statement is wrong. GM crops are controversial in different parts

of the world not only in Europe.

7) There seems to be no real methodology for capturing either the direct or indirect benefits of GM which

is at odds with farmer adoption rates around the world and various studies showing positive

environmental benefits. (industry)

- Response: Unfortunately this statement is wrong. There are scientific methodologies for

evaluating the socio-economic impacts of GM crops (e.g. Heckman models, propensity score

matching, etc.).

8) The same positioning is erroneously used for potential impact to non-GM growers who choose to go

beyond labelling thresholds (and should cover cost of voluntary schemes). (industry)

- Response: The comment is unclear. Pollen flow from GM crops could cause socio-economic

impacts on non-GM farms (due to e.g. minimum/maximum thresholds from non-GM voluntary

certification schemes).

72

1) What is the impact of the introduction of GM crops on the welfare effects in

different countries in comparison to a situation where there are restrictions

on GM cultivation?

1) Basically any process of academic research or systematic review will always be behind the curve of

rapidly changing market and supply chain dynamics - no unique assumptions or models can be

attributed to GM crop supply chains. (industry)

- Response: The aim of a systematic review is to provide an exhaustive overview and identify the

findings of existing studies on the welfare effects of GM cultivation at macro level. While an

important part of a systematic review is to assess the quality and accuracy of the findings and

methods of all identified studies, it is out of the scope of a systematic review to judge and draw

conclusions on whether a methodological approach is a valid instrument per se to examine GM

crop supply chains. In addition, we take the rapid changing market and supply chain dynamics in

account by including also the very recent studies, which are – in some cases – not (yet)

published in academic journals if they fulfil our quality criteria.

2) The review document also seems to suggest that farmers derive greater benefit from new technology

in those countries where intellectual property protection is weaker – actually it is the existence of

effective frameworks of Intellectual Property (IP) protection which provides the commercial incentives

for upfront research investment to take place, supporting access to innovation. (industry)

- Response: We agree with the important role of Intellectual Property Rights (IPR) in providing

firms with incentives to invest in innovation. In the review, we simply present the information on

the benefits for the different partners in the supply chain and the reasons behind these

differences (e.g. more benefits for farmers in case of poor IPR). However, we do not make any

value judgment stating that “more benefits for the farmers are better” or “this is a fair way of

distributing the benefits”. This is not the aim of the review. A review should simply give an

overview of the evidence without making any value judgment.

73

2) What is the impact on GM regulation of different political actors and other

drivers in the EU in comparison to the US?

1) Global uptake data are outdated. (industry)

- Response: We have revised the protocols and updated the global uptake data.

2) Preoccupied with the controversy surrounding GM crops, and suggesting that the primary concern

and motivation of GM opponents is that only chemical companies will benefit from the technology.

(industry)

- Response: As indicated before, the aim of a review is to provide an objective overview of the

evidence that there exists on a specific topic. In this overview we will make no value judgments.

We simply provide an overview of the different drivers that are found to affect GM regulation in

the EU as compared to the US.

3) The justification for producing an evidence map of the drivers behind the discrepancies between EU

and US regulation of GMOs appears to be that no one else has done it yet. No reference to the value

or utility of producing such an evidence map. (industry)

- Response: We have revised the text accordingly and now point out the value and utility of

producing such an evidence map. The main value that this evidence map has that it provides an

objective and unbiased overview of the scientific literature that is available on the different

drivers of GM regulation in the EU as compared to the US. While there is less a direct link with

impact assessment as for example for the macro level effects or the trade effects, this

information is still useful for policy makers as it provides them with a background on the different

drivers that have affected GM regulation in the past and which are likely to play a role in the

future.

4) The suggestion that EU farmers and chemical companies may be lobbying for more stringent GM

regulation in Europe as one of the key drivers is obviously incorrect. (industry)

- Response: The aim of a review is to give an objective and unbiased overview the existing

evidence on a specific research question. This implies that if there exist studies that find

evidence of lobbying by EU farmers and/or chemical companies and these studies fulfill the

quality criteria, these studies will be included in the review.

5) The scene-setting for this evidence map is not focused on the political economy, but rather on

recycling the tired and outdated social science literature on this issue (eg a number of academic

publications are cited, one from 1999, two from 2001, two from 2003, four from 2004, one each from

2005 and 2007). Not a single reference is cited post-2008, the year when concerns over global

population growth, food security and commodity price volatility became a significant component of the

political debate and agenda surrounding GM crops and the application of new technology in

agriculture more generally. (industry)

- Response: Ok, we have revised the protocol accordingly and included more recent references.

6) It is not good enough to simply focus on historical points of difference dating back to the late 1990s

and early 2000s rather than to track much more recent trends in consumer attitudes which suggest a

significantly positive shift in favour of yield boosting technologies which can help keep food price

inflation and food security concerns in check. (industry)

74

- Response: We include studies from 1994 to now. This implies that if there is a shift in consumer

attitudes and this shift has been documented in the recent literature, then this information will be

included in the review.

75

Revisions, peer-review and final versions of the Draft Systematic

Review Protocols

The draft versions of the review protocols discussed in the previous sections were revised considering the

comments provided by stakeholders and submitted to the scientific journal ‘Environmental Evidence’.

The manuscripts were subjected to peer-review and the protocols were revised again based on the

reviewer’s comments. By the time of the publication of this report at the GRACE website (September

2014) a number of protocols were already accepted and published as open access papers.

The only exception to this procedure applies to the three protocols on macro-level socio-economic

impacts of GMOs which were not published.4

In the following sections references are provided to the published review protocols.

Environmental Impacts of GMOs

Published:

Meissle M, Naranjo SE, Kohl C, Riedel J and Romeis J: Does the growing of Bt maize change abundance

or ecological function of non-target animals compared to the growing of non-GM maize? A systematic

review protocol. Environmental Evidence 2014, 3:7.

http://www.environmentalevidencejournal.org/content/3/1/7.

Sweet JB and Kostov K: What are the effects of the cultivation of GM herbicide tolerant crops on

botanical diversity ? A systematic review protocol. Environmental Evidence 2014, 3:8.


Kostov K, Frølund Damgaard C, Bohse Hendriksen N, Sweet JB and Krogh PH. Are population

abundances and biomasses of soil invertebrates changed by Bt crops compared with conventional crops?

A systematic review protocol. Environmental Evidence 2014, 3:10.


Kostov K, Krogh PH, Frølund Damgaard C, Sweet JB and Bohse Hendriksen N: Are soil microbial

endpoints changed by Bt crops compared with conventional crops? A systematic review protocol.

Environmental Evidence 2014, 3: 11. http://www.environmentalevidencejournal.org/content/3/1/11.

Gathmann A and Priesnitz KU. How susceptible are different lepidopteran/coleopteran maize pests to Bt-

proteins: a systematic review protocol. Environmental Evidence 2014, 3:12.


4 For budgetary reasons as the journal required page fees.

http://www.environmentalevidencejournal.org/content/3/1/7


http://www.environmentalevidencejournal.org/content/3/1/10/abstract






76

Gathmann A and Priesnitz KU: What is the evidence on the inheritance of resistance alleles in

populations of lepidopteran/coleopteran maize pest species: a systematic map protocol Environmental

Evidence 2014, 3:13.


Health Impacts of GMOs

Publications in preparation:

Stoykova P, Kostov K, Vlahova M, Atanassov A, Kleter* G, Noordam M, Adamse P, Kok E, Kohl C,

Pares D, Pla M, Nadal A, Wal JM, Patient K, Paris A, Craig W. Evidence map protocol: What are the

characteristics of toxicity studies in which animals receiving newly expressed proteins from GM crops are

compared to animals that are administered appropriate controls without these proteins?

Stoykova P, Kostov K, Vlahova M, Atanassov A, Pares D, Pla M, Nadal A, Kleter* G, Noordam M,

Adamse P, Kok E, Kohl C, Wal JM, Patient K, Paris A, Craig W. Evidence map protocol: What are the

characteristics of comparative studies of assessment of key chemical crop constituents in GM crops

compared to non-GM crops?

Stoykova P, Kostov K, Vlahova M, Atanassov A, Wal JM, Patient K, Paris A, Kleter* G, Noordam M, van

der Spiegel M, Adamse P, Kok E, Kohl C, Pares D, Pla M, Nadal A, Craig W. Evidence map protocol:

What are the characteristics of studies on the risk of allergic sensitization and elicitation in humans and

animals exposed to an allergenic plant that has been genetically modified as compared to individuals

exposed to its non-GM counterpart?

Stoykova P, Kostov K, Vlahova M, Atanassov A, Kleter* G, Noordam M, van der Spiegel M, Adamse P,

Kok E, Kohl C, Pares D, Pla M, Nadal A, Wal JM, Patient K, Paris A, Craig W. Evidence map protocol:

What are the characteristics of repeated-dose feeding studies in which experimental animals receiving

whole food or feed products derived from genetically modified (GM) crops are compared to animals

receiving conventional non-GM counterparts?

Socioeconomic Impacts of GMOs

Van Herck K and Garrone M. Systematic review protocol: What is the impact of the introduction of GM crops on the welfare effects in different countries in comparison to a situation where there are restrictions on GM cultivation?

Van Herck K and Garrone M. Systematic review protocol: What is the impact on GM regulation of different political actors and other drivers in the EU in comparison to the situation in the US?

Van Herck K and Garrone M. Systematic review protocol: What is the impact of trade restrictions of GM products in different countries on the competitiveness of different partner countries and corresponding sectors in comparison to a situation where there are no restrictions on GM trade?





77

Accepted for publication:

Garcia-Yi J, Lapikanonth T, Vionita H, Vu H, Yang S, Zhong Y, Li Y, Nagelschneider V, Schlindwein B,

and Wesseler J. "What are the socio-economic impacts of genetically modified crops worldwide? A

systematic map protocol”. Environmental Evidence.

78

Annex

Stakeholder comments received

German Federal Agency for Nature Conservation (Bundesamt für Naturschutz – BfN), Germany (CA).

Scientific Institute of Public Health, Belgium (Academia/Scientific community).

EuropaBio, Belgium (Industry).

European Food Safety Authority (EFSA), Italy (CA).

University of Nebraska, USA (Academia/Scientific community).

Individual scientific expert, The Nederlands (Academia/Scientific community).

grace stakeholder consultation on good review practice in ... good review... · grace stakeholder...

Documents