HORIZON 2020 Proposal Evaluation

Briefing for Experts Societal challenge 2

First stage of two-stage calls 2016

HORIZON 2020

2

Content

Horizon 2020: a new type of EU R&I programme

Overview of the Societal Challenge 2

Role of independent experts

The evaluation procedure in practice

A new type of EU R&I programme

HORIZON 2020

4

New types of calls and proposals

• Calls are challenge-based, and therefore more open to innovative proposals

− Calls are less prescriptive - they do not outline the expected solutions to the problem, nor the approach to be taken to solve it

− Calls/topics descriptions allow plenty of scope for applicants to propose innovative solutions of their own choice

• There is a greater emphasis on impact, in particular through each call or topic impact statements

− Applicants are asked to explain how their work will contribute to bringing about the described impacts

− During the evaluation, you are asked to assess this potential contribution

• There is more emphasis on innovation

− Horizon 2020 supports all stages in the research and innovation chain including non-technological and social innovation and activities closer to the market

• Proposals may bring together different disciplines, sectors and actors to tackle specific challenges

− e.g. scientists, industry, SMEs, societal partners, end-users…

HORIZON 2020

5

More emphasis on innovation

• A balanced approach to research and innovation

− not only limited to the development of new products and services on the basis of scientific and technological breakthroughs

− but also incorporating aspects such as the use of existing technologies in novel applications and continuous improvements

• Activities closer to the market emphasise the widest possible use of knowledge generated by the supported activities up to the commercial exploitation of that knowledge

• There is a particular emphasis on activities operating close to the end-users and the market, such as demonstration, piloting or proof-of-concept

− can also include support to social innovation, and support to demand side approaches (standardisation, innovation procurement, user-centred measures …) to help accelerate the deployment and diffusion of innovative products and services into the market

The definitions of the terms used are available in the Horizon 2020 Glossary on the Participant Portal

HORIZON 2020

6

Cross-cutting features

Those features are fully integrated in the work programme (WP)

• Social Sciences and Humanities (SSH) are integrated across all Horizon 2020 activities to successfully address European challenges

• Gender dimension in the content of R&I - a question on the relevance of sex/gender analysis is included in proposal templates

• The new strategic approach to international cooperation consists of a general opening of the WP and targeted activities across all relevant Horizon 2020 parts

− The approach to providing 'automatic funding' to third country participants is restricted – see list of countries

• Other cross-cutting features may also be included in the WP such as Responsible Research and Innovation (RRI) including science education, open access to scientific publications, ethics…; standardisation; climate and sustainable development …

Guidance on evaluation of some of those aspects is available on the Participant Portal

HORIZON 2020

7

Impact of grant preparation on evaluation

• No grant negotiation phase!

The time from submission of a proposal, evaluation and signature of the grant has been reduced to a maximum of 8 months

• What does this mean for the evaluation of proposal?

− You evaluate each proposal as submitted not on its potential if certain changes were to be made

− If you identify shortcomings (other than minor ones), you must reflect those in a lower score for the relevant criterion

− You explain the shortcomings, but do not make recommendations

− Significant shortcomings (preventing the project from achieving its objectives or seriously over-estimated resources) must lead to a below-threshold score for the relevant criterion

− Any proposal with scores above threshold and for which there is sufficient budget will be selected as submitted

Overview of the Societal Challenge 2

1. Health, demographic change and wellbeing

2. Food security, sustainable agriculture and forestry, marine and maritime and inland water research, and the bioeconomy

3. Secure, clean and efficient energy

4. Smart, green and integrated transport

5. Climate action, resource efficiency and raw materials

6. Inclusive, innovative and reflective societies

7. Secure societies

Seven Societal Challenges

Sustainable Food Security

(SFS)

€ 180,5 million (33 expected projects)

• More resilient and resource efficient value chains • Environment-smart and climate-smart primary

production • A competitive food industry • Healthy and safe foods and diets for all • EU-Africa Partnership and EU China Flagship

Initiative

Blue Growth (BG)

• Boosting Innovation from emerging BG activities • Linking healthy oceans with healthy people • The Arctic Dimension • Valorising the Mediterranean Sea Basin

€ 82 million (10 expected projects)

• Securing sustainable biomass supply for bio-based goods and services

• Building bio-based markets of the future- mobilising stakeholder engagement

€ 12 million (3 expected projects)

Bio-based Innovation

(BB)

Rural Renaissance

(RUR)

• New approaches towards policies and governance • New value chains and business models • Innovation and skill development

€ 66 million (14 expected projects)

4 calls in 2016 (single and two stages)

Publi-

cation

Submission

procedure Topics Stage Deadline

Info to

appli-

cants

Grant

Agree-

ments

14 Oct.

2016

single

stage

19 topics (8 CSA, 4 IA,

3 ERANET, 3 RIA)

144 M€

Single

stage

17 Feb.

2016

June

2016

Oct.

2016

2-stages

25 topics (RIA Only)

197 M€

1st stage 17 Feb.

2016

May

2016 -

2nd stage 13 Sept.

2016

Nov.

2016

May

2017

Schedule 2016 Evaluation schedule

Role of Independent Experts

HORIZON 2020

13

Evaluation Process: Who is who

Evaluators

• Evaluate proposals

• Submit Individual Evaluation Reports (IERs) remotely

• Participate in remote consensus discussions to agree on scores and comments

Vice-chairs

• Do not evaluate proposals

• Quality check of Individual and Consensus Reports

• Support moderators in monitoring evaluation progress and deadlines

Observers (Vasconcelos, Calzarossa)

• Observe the evaluation process

• Provide advice / improvement suggestions to the REA

Contact persons:

• Primarily: your REA topic leader

• Call coordinator: Tiziana della Ragione

HORIZON 2020

14

Role of independent experts

• As an independent expert, you evaluate proposals submitted in response to a given call

• You are responsible for carrying out the evaluation of the proposals yourself

− You are not allowed to delegate the work to another person!

• You must close reports in the electronic system within a given deadline

− This is part of your contractual obligations!

− The allowance/expenses you claim may be reduced or rejected otherwise

• Significant funding decisions will be made on the basis of your assessment

HORIZON 2020

15

Guiding principles

• Independence

− You are evaluating in a personal capacity − You represent neither your employer, nor your country!

• Impartiality

− You must treat all proposals equally and evaluate them impartially on their merits, irrespective of their origin or the identity of the applicants

• Objectivity

− You evaluate each proposal as submitted; meaning on its own merit, not its potential if certain changes were to be made

• Accuracy

− You make your judgment against the official evaluation criteria and the call or topic the proposal addresses, and nothing else

• Consistency

− You apply the same standard of judgment to all proposals

HORIZON 2020

16

Confidentiality

You must:

• Not discuss evaluation matters, such as the content of proposals, the evaluation results or the opinions of fellow experts, with anyone, including:

− Other experts or Commission/Agencies staff or any other person (e.g. colleagues, students…) not directly involved in the evaluation of the proposal

− The sole exception: your fellow experts who are evaluating the same proposal in a consensus group or Panel review

• Not contact partners in the consortium, sub-contractors or any third parties

• Not disclose the names of your fellow experts

− The Commission publishes the names of the experts annually - as a group, no link can be made between an expert and a proposal

• Maintain the confidentiality of documents, paper or electronic, at

all times and wherever you do your evaluation work (on-site or remotely)

− Please take nothing away from the evaluation building (be it paper or electronic) − Return, destroy or delete all confidential documents, paper or electronic, upon

completing your work, as instructed

HORIZON 2020

17

Conflicts of interest (COI) (1)

You have a COI if you:

• were involved in the preparation of the proposal

• stand to benefit directly/indirectly if the proposal is successful

• have a close family/personal relationship with any person representing an applicant legal entity

• are a director/trustee/partner of an applicant or involved in the management of an applicant's organisation

• are employed or contracted by an applicant or a named subcontractor

• are a member of an Advisory Group or Programme Committee in an area related to the call in question

• are a National Contact Point or are directly working for the Enterprise Europe Network

HORIZON 2020

18

Conflicts of interest (COI) (2)

• In the following situations, the Commission/Agency will decide whether a COI exists

− You were employed by an applicant or sub-contractor in the last 3 years

− You were involved in a grant agreement/decision, the membership of management structures or a research collaboration with an applicant in the last 3 years

− You are in any other situation that casts doubt on your impartiality or that could reasonably appear to do so

COI conditions are spelled out in your contract, and in the Code of Conduct (Annex 1)

HORIZON 2020

19

Conflicts of interest (COI) (3)

• You must inform the Commission/Agency as soon as you become aware of a COI

− Before the signature of the contract

− Upon receipt of proposals, or

− During the course of your work

• If there is a COI for a certain proposal you cannot evaluate it

− Neither individually

− Nor in the consensus group

− Nor in the panel review

− The Commission/Agency will determine if there is a COI on a case-by-case basis and decide the course of action to follow

• If you knowingly hide a COI, you will be excluded from the evaluation and your work declared null and void

− The allowance/expenses you claimed may be reduced, rejected or recovered

− Your contract may be terminated

The evaluation procedure in practice

HORIZON 2020

21

Overview of the evaluation process

Receipt of proposals

Individual evaluation

Consensus report

Evaluation Summary

report

Evaluators

Individual Evaluation Reports (remote)

Consensus Report

Is submitted by rapporteur and approved by the other

experts

Evaluation Summary Reports

are finalised

Above threshold proposals list is

prepared

Eligibility check

Allocation of proposals to evaluators

REA REA

Finalisation

ESRs are sent to applicants.

Above threshold proposals are invited to the second stage

Fully remote

Vice-chairpersons

HORIZON 2020

22

Admissibility and eligibility checks

• Admissibility is checked by the Commission/Agency:

− Readable, accessible and printable

− Completeness of proposal presence of all requested forms

− Inclusion of a plan for exploitation and dissemination of results (unless otherwise specified in the WP)

• Eligibility checked by the Commission/Agency - however, if you spot an issue relating to eligibility, please inform the Commission/Agency

− Minimum number of partners as set out in the call conditions

− Other criteria may apply on a call-by-call basis as set out in the call conditions

• “Out of scope” – you need to check if the content of a proposal corresponds, wholly or in part, to the description of the call or topic

− A proposal will only be deemed ineligible in clear-cut cases

Page limits: Clearly set out in electronic system; excess page(s) marked

with a watermark

HORIZON 2020

23

Admissibility and eligibility checks

• Admissibility is checked by the Commission/Agency:

− Readable, accessible and printable

− Completeness of proposal presence of all requested forms

− Inclusion of a plan for exploitation and dissemination of results (unless otherwise specified in the WP)

• Eligibility checked by the Commission/Agency - however, if you spot an issue relating to eligibility, please inform the Commission/Agency

− Minimum number of partners as set out in the call conditions

− Other criteria may apply on a call-by-call basis as set out in the call conditions

• “Out of scope” – you need to check if the content of a proposal corresponds, wholly or in part, to the description of the call or topic

− A proposal will only be deemed ineligible in clear-cut cases

Page limits: Clearly set out in electronic system; excess page(s) marked

with a watermark

HORIZON 2020

24

Types of actions in SC2 two-stage calls

Research and Innovation Actions (RIA) only:

SFS BG BB

BB-1

Type of action

Research and Innovation

Action (RIA)

RUR

SFS-1 SFS-2 SFS-3 SFS-6 SFS-7 SFS-11 SFS-14 SFS-15 SFS-21 SFS-23 SFS-26 SFS-31 SFS-33 SFS-37 SFS-38 SFS-42 SFS-44 SFS-45

BG-12

RUR-1 RUR-4 RUR-6 RUR-7

RUR-14

Type of action Funding Aim of action Evaluation

Research and Innovation Action

(RIA) 100%

To establish new knowledge or explore the feasibility of a new technology, product, process,

service or solution

TWO stages

(with a few exceptions)

Innovation Actions (IA)

70%

(100% for non-profit entities)

To produce plans and arrangements or designs for

new, altered or improved products, processes or services

SINGLE Stage Coordination and Support Actions

(CSA) 100% Accompanying measures

ERA-NET Cofund up to 33%

To support public-public partnerships, including joint

programming initiatives between Member States

Type of actions (1)

HORIZON 2020

26

Type of actions (2)

Research and Innovation Action (RIA)

• Action primarily consisting of activities aiming to establish new knowledge and/or to explore the feasibility of a new or improved technology, product, process, service or solution

− For this purpose they may include basic and applied research, technology development and integration, testing and validation on a small-scale prototype in a laboratory or simulated environment

− Projects may contain closely connected but limited demonstration or pilot activities aiming to show technical feasibility in a near to operational environment

Evaluation criteria (1st stage, RIA) • Two evaluation criteria:

Excellence Impact

• Clarity and pertinence of the objectives;

• Soundness of the concept and credibility of the proposed methodology;

• Extent that the proposed work is beyond the state of the art, and demonstrates innovation potential (e.g. ground-breaking objectives, novel concepts and approaches, new products, services or business and organisational models)

• Appropriate consideration of interdisciplinary approaches and, where relevant, use of stakeholder knowledge.

• Extent to which the outputs of the project would contribute to each of the expected impacts mentioned in the work programme under the relevant topic;

HORIZON 2020

28

Multi-actor approach Applies to topics: SFS-01, SFS-02, SFS-06, SFS-07, SFS-11, SFS-15, SFS-26, SFS-37, SFS-42, RUR-01, RUR-04, RUR-06, RUR-07, RUR-14 Aims at demand-driven innovation through the genuine and sufficient involvement of various actors (end-users such as farmers/farmers' groups, fishers/fisher's groups, advisors, enterprises, etc.)…

… all along the project: from the participation in the planning of work and experiments, their execution up until the dissemination of results and a possible demonstration phase.

→ A dedicated workshop was organised by DG AGRI to present the

multi-actor approach: https://scic.ec.europa.eu/streaming/workshop-on-multi-actor-approach-and-thematic-networks-under-horizon-2020

→ Multi-actor briefing slides are available on the briefing page

HORIZON 2020

29

Proposal scoring

• You give a score of between 0 and 5 to each criterion based on your comments

− Half-marks can be used − The whole range of scores should be used − Scores must pass thresholds if a proposal is to be considered for

funding

• Dynamic threshold introduced in first stage evaluations

Criteria Threshold

Excellence 4

Impact 4

Implementation n/a

TOTAL Dynamic

total requested budget of proposals admitted to stage 2 is as close as

possible to three times the available budget

HORIZON 2020

30

Interpretation of the scores

The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information.

Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses.

Fair. The proposal broadly addresses the criterion, but there are significant weaknesses.

Good. The proposal addresses the criterion well, but a number of shortcomings are present.

Very Good. The proposal addresses the criterion very well, but a small number of shortcomings are present.

Excellent. The proposal successfully addresses all relevant aspects of the criterion. Any shortcomings are minor.

0

1

2

3

4

5

Half

sco

res g

iven

HORIZON 2020

31

First stage evaluation phase – Remote ONLY

Individual Evaluation

Report

Individual Evaluation

Report

Individual Evaluation

Report

Rapporteur

prepares Draft

Consensus Report

Approved Consensus

Report

Individual Evaluation

Report

Expert Expert Expert Expert Expert

Minimum 3 experts

Individual evaluation IER to be finalised by 31 March at the latest

Draft Consensus Reports to be finalised by 8 April at the latest Remote consensus via SEP / e-mails

Proposal Eligible proposal

Individual Evaluation

Report

Quality check by Vice-Chairs

Quality check by Vice-Chairs Finalised Consensus Reports by 15 April at the latest

HORIZON 2020

32

Evaluation Planning

• Call closure 17/02/2016

• Remote IER phase 11 – 31/03

• Evaluators accept tasks by 17/03

• First 2 draft IERs ready by 20/03 → VC check quality of IERs

• Remote consensus 01/04 – 15/04

• Draft CR by rapporteur by 08/04

• CRs finalised 15/04/2016

→ VC check quality of CR

HORIZON 2020

33

Vice Chairs (VC)

Vice Chairs are experts who assist REA officers in order to ensure that the evaluation exercise runs in a smooth way.

VCs do not evaluate proposals, and cannot give their opinion on any proposals.

Their contribution relates to support regarding the evaluation management

HORIZON 2020

34

Vice Chairs (VC)

VC will contact experts at the start of the evaluation and will be in contact through the entire remote evaluation exercise. VCs ensure a quality and timely delivery of evaluators' reports – Individual Evaluation Reports (IERs) and Consensus Reports (CRs)

VCs provide guidance and feedback to evaluators, under the control and overall supervision of REA topic leaders. Please be receptive to their comments and guidance.

VCs are in constant touch with REA staff to raise any relevant issue or report on any necessary event.

HORIZON 2020

35

Individual evaluation

• You read the proposal and evaluate it against the evaluation criteria

− Without discussing it with anybody else

− As submitted - not on its potential if certain changes were to be made

− Do not penalise applicants that did not provide detailed breakdown costs – they are not required

• You disregard excess pages marked with a watermark

• You check to what degree the proposal is relevant to the call or topic

• You complete an Individual Evaluation Report (IER)

− Give comments and scores for all evaluation criteria (scores must match comments)

− Explain shortcomings, but do not make recommendations

• You then sign and submit the form in the electronic system

Look at the substance: Some proposals might be handicapped by language

difficulties, others deceptively well written

HORIZON 2020

36

If a proposal • Is only marginally relevant in terms of its scientific, technological

or innovation content relating to the call or topic addressed, you must reflect this in a lower score for the Excellence criterion

− No matter how excellent the science!

• Does not significantly contribute to the expected impacts as specified in the WP for that call or topic, you must reflect this in a lower score for the Impact criterion

• If cross-cutting issues are explicitly mentioned in the scope of the call or topic, and not properly addressed (or their non-relevance justified), you must reflect this in a lower score for the relevant criterion

− A successful proposal is expected to address them, or convincingly explain why not relevant in a particular case

− Proposals addressing cross-cutting issues which are not explicitly mentioned in the scope of the call or topic can also be evaluated positively

HORIZON 2020

37

Individual Evaluation Report (IER)

• Under each sub-criterion, all elements must be commented

• it is recommended to organise comments under headers 'strengths' or 'weaknesses'

• Evaluators will complete first 2 IERs by 20th March

• Do not submit it – but inform your vice chair and your topic leader by e-mail

• Vice chair will provide feedback on quality / consistency

• Please submit all your IERs by 31th March

HORIZON 2020

38

Remote drafting of CR by Rapporteur

Rapporteur's task

• The rapporteur is one of the evaluators.

• Open the Write CR tasks and save the draft 'initialised' CR so all evaluators can see each others' comments

• Put a comment in the task comment box indicating that compiled IERs are visible

• Prepare draft CRs:

• 'Initialise' the draft CR in SEP

• Identify points of convergence and propose a consensus wording

• Identify divergences as points for discussion or clarification with other experts (via SEP's task comments box or email), highlighting strengths and weaknesses of proposals to guide remote consensus and potential scores

• Put a comment in SEP's task comments box to invite other evaluators to check the draft CR before submitting it for vice-chair's reviewing

• Draft CR to be completed by 08/04

HORIZON 2020

39

Consensus - Remote only

The rapporteur is responsible for drafting the consensus report (CR)

− The draft CR is based on the comments and scores of all the evaluators as expressed in the Individual Evaluation Reports.

• The aim is to find agreement on comments and scores

Use the electronic system for evaluation (comment box) to discuss when needed.

− Agree comments before scores!

• “Outlying” opinions need to be explored

− They might be as valid as others – be open-minded

• The draft CR should be commented by all experts after it is reviewed by the vice-chair (Review CR task)

• To finalize the CR, all experts have to approve it in the system.

• The EC will perform a final quality check of the CR and may request editorial changes to the rapporteur.

HORIZON 2020

40

Consensus report (CR) (1)

• The rapporteur is responsible for drafting the CR

− Including consensus comments and scores

• The quality of the CR is paramount

• The aim of the CR is to give:

− A clear assessment of the proposal based on its merit, with justification

− Clear feedback on the proposal’s weaknesses and strengths

• Avoid:

− Comments not related to the criterion in question

− Comments that are too short or too long or use inappropriate language you should explain what you mean in an adequate length and clear manner

− Categorical statements that have not been properly verified e.g. “The

proposal doesn’t mention user requirements” – when there is a short reference…

Applicants can challenge those through evaluation review procedures

− Scores that don’t match the comments

− Making recommendations

− Marking down a proposal for the same critical aspect under two different criteria

HORIZON 2020

41

Consensus report (CR) (2)

Tips for rapporteurs:

• If you see great discrepancies between evaluators' scores:

first read the IERs, it is very possible that:

• Comments were 'softer' or 'harder' than the scores awarded and therefore the evaluators are not so far apart in their comments (remember first the comments and then the scores)

• A major failing/strength might have been identified by one evaluator which was not noticed by another and it is perfectly normal that the final scores for a CR are very different from an individual's IER

• Evaluators might really appreciate a proposal and score it high even if it only broadly addresses the call topic while another evaluator might also appreciate it but because it doesn't fully address the call topic has scored it low. It's important that the proposals are evaluated in the context of what was requested in the call text, even if an evaluator thinks it is an excellent proposal.

HORIZON 2020

42

Consensus report (CR) (3)

Tips for rapporteurs:

• Depending on the possibilities above and the clarifications you receive, try to draft a compromise report that balances both views and then see how the evaluators feel about it

• Usually several versions of a CR will be required before it is fully approved

• Work on the comments first without considering the scores

• Once you are satisfied with the CR, add a score for each criterion which reflects the comments (it should not necessarily be an average of the 3 individual scores)

→ If a consensus cannot be reached, then your vice-chair will find a

solution with REA moderator (e.g. 2 more evaluators can be added, minority/majority viewpoint)

HORIZON 2020

43

Observers

• Experts: Luis Vasconcelos and Maria Calzarossa

• Appointed by the Commission/Agency may attend any meetings or monitor remote evaluation, to ensure a high quality evaluation

• Check the functioning and running of the overall process

• Advise, in a report, on the conduct and fairness of the evaluation sessions and, if necessary, suggests possible improvements

• Do not evaluate proposals and, therefore, does not express any opinion on their quality

• May raise any questions - please give them your full support

HORIZON 2020

44

Logistics

• The electronic system (SEP) for the evaluation of proposals is available and accessible via your ECAS password

− Please make sure you know your ECAS login and password

You can access through the Participant Portal:

https://ec.europa.eu/research/participants/evaluation/

→ Check the briefing on SEP workflow for further details

HORIZON 2020

45

Feedback

• You will receive an on-line questionnaire on your experience in this evaluation session

• It is important that you complete it as carefully and as promptly as possible

• Your feedback helps us maintain and improve the quality of our evaluation process

• Thank you!

HORIZON 2020 Proposal Evaluation

Briefing for Experts Societal challenge 2

First stage of two-stage calls 2016