mÓdulo ii: el proceso de participaciÓn, preparaciÓn y

MÓDULO II: EL PROCESO DE PARTICIPACIÓN, PREPARACIÓN Y

EVALUACIÓN DE PROPUESTAS

La evaluación de las propuestas

Daniel Garcia-Almiñana Subdirector de Innovación y Calidad, ESEIAAT

UPC - BarcelonaTECH

Parador de la Granja (Segovia) 20 Abril 2016

XXIII SEMINARIO DE ASESORES DE PROYECTOS DE I+D+I DE LA UNIÓN EUROPEA

Horizonte 2020: El Programa Marco de Investigación e Innovación (H2020)

● 29 years experience in Project Management:

▪ Since 1986 – Energy related projects.

▪ Since 1987 – International projects.

▪ Since 1997 – Energy efficiency projects.

• 13 years at the University:

▪ Project and Energy Management courses.

▪ Research - Energy efficiency and energy auditing.

▪ Deputy Director Academics Innovation 2006/2011.

▪ Director of Telstar Chair for Innovation 2011/2015.

▪ Deputy Director Innovation & Quality since 2016.

● 10 years as EU expert:

▪ Expert evaluator since 2006. ▪ Expert reviewer/monitor since 2010.

● And since 2013:

▪ INNO-SMART Barcelona Co-founder.

Personal background

HORIZON 2020 PROPOSAL EVALUATION


The Evaluation Procedure

5

Overview of the Evaluation Process

Receipt of proposals

Individual evaluation

Consensus group

Panel Review

Finalisation

Evaluators

Individual Evaluation Reports

(remote)

Consensus Report

Panel report

Evaluation Summary Report

Panel ranked list

Eligibility check

Allocation of proposals to evaluators

Final ranked list

6

Individual Evaluation

Report


Report Individual Evaluation

Report

Consensus group

Consensus Report


Report


Report

Expert Expert Expert Expert Expert Minimum 3 experts

Individual evaluation

Consensus

Proposal Eligible proposal

Overview of the Evaluation Process


The Evaluator Role

9

Evaluation Criteria

Important issues for evaluators (I):

Independence (you are evaluating in a personal capacity, you represent neither your employer, nor your country).

Impartiality (you must treat all proposals equally and evaluate them impartially on their merits, irrespective of their origin or the identity of the applicants).

Objectivity (you evaluate each proposal as submitted; meaning on its own merit, not its potential if certain changes were to be made).

10

Evaluation Criteria

Important issues for evaluators (II):

Accuracy (you make your judgment against the official evaluation criteria and the call or topic the proposal addresses, and nothing else).

Consistency (you apply the same standard of judgment to all proposals).

Confidentiality (not discuss evaluation matters with anyone, not contact partners in the consortium, sub-contractors or any third parties, not disclose names of your fellow experts, maintain confidentiality of documents).

11

Proposal Scoring

Important issues for evaluators (III):

A score of between 0 and 5 in given to each criterion in steps of 0.5

Scores must pass thresholds for proposals to be considered for funding:

Thresholds for individual criteria; default threshold depends on the Call.

Thresholds apply to the total score; default value depends on the Call.

For Innovation actions, the criterion Impact is given a greater weight, and for Research actions, the criterion Excellence is given a greater weight.

12

The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information.

Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses.

Fair. The proposal broadly addresses the criterion, but there are significant weaknesses.

Good. The proposal addresses the criterion well, but a number of shortcomings are present.

Very Good. The proposal addresses the criterion very well, but a small number of shortcomings are present.

Excellent. The proposal successfully addresses all relevant aspects of the criterion. Any shortcomings are minor.

0

1

2

3

4

5

Proposal Scoring


The Excellence criterion

14

Evaluation Criteria - Excellence

• Clarity and pertinence of the objectives • Soundness of the concept, and credibility of the proposed methodology • Extent that proposed work is beyond the state of the art, and demonstrates innovation

potential (e.g. ground-breaking objectives, novel concepts and approaches, new products, services or business and organisational models)

• Appropriate consideration of interdisciplinary approaches and , where relevant, use of stakeholder knowledge.

Excell

en

ce

15


Evaluators job:

Check to what degree the proposal is relevant to the call or topic:

a) Compare and assess the description of the proposal against the SCOPE OF THE CALL.

b) Objectives of the proposal in line with the OBJECTIVES OF THE CALL

c) Description of the State of the art (SOA) and BEYOND SOA.

d) Check TRL of involved technologies and expected TRL after a successful end of the project.

16

Technology Readiness Level

TRL (general annex – G)

TRL 1 – basic principles observed

TRL 2 – technology concept formulated

TRL 3 – experimental proof of concept

TRL 4 – technology validated in lab

TRL 5 – technology validated in relevant environment

TRL 6 – technology demonstrated in relevant environment

TRL 7 – system prototype demonstration in operational environment

TRL 8 – system complete and qualified

TRL 9 – actual system proven in operational environment

19


Evaluators job:

Check to what degree the proposal is relevant to the call or topic.

If a proposal is only marginally relevant in terms of its scientific, technological or innovation content relating to the call or topic addressed, you must reflect this in a lower score for the Excellence criterion


The Impact criterion

21

Evaluation Criteria - Impact

• The expected impacts listed in the work programme under the relevant topic • Any substantial impacts not mentioned in the WP, that would enhance innovation capacity;

create new market opportunities, strengthen competitiveness and growth of companies, address issues related to climate change or the environment, or bring other important benefits for society

• Quality of proposed measures to exploit and disseminate project results (including IPR, manage data research where relevant);communicate the project activities to different target audiences

Im

pact

22


Evaluators job:

Check to what degree the proposal is relevant to the call or topic:

a) Compare and assess the description of the impacts against the EXPECTED IMPACTS OF THE CALL.

b) Description of the COMMUNICATION PLAN, considering the different target audiences.

c) Description of the Property Rights Management - IPR PLAN.

d) Description of a draft plan for the exploitation and dissemination of results (BUSINESS CASE – BUSINESS PLAN).

23

All grant proposals must include a draft plan for the exploitation and dissemination of the results, unless otherwise specified in the call conditions.

• There is no pre-defined structure. • It should be appropriate to the scale and scope of the envisaged project. • It has to fit within the page limit. • It is not a separate document. • Business planning elements can also be located in different sections of the

proposal.

The business case should demonstrate the expected impact of the

proposal in terms of enhanced market opportunities and manufacturing capacities for European enterprises, and thus growth and jobs in Europe, in the short to medium term.

The exploitation strategy should be realistic and identify obstacles, requirements and necessary actions involved in reaching higher TRLs.

For TRLs 6-7, a credible strategy to achieve future full-scale manufacturing in Europe is expected,

In the case of demonstrators and pilot lines, the planned use and expected impact from using the final installation should be considered.

Exploitation & Dissemination

24


Evaluators job:


If a proposal does not significantly contribute to the expected impacts as specified in the WP for that call or topic, you must reflect this in a lower score for the Impact criterion


The Implementation criterion

26

Evaluation Criteria - Implementation

• Quality and effectiveness of the work plan, including extent to which resources assigned in work packages are in line with objectives/deliverables

• Appropriateness of management structures and procedures, including risk and innovation management

• Complementarity of the participants which the consortium as a whole brings together expertise

• Appropriateness of allocation of tasks, ensuring that al participants have a valid role and adequate resources in the project to fulfill that role

Im

ple

men

tati

on

27


Evaluators job:

Check to what degree the proposal is relevant to the call or topic (I):

a) Check appropriateness of the Project Management structure, including pertinence of an External Advisory Board. Clear role sharing, WPL’s and TL’s.

b) Description of the internal Communications Management Plan.

c) Presence of a qualitative & quantitative Risk Management Plan.

d) Description of the Quality Management Plan.

e) Budget sharing amongst the “core” partners.

28


Evaluators job:

Check to what degree the proposal is relevant to the call or topic (II):

f) Role and budget sharing of the SME’s.

g) Operational capabilities of each individual partner.

h) Presence of “make-up” partners.

i) Justification of Third country partners (relevant core technologies).

j) Justification of subcontracting.

k) Justification of “other direct costs” rather than manpower.

l) Travel & Management expenses.

29

• As part of the Individual Evaluation, give your view on whether each applicant has the necessary basic operational capacity to carry out their proposed activity(ies) based on the information provided.

− Curriculum Vitae or description of the profile of the applicant

− Relevant publications or achievements

− Relevant previous projects or activities

− Description of any significant infrastructure or any major items of technical equipment

• At the consensus group, you consider whether an applicant lacks basic operational capacity .

• If yes, you make comments and score the proposal without taking into account this applicant and its associated activity(ies).

Operational capacity

30


Evaluators job:


If a proposal would require substantial modifications in terms of implementation (i.e. change of partners, additional work packages, significant budget or resources cut…), you must reflect this in a lower score for the “Quality and efficiency of the implementation” criterion


Summary

32

Evaluation Criteria - Summary

New important issues:

Disregard excess pages marked with a watermark.

Weight and threshold of each criterion depends on the Call.

Presence & role of SME’s, with R&D capabilities.

Operational capacity of individual partners.

Business case or business model.

Gender and Ethics.

expected scope (Call) & TRL

expected impact (Call)

THANK YOU FOR YOUR ATTENTION! Daniel Garcia-Almiñana (Deputy Director for Innovation & Quality)

UPC – BarcelonaTECH – ESEIAAT

https://www.linkedin.com/in/daniel-garcia-almiñana-5367a93 [email protected]

Acknowledgements: CDTI

mÓdulo ii: el proceso de participaciÓn, preparaciÓn y

Documents