MARIE CURIE ACTIONSLA EVALUACIÓN DESDE EL PUNTO DE VISTA DE UNA EVALUADORA

ROSA M. BAÑOSUNIVERSITAT DE VALENCIA

MCA: Horizon 2020 vs. FP7

IAPP

IRSES

IEF

IOF

IIF

CIG

COFUND

Research networks supporting doctoral training

Action 1 Doctoral

ResearchersITN

Action 2 Experienced Researchers

Support for experienced researchers undertaking mobility

Action 3 Exchange

of Staff

International and inter-sector cooperation through the exchange of research and innovation staff

Action 4 COFUND

Co-funding of regional, national and international programmes covering actions 1, 2 and 3

Marie Curie fellowships: evaluation process

Candidates submit proposals to specific panel CHE, ECO, ENG, ENV, LIF, MAT, PHY, SOC

EVALUATION

EL PROCESO DE EVALUACIÓN

EL REVISOR (EXPERTO)

EXPERTOS DE LA COMISIÓN EUROPEAConvocatoria permanentemente abiertaImportante inscribirse por:

- Formación sobre criterios y proceso de evaluación- Saber cómo funciona el sistema interiormente- Actualización del estado del arte- “Feed-back” para futuros solicitantes- Remuneración

REQUISITOS- Experiencia investigadora- Título universitario (Dr. y no Dr.)- Disponible en períodos cortos de tiempo- Nivel aceptable de inglés

EVALUACIÓN- A distancia (3 días)- Presencial en Bruselas (5 días)

http://ec.europa.eu/research/participants/portal/desktop/en/experts/index.html

INSERISCI COME REGISTRARSI

http://ec.europa.eu/research/participants/portal/page/experts

6

HORIZON 2020

7

Guiding principles• Independence

− The experts are evaluating in a personal capacity− The experts represent neither their employer, nor their country!

• Impartiality− The experts must treat all proposals equally and evaluate them

impartially on their merits, irrespective of their origin or the identity of the applicants

• Objectivity− The experts evaluate each proposal as submitted; meaning on its

own merit, not its potential if certain changes were to be made

• Accuracy − The experts make their judgment against the official evaluation

criteria and the call or topic the proposal addresses, and nothing else

• Consistency− The experts apply the same standard of judgment to all proposals

Conflicts of interest (COI) (2)

• The experts must inform the Commission/Agency as soonas they become aware of a COI— Before the signature of the contract, upon receipt of proposals, or during

the course of their work

• If there is a COI for a proposal they cannot evaluate it

• If the experts knowingly hide a COI, they will be excludedfrom the evaluation and their work declared null and void— The allowance/expenses they claimed may be reduced, rejected or recovered— Their contract may be terminated

The Commission/Agency will determine if there is a COI on a case-by-case basis and decide the course of action to follow

8

Marie Curie fellowships: evaluation process

Candidates submit proposals to specific panel CHE, ECO, ENG, ENV, LIF, MAT, PHY, SOC

Evaluators to declare any conflict of interest and

select proposals (at least 100) to evaluate based on

the abstract

"1" denotes that the proposal is exactly in the expert's field of expertise;"2" denotes that the proposal is in the expert's broader field of expertise;"3" denotes that the proposal is not exactly in the expert's field of expertise s/he

could evaluate it if necessary;"No" "No expertise" (by default this option is selected in all proposals).

Marie Curie fellowships: evaluation process

Depending on range of expertise, evaluators are selected and assigned up to 20 proposals to

evaluate in a relatively short period (3-4 weeks)

• Each proposal has 3 evaluators;• Evaluators will not all be expert in the specific field;• They do not have contact with each others;• They are not allowed to contact the applicant/s.• Proposals are evaluated strictly in relation to the evaluation

criteria;• If proposal longer than allowed word count, anything over

the word count will NOT be read or evaluated.

REMOTE EVALUATION

IAR

Criterios de Evaluación: umbrales y pesos

• Recordar que se tiene queconseguir buena puntuación encada criterio

• Dividir el esfuerzo sobre todos loscriterios de evaluación. Nomenospreciar ninguna parte de lapropuesta

Criterios de Evaluación: umbrales y pesos

HORIZON 2020

12

Proposal scoring

• The experts give a score of between 0 and 5 to each criterion based on their comments− Half-marks can be used− The whole range of scores should be used− Scores must pass thresholds if a proposal is to be considered for funding

• Thresholds apply to individual criteria…The default threshold is 3 (unless specified otherwise in the WP)

• …and to the total scoreThe default overall threshold is 10 (unless specified otherwise in the WP)

REMOTE EVALUATION: scoring

HORIZON 2020

13

Interpretation of the scores

The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information.

Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses.

Fair. The proposal broadly addresses the criterion, but there are significant weaknesses.

Good. The proposal addresses the criterion well, but a number of shortcomings are present.

Very Good. The proposal addresses the criterion very well, but a small number of shortcomings are present.

Excellent. The proposal successfully addresses all relevant aspects of the criterion. Any shortcomings are minor.

0

1

2

3

4

5

HORIZON 2020

14

Marie Curie fellowships: evaluation process

Rapporteur to write CONSENSUS REPORT Draft CR

The rapporteur is one of the 3 evaluators of a proposal. He/she has the responsibility to:• Collate a draft CR taking into account all of the 3

evaluators’ comments.• Lead the consensus meeting (normally in

Brussels).• Write up the final CR and get it approved by

other evaluators and proof-read by his/her vice-chair before final submission.

Marie Curie fellowships: evaluation process

Remotely or in Brussels, consensus meeting among 3 evaluators takes place.

• The 3 evaluators discuss the contents of the 3 IARs and draft CR;• Comments in agreement are normally included in CR;• Conflicting evaluations are discussed in more detail until (if)

agreement among evaluators is reached;• Marks for each criterion are then decided on the basis of the

evaluations.

CONSENSUS MEETING

2nd draft of CR

Marie Curie fellowships: evaluation process

• 2nd draft of CR is checked and approved by all evaluators;• Rapporteur’s vice-chair proof reads, checks for

inconsistencies and match of comments to mark;• If other evaluators do not approve the report, this process

is reiterated until an agreement is reached on the final CR.

If agreement evaluators is reached…

Final CR submitted to EC

Marie Curie fellowships: evaluation process

• Second (third/fourth!) meeting may be scheduled to allow more time for discussion;

• Fourth evaluator may be selected, he/she will draft an independent IAR and join the following consensus meeting/s;

• A moderator may be brought in (vice-chair; chair; project officer.

• A consensus must be reached!

If agreement among evaluators is not reached…

Several iterations of CR until consensus is reached

Marie Curie fellowships: evaluation process

• Those with an overall mark below threshold are out.• Those with any mark below the threshold for specific

criteria are out.• Funding is allocated from highest scoring proposal

going down, depending on funds (this changes from one year to the next).

All proposals in panel are ranked

Not a rule of thumb but any proposal below 85% is unlikely to get funded!

HORIZON 2020

20

Observer(s)

• Appointed by the Commission/Agency may attend any meetings, to ensure a high quality evaluation

• They check the functioning and running of the overall process

• They advise, in their report, on the conduct and fairness of the evaluation sessions and, if necessary, suggest possible improvements

• They do not evaluate proposals and, therefore, do not express any opinion on their quality

• They may raise any questions - please give them your full support

HORIZON 2020

21

Feedback

• When the experts get home, they will receive an on-line questionnaire on their experience in this evaluation session

• It is important that they complete it as carefully and as promptly as possible

• Their feedback helps us maintain and improve the quality of our evaluation process

EU Marie Curie proposals

1. Good science;

2. Well explained and at the right level (evaluators may not be overly familiar with the science but they are scientists and they will have very limited time to read your proposal);

3. Do your homework: check criteria/requirements/etc. and stick to them;

4. Make sure you address every single evaluation criterion;

5. You must ‘prove’ everything! It is not enough that the host institution is ‘well-known’ for…how can you tell?

6. Outreach ‘points’ are easy to get if you make a little effort.

Tips for success:

• Evaluation scores awarded for each criteria from 0 to 5• Each award criterion has a weighting• Total score is subject to a threshold of 70%• Proposals ranked• Proposals funded in ranking order• Evaluation summary reports provided• No restrictions on re-application

Award Criteria

Excellence(50%)

Impact(30%)

Implementation(20%)

RECOMENDACIONES Redactar pensando en ¿quién y cómo van a evaluar?Ponerse en el lugar del evaluador (poco tiempo y

mucho que leer) Presentar la información claramente:

• Formato inteligible (lenguaje claro, contenidos bien estructurados,índice, figuras, tablas, etc.)

• Propuesta completa pero concisa: ajustarse a lo que piden (“ir algrano”)

El evaluador evalúa lo que lee, no asumir otra cosa

REPASO: No dejar contradicciones o incoherenciasevidentesSolicitar a terceras personas que evalúe la propuesta

previamente

Los toques finales que diferencian el trabajo de calidad

• Lenguaje claro. Inglés “Europeo”.

• Uso de negrita, subrayado, cursiva (no abuso)

• Contenidos bien organizados: Diagramas, tablas-resumen, son muy útiles

• Sin frases hechas vacías de contenido (referencias), sin inconsistencias, sin corta-pegas, sin cifras que no aportan nada, sin páginas perdidas…

• Respetar el límite de páginas impuesto, pero tampoco ser mas parco de lo necesario,

• El evaluador no asume nada, EXPLICA las VENTAJAS de tu proyecto. No dejar que él se imagine que nuestra propuesta es buena, hay que dejarlo claro por escrito.

• Tampoco escribir demasiado ni ser repetitivo

• Ponerse en el lugar del evaluador y tener muy en cuenta los CRITERIOS DEEVALUACIÓN y como se puntúan.

Elaborar una propuesta concisa y fácil de leer

“toques finales”

Muchas gracias por vuestra atención

Cada mediopunto cuenta!!

26