10 December 2015 – Parallel Session

GOOD PRACTICES IN SELECTING MSCA

COFUND FELLOWS

Dr. Jennifer Brennan MRSC, Irish Universities Association

M. Nicolas Walter, European Science Foundation

“Synergies to fuel Researchers’ Careers”Luxembourg, 10 – 11 December 2015

organised by

• AXA Research Fund Postdoctoral FellowshipsNot M(S)CAIncoming scheme

• University of Turin Train2Move ProgrammeFP7 MCA COFUNDIncoming scheme

• IRCSET INSPIRE – FP7 MCA COFUNDOutgoing scheme

• IRC ELEVATE – FP7 MCA COFUND Outgoing scheme

2

Cases Studied

Note: In 2011, IRCSET & IRCHSS merged to form IRC

• AXA Research FundPromoting fundamental research related to risk through mobility grants at the global level https://www.axa-research.org/

• Train2Move Maximising career opportunities for talented early career scientists and attracting promising researchers from all over the world http://cordis.europa.eu/project/rcn/185527_en.html

• IRCSET INSPIREOffering a mobility experience for researchers based in Ireland http://cordis.europa.eu/project/rcn/90290_en.html

• IRC ELEVATEOffering a mobility experience (including into industry) abroad for researchers based in Ireland http://cordis.europa.eu/project/rcn/104975_en.html

3

Objectives of the Programme

4

Boundary Conditions

Disciplinary Coverage Number of applications/awards Co-funded by

AXA Research Fund

• Environmental, Life or Socio Economic Risks

• No Technology development/applied research

• Conform with ethics

• 380-410/year for 25-30 fellowships/year (2014-2015)

• Success rate 6-8%

• N/A

Train2Move • 252 (2014) or 134 (2015) research topics – all disciplines except engineering and architecture

• Conform with ethics

• 156 for 14 grants in 2014198 14 grants in 2015

• Success rate 7-9%

• Compagnia di San Paolo (private foundation)

INSPIRE • All science, engineering & technology• Conform with FP7 ethics

• 47 for 18 grants in 2009Success rate 38%

• 91 for 32 grants in 2010Success rate 35%

• Irish Government

ELEVATE • All science, engineering, technology, humanities and social sciences

• Conform with FP7 ethics• Address potential gender dimension

• 74 for 15 grants in 2012Success rate 20%

• 68 for 30 grants in 2013Success rate 44%

• Irish Government

5

Assessment Structure - 1

Assessment Process Assessment Criteria Implementing agent

Outcome and Selection

AXA Research Fund

Two steps:•Shortlisting of (90) outline proposals by panels •Ranking of full proposals with inputs from external reviewers and candidate’s rebuttal

•Research environment (/4)•Research Project (/4)•CV (/4)

• European Science Foundation as independent third party

• One integrated ranked list across 3 panels

• Selection includes additional criteria

Train2Move Two steps:•Shortlisting of (42) proposals by panels• Interviews and ranking of shortlisted applicants

•CV (/5)•S&T quality (/5)• Implementation (/5)• Impact (/5)

• One integrated ranked list across 4 panels

• Selection follows the ranked list

6

Assessment Structure - 2

Assessment Process Assessment Criteria Implementing Agent

Outcome and Selection

INSPIRE One step:• Shortlisting and

ranking of proposals by panels

•Track record/research potential (30)•Scientific Quality (22)•Proposed Impact (20)•T & CD (15)• Implementation (13)

• IRCSET with assistance from ESF in finding assessors

• 8 ranking lists, one per panel

• Budget distributed based on number of proposals per panel

• Selection follows the ranked list

ELEVATE Call 1 – one step:• Shortlisting and

ranking of proposals by panels

Call 2 – two steps:• Assessment by

external reviewers• Shortlisting and

ranking of proposals by two multi-disciplinary panels

•Track record/research potential (30)•T & CD and Impact (25)•Quality of Project (25)•Host

Org/Implementation (20)

• Irish Research Council

• Call 1 – 11 ranking lists

• Call 2 - 2 ranking lists

• One ranking list per panel

• Budget distributed based on number of proposals per panel

• Selection follows the ranked list

• Challenge in assessing applications from different parts of the world• Different research experience and research environment impacts the

scientific quality of applications although the potential may be present.• Importance of specific and precise guidelines for applicants • Importance of mentoring capability. • Some positive discrimination procedures could be applied (e.g.

identifying a list of ‘priority’ countries upon which identical success rate would be applied)

Need to be thoroughly considered and addressed upstream, particularly if the programme also involves a capacity building

element7

Lessons Learned:

Good Practice in Assessment 1

Challenge in addressing the mobility aspect• Significant number of applications are not proposing real

mobility as they involve applicants returning to their home countries or foreign applicants wishing to stay in the same organisation

• Challenging for panel members to approach• Train2Move set an eligibility criteria (<12 months in Italy

over the past 3 Years) – Allows to better streamline the issue• INSPIRE & ELEVATE set a strict criteria about outgoing

mobility

8

Lessons Learned:

Good Practice in Assessment 2

Diversity Measures - ELEVATE• Details regarding age, nationality and gender of applicants is

hidden from evaluators so as to reduce the effects of unconscious bias. • Ongoing process as Referees/Supervisors/Mentors sometimes

inadvertently provide revealing information

• Applicants should include consideration of possible gender dimensions in their proposed research.• Training and information on gender dimensions in research should be

provided to applicants and other participants

• IRC Gender Action Plan – aim for gender balance in the membership of all assessment panels

9

Lessons Learned:

Good Practice in Assessment 3

• Assessing potential complementarities and synergies with the host department/institutions• Motivating or making it mandatory for candidates to contact and interact

with the host and possibly their local supervisors and asking them to report on this

• Involving representatives of the host organisation in the process and asking for direct feedback to the evaluators (letter and/or interview)

• INSPIRE/ELEVATE – application ineligible without written support from host at two levels, supervisor and institution

Overcomes any lack of detailed knowledge of the host organisation and better consideration of the feasibility of the research project

proposed. 10

Lessons Learned:

Good Practice in Assessment 4

• Maximising the interaction with the candidates• AXA RF:

• Rebuttal step allowing clarification of issues in the external evaluation reports• Reports provided to shortlisted candidates before they submit full applications –

allow consideration of comments• Career statements: allows a broader perspective on the candidates and their

trajectory

• Train2Move:• Specific questions to be addressed during the interview in the shortlisting

assessment report• Skype/phone interview

Interactions with candidates allows clarity around the application, removes potential misunderstandings and also improved perspectives

on candidates and their trajectory 11

Lessons Learned:

Good Practice in Assessment 5

Ensuring a high quality evaluation (and dedicating enough time for this!)• Critical that evaluators have all the same understanding of criteria

• E.g. the ‘Impact’ criteria often hard to approach: impact for the candidate, for the host, on the discipline? Needs to be carefully explained

• Disciplinary review panels should be composed of experienced researchers with a broad knowledge and understanding of the disciplines

• External (mail) reviewers should be precisely targeted to hold the required expertise and allow review panels to make better informed decisions

• Assessments by external reviewers should be checked and validated to provide a coherent level of information

• Reports from panel members should be checked and validated to ensure they are useful for the applicant

• Conflict of Interest should be dealt with in a systematic and consistent manner• Importance of the panel secretaries in ensuring procedure is followed 12

Lessons Learned:

Good Practice in Assessment 6

Evaluators need to be well treated!• Even if some proposals may be rather short, going through them,

checking some background and producing useful assessment always take more time than expected• Limit the number of applications per reviewer/panel member

AND/OR• Allow more time to produce

• Support from the office is critical - Importance to provide smooth operations • at the preparation/reporting phases (online platform)• during the meetings (experienced staff and well defined guidelines)

• Some consideration could be given to compensate evaluators (IRC does)• Allow reviewers to address consolidated comments (e.g. panel chairs’

declaration13

Lessons Learned:

Good Practice in Assessment 7

Assessment is an investment• The application evaluation is part of the critical path towards

maximising the value and impact of a fellowship programme• Targeted identification of evaluators (panel members and external

(mail) reviewers) is key• Well defined and coherent guidelines need to be produced and

respected• Check and validation of assessments and reports is very

important, but is time-consuming

All these take a lot of time and require specific expertise14

Lessons Learned:

Good Practice in Assessment 8

Partnering with a third party with a core competency in assessment

The ESF experience of partnering with fellowship programmes allows strengthening of the robustness of the assessment process while lifting some pressure on the programme management team• Allows separation of the assessment from the selection process – increases

the independence• Allows mobilisation of multidisciplinary teams and buffers workload peaks

when time is limited• Allows use of existing optimised and proven procedures, infrastructure and

experience• No need to develop in-house know-how in case of non-recurring process• Allows gain and development of in-house capacity and know-how to build

up internal capacity• Allows access to dedicated databases of experts 15

Lessons Learned:

Good Practice in Assessment 9

Dr. Kate Ryan, Irish Research Council

16

Acknowledgements