dir s erc/european commission rtd, directorate s
DESCRIPTION
The European Research Council. The evaluation of the ERC-2007-StG Call. Dir S ERC/European Commission RTD, Directorate S. IDEAS Programme Committee, January 31, 2008. Overview. Status of evaluation report Key data – both stages The eligibility process Budgets by domains and panels - PowerPoint PPT PresentationTRANSCRIPT
Dir SERC/European Commission
RTD, Directorate S
The European Research Council
The evaluation of the ERC-2007-StG Call
IDEAS Programme Committee, January 31, 2008
European Research Council
│ 2
Overview
1. Status of evaluation report
2. Key data – both stages
3. The eligibility process
4. Budgets by domains and panels
5. The panel meetings and the ranked lists per panel
6. The consolidation by the Panel Chairs
7. Recommendations on grant levels
8. Feedback to applicants
9. Status of redress
European Research Council
│ 3
Status of evaluation report
Will be conform to that of other programmes
• Offering same facilities to PMC members
• We did not succeed having a fully quality checked document for this meeting. However, key data tables are distributed
Formal E.R. statistical reporting will be based on the sample of 201 proposals in the main list – by convention
Informal reporting, including public dissemination, is based on a sample of the top 300 proposals
This is a reasonable estimate of the number that may be funded
European Research Council
│ 4
Key data- number of proposals by evaluation step
9167
8794
559
554
547
368 ineligible
5 withdrawn
8235 rejected
5 not submitted to second stage
2 passed away, 4 ineligible
1 withdrawn
Submitted stage 1
Evaluated stage 1
Selected stage 1
Submitted stage 2
Evaluated stage 2
201 in main list
116 in reserve list
113 reserve: reject for no budget
117 rejected: below thresholds
European Research Council
│ 5
Key data: Evaluation processStage 2: 559 proposals expected
Eligibility and withdrawals
Reception of proposals
INDIVIDUAL ASSESSMENTS
Panel members Referees
INTERVIEWS
20 panel-ranked lists
PANEL MEETINGSPANEL CHAIR
MEETING
Single Consolidated
list
1 day
554 7
547
2 to 3 days
European Research Council
│ 6
The eligibility process
Eligibility decisions have been taken conform the standard FP7 methodology - and in accordance with its ‘case-law’
Some of the decisions have been complex, and have required discussion in parallel with the review
We attempt to minimise the number of such cases
However, given the numbers and the time-constraint, partial parallel processing of review and eligibility is necessary – and is in accordance with the Rules
European Research Council
│ 7
Budgets by domains and panels,
based on adopted / published call budget, 289.5M€
80% 20%
8 PE panels 7 LS panels 5 SH Reserve
Nominal panel budgets, proportional to sum of stage-1 requested grant
E.g. PE 1 2 43
E. g. PE4
Fits in nominal panel budget,Score > 9, no ordering
…
Main list Reject
Candidate for reserve budget,Ranked in priority order, with scoring convention
Reserve ………..
289.5M€Additional budget
contributions add to reserve
Additional budget
Not fundable,Score < 8
European Research Council
│ 8
The stage-2 interviews / panel meetings
Panels operated independently, typically 3 days
Interviews with applicants perceived as extremely useful complement to individual assessments
Panels identified four groups of proposals:1. Main list proposals, inside nominal panel budget
2. Reserve list, serious candidate, priority ordering
3. Reserve list, good proposal, clearly outside budget
4. Rejected proposal, failing threshold
Panels tagged inter-disciplinary, cross-panel / domain proposals
For groups 1 and 2, panels recommended the grant level
European Research Council
│ 9
The panel chair consolidation meeting
Purpose of the meeting: to establish a consolidated ranking of the serious candidate reserve (group 2) proposals
1. With special emphasis on inter-disciplinary proposals
2. Given 20 panels across all scientific fields, not a trivial affair ….
3. Because no absolute excellence standard exists: small score differences are meaningless across panels
To faciliate, a starting point ranking was needed1. Not constraining the freedom of the panel chairs
2. ‘Bureaucratically fair’: purely based on an algorithm
3. Accounting for different panel and proposal sizes
4. Without prejudice to any possible differences of excellence between panels
European Research Council
│ 10
The consolidation method (simplified)
By convention, all panels scored their serious reserve candidates at 8.9, and sub-ranked them: first, second, ….
Accum
ulated grant
Norm
alised accum
ulated grant
Normalise each panel on its nominal budget
Res 1Res 2Res 3Res 4Res 5Res 6Res 7Res 8Res 9…..
Tagged as inter-disciplinary
1.0
Nominal budget
Ordering by normalised accumulated grant
European Research Council
│ 11
The final ranking by the panel chairs
Regarding inter-disciplinary proposals, the panel chairs:
1. Considered that they did not have the resources to re-examine these in detail
2. Confirmed their confidence in the prior work of the panels – ‘mainstreaming’
Regarding the ranked list, the panel chairs unanimously adopted the starting point proposed by the Commission as a fair result
This ranking has not been modified, neither by Commission nor by the Scientific Council
European Research Council
│ 12
The issue of recommended grant levels - I
Panels’ remit included providing recommendations on award levels
Individual panels faced some difficulties in this respect:
1. Proposals not overly detailed, over-estimates by PI or host …
2. Incomplete understanding / interpretation of the Rules of Participation
3. Contradictory information during interviews
Driven by this, and by different ‘needs of the field’, panels arrived at different solutions:
1. No grant reductions; reductions across the board; big reductions on some proposals
2. Some panels have removed the PI salary for PI’s with permanent positions – problematic in view of Rules and of grant mobility
3. Some panels have calculated the grant ‘bottom-up’
European Research Council
│ 13
The issue of recommended grant levels - 2
The ‘reductions issue’ was discussed in the Panel chair meeting
1. In general, panel chairs confirmed the positions taken by their individual panels as fair and reasonable in the context of the field
2. A small number of corrections was introduced during the meeting
3. On the specific issue of the PI’s salary, panel chairs realised the difficulty of the situation and recommended that the Commission applies appropriate corrections
The Commission has applied a correction to all cases where the PI’s salary was explicitly removed
The Scientific Council has strongly endorsed the position of the panel chairs, and requested the Commission to award the grants accordingly, without negotiation
European Research Council
│ 14
Towards granting
The Commission has maintained the possibility for successful applicants to seek redress against the level of awarded grant, using the redress procedure
• In reality, no such request for redress has been received
The Commission has been cautious in its feedback to applicants
1.Pending formal decisions on 44.5M€ of third-country contributions
2.In view of the pending redress
Granting has started on the main list – 201 proposals
1.About 40 ethical reviews are ongoing
2.The expectation is that about 300 grants will be awarded
European Research Council
│ 15
Transparency towards applicants
Four messages to the four groups of applicants:
1. 201 proposals in main list: granting is imminent, started for 65
2. 116 serious reserve candidates; all have score 8.9; probability of a grant varies from 1 to 0 down the ranking
3. 113 good proposals but clearly outside budget; score between 8.0 and 8.8; clear information given
4. 117 proposals fail threshold; score < 8.0; clear information given
Subject to individual disclaimers, list of all 430 proposals above threshold is now published on ERC web-site
1. Most of the 116 serious reserve candidates can make a reasonable assessment of their probability
European Research Council
│ 16
Status of redress
The ‘Redress’ procedure has worked well
For Stage-1, 245 requests for redress received, three main areas:1. Eligibility – none sustained
2. Factual errors by reviewers – 15 sustained
3. Scientific judgement – none sustained
15 stage-1 proposals were re-reviewed by panels
1. One applicant has been invited to submit a stage-2 proposal
Stage-2 redress: has just started1. 27 requests were received
European Research Council
│ 17
Details of stage-1 redress
Scientific judgement of panels 150
The review process 201. Wrong panel 6
2. Conflict of interest 14
Very near to threshold 53
Negative, offensive 251. Due to editing / English 18
2. Discriminatory 7
Eligibility 49
Factual errors by reviewers 26
Rebut reviewers / complain size of proposal 6
NOTE: a single redress request may address multiple categories
European Research Council
│ 18
Main lessons from stage-2
Overall, the process worked well: consolidation effective
Interviews successful, to be maintained for StG
Review the framework in which panels operate1. Respect autonomy, specificities of scientific fields
2. But more need for coherent decision-making
The ‘mainstreaming’ approach to inter-disciplinarity has worked, but reflection needed:
1. It does not give much visibility
2. Difficult to achieve inter-panel coherence
Reviewers comments in feedback to applicants:1. Reveals some flawed judgements – redress
2. Transparency inevitably drives improvement