estonian experience in evaluating research proposals madis saluveer head, department of research...
TRANSCRIPT
Estonian experience in evaluating research proposals
Madis Saluveer
Head, Department of Research FundingEstonian Research Council
1Regional workshop "Science evaluation as a prerequisite for promoting excellence in research", Chisinau 6 November 2012
Outline
• Institutional restructuring in Estonia in 1990s and its outcomes
• International evaluations of Estonian R&D Changes in the institutional financing system
• Investing into the development of R&D system• Evaluating research - issues for discussion
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 2
R&D and higher education reform in 1990s
• Legislative acts to support the functioning of the new system were adopted
• Institutional reform launched
• International evaluations of research areas and the R&D funding system carried out
• Higher and post-graduate education brought into accordance with international criteria
• Research financing system reformed
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012
3
Institutional reform in 1990s
• 1989-1990 - the Estonian Academy of Sciences was re-organised into a classical (personal) type of Academy -> former Academy research institutes were merged with universitiesNature 1 Oct 2009: “With the exception of Estonia, the basic research in the region (CEE) is still controlled by the national academies of science...”
• 1990-1991 All Soviet Union ministerial branch institutes dissolved
•1990 - Estonian Science Foundation and Estonian Innovation Foundation founded• 1997 – Research Competence Council (=>Estonian Research Council)
established• 1997 - Archimedes Foundation founded (management of EU
framework programmes + single NCP network)4Regional workshop "Science evaluation as a prerequisite for promoting excellence in
research„ Chisinau 6 November 2012
Establishing the R&D legal „backbone“• 1994 – „Organization of Research and Development Act“ adopted by
the Estonian Parliament; latest amendments approved by the Parliament in February 2011 (researcher career path, financing instruments, re-organisation of funding agencies)
• 1995 – Universities´Act adopted• 1997 – Estonian Academy of Sciences Act• 2001 - Estonian R&D Strategy 2002-2006 “Knowledge-based Estonia”
approved by the Estonian Parliament• 2006 - Higher Education Strategy for 2006-2015 approved
• 2007 - Estonian R&D and Innovation Strategy 2007-2013 “Knowledge-based Estonia II” adopted by the Estonian Parliament
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research", Chisinau 6 November 2012 5
Rational of performance-based funding
The rationale of performance funding is that funds should flow to institutions where performance is manifest:‘performing’ institutions should receive more income than lesser performing institutions, which would provide performers with a competitive edge and would stimulate less performing institutions to perform.
Output should be rewarded, not input.
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research", Chisinau 6 November 2012 6
Internationational evaluations of Estonian R&D system 1
Evaluation of all R&D areas by Royal Swedish Academy of Sciences and the Swedish Research Councils 1991-1992•Approach: broad and deep evaluation of the whole system; no broad legislative background existed •Goals: to receive an unbiased opinion on the level of research according to the standards used in Scandinavia; to introduce the international norms; to get advice what to do•Outcome: More than 400 reports presented by all research groups in all fields of research; 8 expert panels paid site visits; 19 reports producedObservations and recommendations: •Restructure the policies for decision-making•Integrate research and education• Unequal quality across the fields•Stop publishing in local papers without peer review-> promote peer review•Lack of young scientists -> normalise age structure, increase the number of research students•Establish/ promote international collaboration
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research", Chisinau 6 November 2012 7
Internationational evaluations of Estonian R&D system 2
2000-2003 – the first systematic evaluation based on established legal basis (ORDA)
Approach: full and deep evaluation by the fields of science – 46 expert groups (approx. 200 foreign experts from 18 countries)
Outcome: Very good job done, a great number of substantial recommendations put forward; however, the results were heavily underused. Process costly, slow, involved a lot of bureaucracy; assessment methodology changed/improved over the evaluation period; units of assessment varied from small research groups to whole institutes; rapid change of R&D landscape over the evaluation period
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 8
Internationational evaluations of Estonian R&D system 32009 – new regulation in ORDA: introduction of 2 types of evaluations
2010 - regular evaluation ( “wide and shallow”, every 7 years):Approach: voluntary, covers the whole R&D system, fast, low cost, low bureaucratic burden - based mostly on existing information, full usage of Estonian Research Information System ETIS; peer-review carried out by foreign experts (16 peers); strict rules; simple outcome (yes/no)
Consequences:•financial: access to state research funding instruments – base-line funding, targeted funding•for HEI-s – prerequisite for the PhD studies
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 9
Internationational evaluations of Estonian R&D system 4
Targeted evaluation (“narrow and deep”; „tailor-made“ : topic, subjects of evaluation, terms of references, evaluators and the detailed mechanism of evaluation may vary)
Goals may vary: •gaining a profound insight of a research field•gaining information for outlining new research policy measures, e.g. R&D strategies, research programmes, major research policy decisions, funding mechanisms, etc, and/or for elaborating the existing ones•may be carried out in single or in multiple recearch areas
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 10
Internationational evaluations of Estonian R&D system 5
2011 – first targeted evaluation carried out (plant and soil science) Approach: based on „tunnetatud vajadus“; self-assessment reports; international peer review (5 experts from 5 countries), site visits; no grades but policy recommendations
Outcome: evaluation committee report with observations and recommendationshttp://www.etag.ee/wp-content/uploads/2012/05/SEV_PMT_report_2012.pdf
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 11
Estonian R&D expenditure 2001-2010
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 12
Number of publications of Estonian researchers indexed in Thomson Reuters WoS
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 13
Estonian participation in EU framework programmes
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012
14
FP7 budget per researcher (thousand €)
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 15
Estonian Research Information System ETIS: some facts• ERIS (predecessor of ETIS) 1999-2006• Need for large scale developments (real-time multi-partner dynamic
system)• Spring 2006: launch of ETIS with pre-installed data (users,
publications, projects, etc)
Evolution of ETIS as data repository: • 2008 - 78 000 publication records• 2010 - 100 000 publication records• 2012 - 134 000 publication records• 2012 - 39 different electronic forms in use • September 2012: more than 76 000 visitors (Estonia 71 393, Finland
1 212, Germany 860, USA 581, UK 569, Sweden 547, Latvia 312, Italy 300, France 273, Spain 243)
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012
16
ETIS functionalities
• an information channel and a tool for researchers and R&D institutions
• a gateway where researchers can submit grant applications and reports
• an instrument for funding bodies to process applications, reports and reviews
• a state register of R&D institutions, researchers, financiers and research activity
• a gateway to Estonian science landscape• a means to get fast and adequate R&D statistics• a trusted data repository
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 17
ETIS classification schemes
• Research Classification Scheme• Classification of publications in ETIS• List of internationally acknowledged
publishers
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 18
Classification of publications in ETIS1. Articles in journals 1.1. Scholarly articles indexed by TR Web of Science or published in journals indexed by ERIH categories INT1 and INT2; 1.2. Peer-reviewed articles in other international research journals with an ISSN code and international editorial board, which are circulated internationally and open to international contributions; articles of ERIH category NAT; 1.3. Scholarly articles in Estonian and other peer-reviewed research journals with a local editorial board; scientific articles in journals important for Estonian culture Akadeemia, Looming, Vikerkaar.
2. Books/monographs 2.1. High-level peer-reviewed research monographs; 2.2. Other monographs; 2.3. Dissertations published in a series of dissertations (excluding manuscripts).
3. Articles in proceedings/a chapter in a book or in a collection 3.1. Articles/chapters in books published by the publishers listed in Annex (incl collections indexed by the TR Conf Proc Citation Index); 3.2. Articles/chapters in books published by the publishers not listed in Annex; 3.3. Specific research publications (dictionaries, lexicons, sets of maps, (field) guides, text-critical publications); 3.4. Articles/presentations published in conference proceedings not listed in Section 3.1; 3.5. Articles/presentations published in local conference proceedings
4. Editing scientific publications 5. Published meeting abstracts 6. Other publications 6.1. Full articles in encyclopedias; 6.2. Text books and other study materials; 6.3. Popular science articles; 6.4. Popular science books; 6.5. Essays accompanying books of fiction; 6.6. Articles in other journals and newspapers; 6.7. Other creative activitiesRegional workshop "Science evaluation as a prerequisite for promoting excellence in
research„ Chisinau 6 November 2012 19
Classification of research fields in ETIS Biosciences and Environment (12 sub-fiels): Biochemistry, Microbiology, Genetics, Ecology, Biosystematics and –physiology, Forestry, Agricultural Sciences, Food Sciences, State of the Environment and to Environmental Protection, Substances Hazardous to the Environment, Geography and Regional Studies, Environmental Policy, Environmental Economy and Environmental Law, Biotechnology, Molecular Biology, Cell Biology, Biophysics and Economic and Technological Research relating to the above fields
Culture and Society (14 sub-fields): Philosophy, Theology, History and Archaeology, Cultures Research, Aesthetics and Arts Research, Philology and Linguistics, Law, Psychology, Special Education, Educational research, Social Sciences, Economics, Political Science and Administration, Communication and Information Sciences
Health (11 sub-fields): Biomedicine, Veterinary Medicine, Pharmacy, Dental Science, Nursing Science, Public Health Science, Clinical Medicine, Sport Sciences, Nutrition, Occupational and Environmental Medicine, Biochemistry, Genetics, Microbiology, Biotechnology, Molecular Biology, Cell Biology, Biophysics and Bioinformatics relating to the above fields
Natural Sciences and Engineering (17 sub-fields): Architecture and Industrial Design, Geosciences, Space Research and Astronomy, Mathematics, Statistics, Information Processing Sciences, Telecommunications, Electrical Engineering and Electronics, Medical Engineering, Physics and Technical Physics, Chemistry and Chemical Technology, Process Technology and Materials Science, . Mechanical Engineering, Automation Technology and Manufacturing Technology, Industrial Engineering and Management, Construction and Municipal Engineering, Energetics, Biotechnology relating to the above fields
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012
20
Evaluating research proposals - issues for discussion
• Types, scope and frequency of evaluations• Peer-review and/or „metrics“• Scope of use of bibliometrics• Impact of research evaluations on funding
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 21
Evaluation of research proposals 2
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 22
1997 – 2012: targeted financing of research topics (multi-annual team funding)•Assessment scheme: set requirements for PI and team (presence of mandatory publications), international (blind) peer review (2-3 peers), expert panels, final approval by MER => medium competition, small grants (~100 000 €/year), average number of financed topics (~200)2012 replaced by Estonian Research Council’s institutional funding (multi-annual team funding of institutions)•Assessment scheme: international blind peer review, expert panels, final approval by MER => high competition, average grants (~200 000 €/year), low number of grants (2012: 31; total ~100)
1994-2012: Estonian Science Foundation (personal) grants •Assessment scheme: set requirements for PI (presence of mandatory publications), international peer review (2 peers), expert panels => low competition, very small grants (~10,000 €/year), high number of grants (total 500 – 600)2012 replaced by Estonian Research Council’s personal grants•Assessment scheme: international blind peer review (2-3 peers), expert panels, final assessment by the Evaluation Committee => very high competition, average grants (50 – 70 000 €/year), small number of grants (2012:~20, total ~100)
Evaluation of research proposals 3
MER Centres of excellence programme 2001-2007 (2 calls)•Assessment scheme: peer review by local experts ; final approval by MER => 4 – 11 research groups per CoE; average competition, total 10 CoE, total programme 6.4 M€
Centres of excellence in research programme 2008-2013 (2 calls)•Assessment scheme: international peer review (30 peers); final approval by Programme Steering Committee => 4-8 research groups; high competition, total 12 CoE, total programme 55.3 M€
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012
23
Evaluation of national programmes
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research", Chisinau 6 November 2012 24
EU-financed national thematic R&D programmes 2007-2013 (6 programmes, total 51.8 M€) •Assessment scheme: peer review by local and international experts; panel meeting and final approval by the Steering Committee => complementary funding, targeted R&D domains, low competition, grant sizes vary by domain (e.g. Health technologies 200 -500 000 €, environmental technologies 130 – 400 000 €, materials and nanotechnology 200 – 850 000 €, etc.)
MER national programmes (limited domains, e.g Estonian language & literature, history, folklore; national collections; language technology)•Assessment scheme: peer review by local experts; final approval by the Steering Committee => small teams, low competition, small grants (15 -50 000 €/year)
Research infrastructures of national importance
(Roadmap, investment plan)SUPPORT 29,4 mil eur
Research infrastructures of national importance
(Roadmap, investment plan)SUPPORT 29,4 mil eur
Measure 2.4: Modernising research apparatus and equipment
SUPPORT 110,7 mil eur
Measure 2.4: Modernising research apparatus and equipment
SUPPORT 110,7 mil eur
Copyright
Modernising the general infrastructure for
researchSUPPORT 86,5 mil eur
Modernising the general infrastructure for
researchSUPPORT 86,5 mil eur
Modernising the general infrastructure for higher
educationSUPPORT 42,1 mil eur
Modernising the general infrastructure for higher
educationSUPPORT 42,1 mil eur
OP 2 – Operational Programme for the Development of Economic Environment
Measure 2.1: Thematic R&D programmes aimed at long-term
economic development SUPPORT 51,8 mil eur
Measure 2.1: Thematic R&D programmes aimed at long-term
economic development SUPPORT 51,8 mil eur
Priority 2: Improving the competitiveness of Estonian R&D
activities through the research programmes and modernisation of
higher education and R&D institutionsSUPPORT 346,7 mil eur
Priority 2: Improving the competitiveness of Estonian R&D
activities through the research programmes and modernisation of
higher education and R&D institutionsSUPPORT 346,7 mil eur
Measure 2.3: Modernising the general infrastructure of R&D institutions
SUPPORT 128,7 mil eur
Measure 2.3: Modernising the general infrastructure of R&D institutions
SUPPORT 128,7 mil eur
Modernising the educational environment
of institutions of professional higher
educationand universities
SUPPORT 14 mil eur
Modernising the educational environment
of institutions of professional higher
educationand universities
SUPPORT 14 mil eur
Measure 2.2: Centres of excellence in research and participating in
research cooperationprogrammes of the EU and the Baltic
Sea regionSUPPORT 55,3 mil eur
Measure 2.2: Centres of excellence in research and participating in
research cooperationprogrammes of the EU and the Baltic
Sea regionSUPPORT 55,3 mil eur
Research equipment for R&D institutions
(R&D institutions, open call)SUPPORT 46,3 mil eur
Research equipment for R&D institutions
(R&D institutions, open call)SUPPORT 46,3 mil eur
E-resources programmeSUPPORT 6,4 mil eur
E-resources programmeSUPPORT 6,4 mil eur
Small-scale research equipment for research groups
(Research groups, open call)SUPPORT 14,5 mil eur
Small-scale research equipment for research groups
(Research groups, open call)SUPPORT 14,5 mil eur
researchresearch higher educationhigher education research and higher educationresearch and higher education25
Estonian RI Roadmap• RI roadmap – a long term (10-20 years)
planning instrument that lists research infrastructures of national importance, either new or in need of upgrading
• The roadmap will be updated regularly (at an interval of 3 to 5 years) to take into account the changing circumstances and opportunities
• Being part of the roadmap does not mean that a RI will be funded but the roadmap will be used as an input for the investment decisions
• Budget 29.4 M € for 2010-2015• Financed from EU ERDF (85%), state
budget (10%) and own contribution (5%)• https://
www.etis.ee/Portaal/infrastruktuur.aspx
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 26
Evaluation of research infrastructure proposals
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 27
Support for R&D institutions (total: 46.3 M€)•Assessment scheme: open calls; R&D institutions; 2-step review: local peer review and approval by programme Steering Committee; RIs of national importance (total: 29.4 M€)•Assessment scheme: call for expressions of interest; call for full proposals (investment plans); R&D institutions; main and reserve list, final approval by Estonian Government;
Support for small-scale research equipment (total: 14.5 M€)• Assessment scheme: annual open calls; research groups; 2-step review: local peer review and approval by ETAg Evaluation Committee;
Thank you for your attention!
Regional workshop "Science evaluation as a prerequisite for promoting excellence in research„ Chisinau 6 November 2012 28