evaluation in r&d sphere in ukraine: real practice and problems of transition to new standards
DESCRIPTION
Evaluation in R&D sphere in Ukraine: Real practice and problems of transition to new standards. Igor Yegorov Centre for S&T Potential and Science History Studies, National Academy of Sciences of Ukraine and the Institute of Statistics, State Service of Statistics of Ukraine - PowerPoint PPT PresentationTRANSCRIPT
Evaluation in R&D sphere in Ukraine: Real practice and
problems of transition to new standards
Igor YegorovCentre for S&T Potential and Science History Studies, National Academy of
Sciences of Ukraine and the Institute of Statistics, State Service of Statistics of
Ukrainee-mail: [email protected]
Paper presented at the International workshop
Science evaluation as a prerequisite for promoting excellence in research
6 November 2012
Chisinau, Moldova
Key objects of evaluation in R&D and innovation sphere in Ukraine
a) Individual researchers
b) Research organizations (institutes, design bureaus and so on)
c) Research and innovation projects
d) R&D programs and R&D sections of development programs
Main sources of data
Expert opinion (individual experts or expert panels) State statistics (State Service of Statistics of Ukraine) Department of Intellectual Property Institute of S&T and Economic Information Internal statistics of different ministries and agencies (for
instance on functioning of technoparks) Information from special surveys and databases Foreign sources (internationally recognized publications,
awards and so on)
Problems with utilization of official statistics
Rigidity in approaches and difficulties in changes of indicators and questionnaires
Deviation from international standards (examples: system of scientific degrees, calculation of number of publications and so on)
The willingness to serve ‘political’ interests (example: number of research projects and other indicators, which are working ‘in favour’ of old system)
Key regulatory acts
Law on Scientific and S&T Activities (1992) Law on Innovation Activities (2002) Law on S&T Expertise (2006) Law on Higher Education (1993) Law on State S&T Programs (2004) Total number of laws and other regulatory acts on R&D
and innovation exceeds 80. Almost all of them contain clauses on evaluation at different levels.
Important features of the Ukrainian research system
High level of centralization in Ukrainian R&D system: administrative power of rectors and directors of the research institutes is ‘excessive’ (including full control over all financial transactions)
System of the evaluation of research results with clear domination of internal publications, not international publications as indicators of success
The procedures of evaluation and selection of R&D projects are often not transparent and fair for potential participants.
Recent changes in R&D and Innovation statistics
Introduction of the CIS-type survey in Ukraine (2009-2010, data for 2008-2009). New survey is expected in 2013
Switching (partial) to international classification of sectors - since 2009
Implementation of new NACE-type classification (new version of KVED) – since 2012
Implementation of indicators of international publications in some universities and research institutes
Publications of Ukrainian authors in scientific journals in 1996-2005, tous. (Thomson
Reuters)
3700
3800
3900
4000
4100
4200
4300
4400
4500
4600
4700
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
Number of scientific publications according to the State Statistical Service of Ukraine,
1991-2011, thous.
Evaluation of individual researchers
Method: Panels of experts from the same institute (as a rule: at the level of department and at the level of the institute) discuss the work of researcher and vote openly for the decision. This decision has to be confirmed by the scientific council of the institute.
Periodicity: Every five years Indicators are used indirectly (number of publications,
participation in conferences and seminars, awards, preparation of the thesis (if any), participation in a work of different scientific councils and commissions and so on)
Evaluation of research organizations
A) Panel of external experts from the same research area prepares a report on activities (every four-six years). Indicators include number of publications, number of patents and so on.
B) Evaluation with implementation of the special metrics (2012)
Evaluation of potential (within the projects or programs) varies according to the goals of evaluation (academic programs, STCU and so on)
Evaluation with implementation of the special metrics (2012)
‘Objective’ indicators (60% of final assessment) Expert evaluation (40% of final assessment) ‘Objective’ indicators:
a) personnel (5 indicators)
b) financial situation and research equipment (8 indicators)
c) research in priority areas (3 indicators)
d) publication activities (2 indicators)
e) innovation activities (4 indicators)
f) international activities (3 indicators)
g)’visibility’ at the international level (2 indicators)
Example1: ’Visibility’ at the international level
Number of references on the research organization in the Google searching system
Number of publications in domestic and international scholar journals with high impact-factor per one employee of the research organization
Example 2: Publication activities
Number of publications in domestic scholar journals per employee of research organization
Number of publications in foreign scholar journals from international databases of scientific publications per employee of research organization (data bases include information from Scopus and some other databases on medical publications)
Expert evaluation of research organization
Level of research activities Relevance to the national priorities Potential for further development Indicators, which reflect specific features of research
area
Note: all indicators have their own weights, depending on the type of organization
Evaluation of research projects Ex- ante evaluation Evaluation of the process (if the project has long duration
or it is divided into several stages) Ex-post evaluation Different types of indicators are used, depending on the
type of the project Special forms are used by different agencies and
ministries, not less than two experts take part in the evaluation procedure
Key idea is to reach the goal of the project (must include, if possible, description of quantitative indicators, which have to be achieved) within certain time and financial constraints
Example of indicators for selection of the project for the Ministry of Economic
Development and Trade (October 2012) Proposed price (60% of the final assessment) Other indicators (40%):
a) Number of Doctors and Candidates of Sciences in the research team (30 points, if more than 7: 3D+4C)
b)Number of articles in scholar journals (30 points, if more than 10) on the topic of the project
c) Monographs on the subject (15points per monograph)
d) Participation in similar projects with the Ministry during the last 3 years (10 points per project)
Evaluation of the programs In accordance with to the normative acts and existing
laws at the state and ministry level Problems:
1. Lack of quantitative criteria and utilization of ‘artificially constructed criteria’
2. Non-transparent expert procedures (as a rule, no foreigners take part in the evaluation, excluding Key Laboratory initiative and some joint programs with foreign partners)
3. Lack of money for financing of R&D programs, which is used as an argument for justification of the level of results achieved within the program
Conclusion Ukraine has not yet created relevant institutions and
proper instruments necessary for effective transformation of R&D sector, including effective system of research assessment. Internal practice of evaluation differs substantially from the international one.
Unfortunately, every year the possibilities for the implementation of effective transformation policy are shrinking, as the number of researchers is declining and the research centres are losing their ability to conduct research.
Conclusion With time running out, remaining research institutions will
face more problems with the creative capabilities needed to compete internationally
Reforms of evaluation of R&D sector have to be an integral part of general transformation of R&D system
EU could contribute to the reforms by stimulating introduction of new standards of R&D project assessment and implementation
Thank you for your attention !