Evaluation and Benchmarkingof RTD Organisations -

Experience from REGPOT

Steering Platform on Researchfor Western Balkan Countries

Zagreb, 29 October 2009

Evaluation Pyramide

many many many many many many many many manymany many many project evaluations

some RTD programme evaluations

just a fewevaluations of

RTD institution

almost no systemevaluations

The different levelsrequire different evaluationskills and methods

Peer Reviews

Bibliometrics, network

analysis etc.

Peer Reviews, SWOT, CBA

multi-level approaches

,etc.

Case Study

FP7-REGPOT-2008-2

SWOT-CHEMISTRY-FOOD (Evaluation of the research capacity and development of a strategy for further growth in chemistry in general and in food science in particular)

University Ss. Cyril and Methodius, SkopjeInstitute of Chemistry (around 50 personnel)

Target Group(s)

1. The management of the research organisation (institute or faculty or university)

2. (The employees of the institute - indirectly affected)

3. The policy-delivery system

RTD managers and policy makers are in the same boat:

Managers – to improve the competitiveness of the institute

Policy makers – to improve the quality of the entire innovation system

Organisations are important elements of the innovation system

The generic model of national innovation systems as presented in OECD (2003)

Policy Objectives

1. To map competitive excellence, resources and competencies and obtain transparent information about the state-of-art of the RTD organisations

2. To improve the performance of RTD organisations and, thus, to improve regional, national or local research and/or innovation systems

3. To have a evidence-based ground for channeling of funds and allocation of resources and factors

4. (to increase the application of a modern management tool in order to foster reflection and understanding over assumptions and beliefs)

A lot could be done, but it needs the power of a mammoth!

Evaluation Functions

1. Learning (information and steering function)

2. Accountability (legitimization and control function)

Target groups may have contradictory interest or not.

„Why evaluation?“

Case Study – Law on Scientific Centre of Excellence (1)

Scientific Centre of Excellence (Art. 23, 47, 48)

Article 23 (DEFINITION)A Scientific Center of Excellence is a scientific institution or a group of researchers who by merit of their originality, significance and relevance of the achieved results based on their scientific research activity belong to the institutions or groups of researchers with highest quality in the world in their area of scientific research.The Scientific Center of Excellence should have accomplished: a strong connection with learning, research and innovations, that is close relationship between research and industry; a focused research program which involves interdisciplinary topics, applicative and fundamental endeavors; to be able to provide a dynamic, flexible and attractive work environment which will result in high accomplishments, at an individual or team level; procedures to educate new generations of scientific and technological talents; as well as the capacity to accomplish dynamic partnership with the organs of the central and the units of local government.

Case Study – Law on Scientific Centre of Excellence (2)

Evalution of the Scientific Centre of Excellence (Art. 23, 47, 48)

- Subordinated to continual monitoring and external evaluation (efficiency, sustainability, efficacy and relevance of activities )

- Evaluation results will be published transparently. - Obligation to perform self evaluation of the quality of their

scientific research activity based on their own system for quality control management in accordance with their statute and the international standards for evaluation.

- The external evaluation is performed exclusively based on international standards by the Agency for Evaluation in accordance with the Law for Higher Education.

Evaluation of the Institute of Chemistry

Mode: voluntarily commissioned independent external evaluation carried out by two independent evaluators proposed by EC (Prof. Stane Pejovnik, Rector of the University of Ljubljana and Klaus Schuch, Director of ZSI)

Context: supported by an EC funded REGPOT project integrated in an overall evaluation of the food sector

Budget: € 54.000,00 out of which € 27.000,00 for external evaluators

Duration: € 1 year

Outputs: evaluation methodology (including questionnaires), evaluation report, action plan

Methodology

Borsi, B., Dévai, K., Papanek, G. and Rush, H. (2004): The RECORD Manual: Benchmarking Innovative Research Organisations in European Accession Countries. ISBN 92-894-7916-7European Commission, RTD info EUR 21238

3 Foci of a modern RTD organisation:- Knowledge generation- Knowledge diffusion- Knowledge utilization

SWOT-Factors:Different factors (categories) for the different foci- Internal factors (strengths and weaknesses)- External factors (opportunities and threats)- Negotiated factors

Data needs

input data (human resources and material and non-material investments)

output data (publications and citations are not enough!) information about framework conditions (organisational

framework conditions and systemic RTDI conditions) – they have a high influence (corridors of action and path dependencies; social and „cultural“ issues)

Internal Factors (1)

(1) Critical mass (size)

(2) Progressive management

(3) Good Human Resource management

(4) Output of a creative and innovative team

ad 1) Researchers (skill differentiation), support staff Infrastructure (scientific, generic technical, library facilities) R&D investments and access to external RTD infrastructures

ad 2) Availability of a strategy (user orientation, problem solving, how to

producte value added, resource development orientation etc.) Leadership, quality management, knowledge management, sales

management, IPR management, PR …

Internal Factors (2)

(3) Good Human Resource management

(4) Output of a creative and innovative team

ad 3) Training and staff development (vocational education, skills forecast,

promotion plans, employee performance, teambuilding measures etc.)

Diversity structure of staff (mentoring systems, gender and age structure, role models, foreign researchers, in- and outflow, trans- and interdisciplinarity of staff etc.)

Flexibility (communication culture, information infrastructure, team compositions etc. - an organisation is flexible if it has the capacity to adapt to new internal or external environments and to change in accordance with emerging organisational needs)

Internal Factors (3)

(4) Output of a creative and innovative team

Ad 4) contribution to product innovations (or product changes)? contribution to process innovations? contribution to service innovations? Patenting and licensing Bibliometrics (publication and citation counts) Research projects (collaborative projects coordinated and not

coordinated, joint projects with industry, contracted research etc.) Other outputs (e.g. research-education projects, PhD supervision,

quality of own scientific journal, events, post-graduate courses, spin-offs)

Negotiated Factors

Research mobility (e.g. in- and outgoing, affiliations abroad etc.)

Links with users (e.g. contracted research, other interactions with companies, market responsiveness)

Image building and ‚lobbying‘ (e.g. transfer of organisational standards, memberships, provison of expertise, PR, government committment towards the organisation)

Financial position (e.g. consistent funding or not, regular and returning clients, liquidity and cash flow management, financial management skills, assets for financial standing with banks, venture capital)

External Factors

Stage of transition of the S&T and economic system (e.g. RTDI policy, coherence among policies, attention paid to managerial aspects of RTDI, independence from political parties etc.)

Sectoral and national economy conditions (focus on food science) (e.g. capital market for financing of RTDI incl. FDI in RTDI, venture capital provision, promotion of long-term research programmes, demanding users and industrial conditions etc.)

Outputs

Evaluation methodology (incl. tailor-made qualitative and quantitative questionnaires)

SWOT based evaluation report (external evaluators) Action plan to improve the situation

(internal self-evaluation) (employee satisfaction evaluation) (auditing of the financial management) (assessment of technical infrastructure incl. IT)

SWOT results (page 1 of 3)

Strengths Weaknesses

Opportunities Threats

* No clear mission and strategy (yet)* Not enough critical mass in the current specialisations* Low number of students compared to the prevailing funding and promotion system* Relative lack of young staff* Low performing support staff* No accreditation of labs* Scarcely equipped and updated library

* Existing intrinsic value system towards scientific research* Increasing inter-disciplinary cooperation* A shared focus on food chemistry* Still sufficient number of researchers* Language skills and internationalisation of staff* Engaged and responsible self-governing management* Good team spirit

*Increasing future demand for food chemistry * UKIM’s intention to professionalise central management functions* More contracted research with industry through strategic partnership* More engagement in LLL* Development of joint study programmes with foreign partners

* Enlarged action space through European integration

* Danger of radically declining number of staff in near future* High insecurity about replacement of retiring personnel* Structurally caused inbreeding of personnel* Still limited management capacities at central levels* Low number of calls for research proposals at national and bilateral level* Low appreciation of RTD at UKIM and in Macedonia (ink. government)

Benchmarking

With whom to compare in order to assess if our data are good or not?

external benchmarking (best would be other IoCs) internal benchmarking (across time) functional benchmarking (analysis of functions and practices vis-a-vis

some accepted standards)

Due to the complexity, diversity and variability of RTO organisations and their environment, benchmarking produces rather signs of good practice than evidence of good performance!

Overall Assessment

1. Evaluation and benchmarking are powerful tools to generate evidence (versus myths), reflection (versus „doing business as usual), learning (what and how to improve), and an evidence-based starting point for interventions and change process

2. No size fits all, thus, many aspects have to be taken into account (see RECORD methodology). Different reference systems are needed for different levels.

3. It is seldomly used as policy tool, but rather bottom-up; shared responsibility might be powerful; independent external evaluators are recommended

4. It can be used for allocation of public money

5. If systematically applied, it will result in an improved system of innovation

6. A benchmark is not value-free, but requires „political“ legitimization.

Proposal for a joint Undertaking Establishment of a Regional RTDI Platform

Establishment as a (virtual) regional Centre of Competence „Owned“ jointly by ministries/agencies and specialised evaluation

institutes Supported by FP7 or SEE programm (higher start-up costs but low

running costs) Based on professional highest standards Provision of certified regional and international evaluators Provision of tools, templates and practices (e.g. ToR blueprints

etc.) Provision of tailor-made legal advice (e.g. national and European

public procurement laws) Provision of a necessary virtual infrastructure (e.g. access to ISI

Web of Knowledge, SCOPUS, Pascal; access to other information and data sources)

Functions of the Regional RTDI Platform (1)

1. Training of evaluators (setting of standards, certification) e.g. innovation surveys, input-output models, econometric models, cost

benefit analysis, control group approaches and quasi-experimentation, social network analysis, bibliometric analysis, patent analysis, foresight and technology assessment, qualitative methods like expert panels and peer reviews, focus groups, participatory evaluation, case study methodologies, logic chart approach etc.

project, programme, organisation and policy evaluations (ex-ante, interim, terminal and ex-post)

concept, design, process and impact evaluations

2. Training of policy-delivery systems (incl. agencies) training about best-suited evaluation designs for certain evaluation

questions information provision on the drafting of professional ToR for evaluation;

methods for technical and financial comparison and selection of offers seminars on national and European public procurement laws and their

implementation into practice

Functions of the Regional RTDI Platform (2)

3. Legal and Technical Advice inventory of good practices in the practical implementation of evaluation

procurement processes individual backstopping and advice for legal issues comparative study about public procurement rules and regulations in the

field of RTDI evaluation among member countries

4. Virtual Evaluation Infrastructure access to {expensive} data bases (such as ISI Web of Knowledge,

SCOPUS, patent statistics etc.) systematic provision of evaluation standards, evaluation reports,

handbooks and manuals, methodology briefs, templates and blueprints provision of an international evaluators database for programme,

organisation and policy evaluations continuous update of joint evaluators database for proposal evaluations

in certain fields of science

Functions of the Regional RTDI Platform (3)

5. Implementation of a series of programme evaluations (based on actual demand and variable geometry of the members of the RTDI Platform)

6. Benchmarking of RTD organisations (based on variable geometry) e.g. identification and assessment of potential Centres of Excellence e.g. benchmarking of RTD organisations in certain fields of science

(physics, ICT, agriculture, social science, humanities etc.) e.g. evaluation of agencies

7. Interface with other evaluation platforms (e.g. American Evaluation Association, DeGEval and others) e.g. through joint conferences e.g. through joint publications

Klaus Schuch

Zentrum for Soziale InnovationLinke Wienzeile 246

A - 1150 Vienna

Tel. ++43.1.4950442-32Fax. ++43.1.4950442-40

email: schuch@zsi.athttp://www.zsi.at

What do you think?