Measuring the Impacts of Science: Institutional Challenges in a Transforming S&T System

Measuring the Impacts of Science

Montreal

17-18 June 2004

© 2004 MPienaar

Overview

• Approach• The National Research Foundation • Context : transforming system• Unique mission of an intermediary institution• Emergence of performance indicators • The NRF approach to key performance indicators • Challenges

© 2004 MPienaar

Approach

A tour through the S&T landscape of SA

and the route the NRF takes to measure

the impact of science

© 2004 MPienaar

Mandate of the NRF

Support and promote research

through funding, human resource development and the provision of the necessary research facilities

in order to facilitate

the creation of knowledge, innovation and development in all fields of science and technology, including indigenous knowledge

and thereby to

contribute to the improvement of the quality of life of all the people of the Republic.

Governance of NRF

Minister of Science and Technology

NRF Board

President and CEO

CorporateResearch & Innovation Support & Advancement

(RISA)

National Facilities

NRF: RISA

© 2004 MPienaar

NRF: Research and Innovation Support and Advancement

•Information and

Strategic Advice

• Evaluation

• IT

Knowledge Management

•Research Capacity

Development

•Focus Area Programmes

•Internl. Science Liaison

Building Knowledge Base

Support Functions

•CGA

•Finance

•Personnel

•THRIP (dti)•Innovation Fund (DST)•Skills Development (DoL)•Other (e.g. DEAT)

Mission Driven

•Science for Society•Decision Makers

•Youth (Educators

and Learners)

SAASTA

National Facilities

© 2004 MPienaar

Context: transforming system

Measuring the impacts of science and science support of an intermediary body in a transforming system has to account for at least the following factors:

• Historical chronology and legacy of S&T system• Status of National System of Innovation• Challenges in NSI and strategies

© 2004 MPienaar

Transforming S&T system: Chronology

Pre-1990: S&T system – apartheid

1992: IDRC review of S&T system

1994: First democratic government

1996: White paper on S&T

1997: SETI reviews

2002: R&D strategy

2004: Dept of Science and Technology

© 2004 MPienaar

Transforming S&T system: legacy

• First democratic government (1994) inherited a fairly strong S&T

• Since 1994: activate stagnating economy to

– improve the quality of life of all, (esp. the deprived)

– broaden the capacity base

– become part of the international scene

• Align new S&T policy with overarching commitments

• Develop a competent S&T system.

© 2004 MPienaar

Status of the National System of Innovation (1)

• HE institutions: 24 ±15 700– Postgraduate students ±41 600

• Government labs (SCs): 9 5 900• National facilities: 5 360• Business sector: ±400 6 200• Researchers / 1000 of labour force: 1.9• GERD as % of GDP: 0.76%• Productivity: ISI journals: 3 592• Patents abroad / million of pop: 2.5

© 2004 MPienaar

Status of the NSI (2):National R&D survey (2001/2002)

• Business sector performs 53.8% of all R&D undertaken

• HE sector: 25.3% of all R&D

• Government laboratories: 19.9% of all R&D

• Basic research: ± 27% of expenditure

• African, Coloured or Indian: 36.5% of R&D

workforce

• Women researchers: 35.3% of R&D workforce

© 2004 MPienaar

Dualistic system

Developing - Developed

Transforming S&T system: Current challenge

• Erosion of S&T base requires clear interventions to reverse this and ensure S&T plays a key role in and for the quality of lives of all our people

• Wealth creation in modern economies is dependent on innovation, research and development. South Africa must grow its investment in this domain and strategically target resources to meet its objectives

Source: Dr Rob Adam, DG: DST

• South Africa’s National R&D Strategy

• The Microeconomic Reform Strategy and dti’s Vision 2014

• Accelerating Growth and Development: The contribution of an Integrated Manufacturing Strategy

• Human Resource Development Strategy

• Strategic Plan for South African Agriculture

• National Water Resource Strategy

• Government’s ICT Initiative

Transforming the system:Transforming the system:National strategies for economic growth, job National strategies for economic growth, job

creation and social upliftmentcreation and social upliftment

Transforming S&T system: NRDS

Quality of life Economic GrowthNRDS goals

Indicators

SET human resources:•Correct demographics•Science & maths: school level•Centres of excellence

Innovation:New missions & platforms: Biotech, ICT, Poverty reduction, etc.

Effective S&T government system:

•Alignment with policy objectives•IPR

© 2004 MPienaar

NSI: Policy and strategy

National needs

Intermediary institution (NRF)

Control: Organisational performance

Influence: Stakeholder performance

Instrument: Funding

Performers of science

HEIs and SCs

NS

I ob

ject

ives

NRF position as intermediary body (1)

© 2004 MPienaar

NRF as intermediary body (2)

The challenge is to design a performance measurement system that will articulate and translate impacts between different levels

Performers

Systemic: NSI

Institutional policy instruments

© 2004 MPienaar

NRF as more than a intermediary body

National S&T steering:Policy, business plan & KPI reporting, reviews

NRF

Nee

ds o

f an

d tr

ansf

orm

-at

ion

in c

lient

sec

tor

Resou

rces

:

Slow

gro

wth,

nati

onal

prio

rities

Ow

n mission &

divergent functions

Outcome & impact measurement system

Global trends:

Strategic science,

networks

© 2004 MPienaar

Evolution of performance measurement

• 1996 White paper on S&T

• 1997 SETI reviews

• 1998: DST assessed Balanced Scorecard (BC) approach as a framework for KPI reporting in SCs

• Consultation and participation in development

• SCs adapted scorecards to fit their specific needs

• 2000/01: SCs submitted KPI reports

Annual submission to Minister© 2004 MPienaar

Generic KPIs for science councils

• Stakeholder/customer perspective

• Financial and investment perspective

• Organisational perspective

• Innovation and learning perspective

• Human resources and transformation

© 2004 MPienaar

Indicators of Science, Technology and Innovation – NRF Targets (1)

Indicator Present Target NRF intervention

R&D staff per million of population

RSA 480

Korea 2770

Australia 4840

2000 by 2012 Significant increase in Post-docs and research support staff

Patents abroad per million of population

RSA 2.5

Korea 779

Australia 75

25 by 2012 Develop NRF policy regarding IPR of publicly-funded research

ISI indexed articles per million of population

RSA 82

Korea 259

Australia 1072

220 by 2008 Alignment with DoE strategy on research output subsidy

Indicators of Science, Technology and Innovation – NRF Targets (2)

Indicator Present Target NRF intervention

HE enrolment in SET

RSA 18%

Korea 34%

Australia 32%

30% by 2012 Create demand pull through significant increase in bursary values

PhDs per million of population

RSA 18.6

Korea 118

Australia 170

45 by 2012 Treble value of doctoral bursaries

Higher Education R&D expenditure as % of GDP

RSA 0,07%

OECD Avg. 0.38%

Australia 0.44%

0.21 % by 2007 Creation and investment in Centres of Excellence

NRF: Core missions and priorities

Core missions• Human resource development

• Knowledge generation in

prioritised areas

• Utilisation of research results,

technology and innovation

• High quality infrastructure

© 2004 MPienaar

Strategic priorities

Redress &

equity

Adherence to quality

Internationalisation

Positioning of N

RF

Focus on A

frica

NRF: Programmes & Functions

RISA • Development Programmes

• Scarce Skills (DoL)

• STAC

• Focus Area Programmes

• CoE

• THRIP

• Innovation Fund

© 2004 MPienaar

Across all programmes & functions:

• Same core missions & strategic priorities

• Different interventions & KPIs

National Research Facilities• SAAO

• HartRAO

• HMO

• SAIAB

• SAEON

• iThemba LABS

Use of performance reports by NRF

– Creation of integrated organisation

– Compliance Public Finance Management Act (1999)

– Control, management

– Influence i.e. shift in behaviour, relationships, actions of partners

– Legitimisation; political acceptance• Causality, co-producers, unintended results, etc.

© 2004 MPienaar

Challenges for NRF

Terminology and definitions

Distinguish different levels of complexity

Operationalisation i.e. how to “count” concepts

Multiple information inputs: reliability & validity

Copying more advanced science systems: own developmental requirements?

© 2004 MPienaar

Challenges (2)

KPIs serves multiple and (potentially) mutually exclusive functions:

External: Political legitimisation of science

Internal: Institutional management tool

Conclusion

The measurement of the impact of science is a scientific endeavour to see

things narrowly and deeply.

The application in government is to see things broadly and politically.

Thank you….

….also, in anticipation, for your advice!

Acknowledgements • Dr RM Adam: Director- General, DST• Dr G von Gruenewaldt, Vice-President. NRF• Dr HC Marais: Head Secretariat NACI

www.nrf.ac.za

© 2004 MPienaar