Fred Gault

UNU-MERIT and TUT-IERIWorkshop on evaluation of STI policies

Mexico City, 19 June, 2014

Public policy is a process driven by government(s) influenced by interest groups◦ Possible objectives or desired impacts

Jobs and growth

Equity

Inclusiveness

Wellbeing

…

For it to work, policy has to be implemented using policy instruments

Evaluation of implemented policy is part of policy learning and it supports public discourse

A part of the process deals with science and technology (S&T) policy and innovation policy◦ The two are different

Knowledge creation ◦ Formal production of knowledge (R&D)◦ Human resource production (HQP)◦ Transfer of knowledge

Codified to other sectors - Commercialization

Tacit through mobility of HQP

Policy can support knowledge creation◦ Grants, contracts (direct)◦ Tax benefits (indirect)◦ Government budget appropriations or outlays for R&D (GBAORD)◦ Public sector performance of knowledge generation

Different implementation of policy in different sectors of the economy◦ Emphasis on public institutions (HE or Government labs)

Driven by the ministries of research and education, perhaps with finance and public works (to deliver the policy)

OECD(2002)

What is innovation?

Putting a new or significantly improved product (good

or service) on the market or finding a better way of

getting it there (transformation and delivery,

organization and business practices, or market

development or new markets). Paraphrase of Oslo

Manual para 146, 150.

Innovation only happens in a firm (could be a sole

proprietorship)

OECD/Eurostat (2005)

Policy can act on ◦ Firms, or on◦ Framework conditions

Shorter term (one mandate)

Competition policy

Trade policy

Intellectual property

Sector specific regulations (ICT)

Transparency

Corruption policy

Public innovation platforms

Longer term (greater than a mandate) Health

Education

Government

Building evaluation into policy

implementation

◦ Not every organization wants to do this

What does the government want to measure?

Impact? (economic and social?) Growth? Why do

it?

Example - the Government of Ontario and its

question put to the Council of Canadian

Academies (CCA)

The question

◦ How can the actual and potential outcomes and impacts of Ontario

government spending on innovation and scientific activities be

measured, including, but not limited to the effects on GDP in

Ontario, generation and transfer of knowledge; creation of new

ventures; and access to seed, development, and growth capital?

The CCA panel convened to answer the question examined six classes of

support programmes and likely impacts

Using existing data and the examination provides a means of setting

priorities

CCA (2013) See also www.nasonline.org/about-nas/mission/

Impact

Program Type

Knowledge

Generation

New

Ventures

Access to

Capital

Employ-

ment

GDP/

Output

Taxes Social

Direct Academic

Support

High Low n/a Mod Low Low Mod

Public & Non-

Profit

Research

Organizations

High Low n/a Mod Low Low Mod

Innovation

Intermediaries

Low Mod Mod Mod Low Low Low

Direct Business

Support

Moderate High High High Mod Mod Mod

Indirect Business

Support

Moderate Mod Mod High Mod Mod Mod

Public

Procurement

Low Low n/a High High High High

Policy intervention can target many or few areas

Broad examples of policy components are

◦ Markets

◦ People

◦ Innovation activities

◦ Public Institutions

◦ International engagement

Gault (2010)

1. Markets

1.1 Brand recognition

1.2 Lead Market

1.3 Competitive engagement

1.4 Financial Services

2. People

2.1 Labour force

2.2 Demographics and Demand for Innovation

2.3 Migration

3. Innovation Activities

3.1 Technology and Practices

3.2 User Innovation

3.3 User-driven Innovation

3.3 Open Innovation

3.4 Demand-driven innovation

4. Public Institutions

4.1 Infrastructure

4.2 Procurement

4.3 Priority Setting

4.4 Standard Setting

4.5 Public Finance

4.6 Government Direct Support

4.7 Education, training and research

4.8 Health

4.9 Monitoring and Evaluation

5. International engagement

5.1 Big science

5.2 International co-operation and development

Governance

◦ One department (Treasury for tax credits)?

◦ Whole of government? Who chairs?

◦ Co-ordination? Authority to co-ordinate?

◦ Leadership? At what level

Governance is a domain for evaluation

See www.innovationpolicyplatform.org/

◦ Information on policies and country activities, a joint OECD – World

Bank service

OECD (2012)

Evaluation approaches require different amounts of

data – in order of increasing demand

◦ Case Studies

◦ Indicators-based frameworks, scorecards and benchmarking*

◦ Econometric approaches

◦ Regression discontinuity design, matching estimation,

difference-in-difference estimation

◦ Random field experiments

CCA(2013: 23)

* See Hollanders and Janz on scoreboards, in Gault (2013)

Level at which to evaluate

◦ Co-ordination level

◦ Sub-system

◦ Project

Support for evaluation (Authority)

Data

◦ Availability, quality, timeliness, relevance…

◦ Survey, administrative, case study, linked data sets, ….

Expansion of the subject (Gault 2012, 2013)

◦ Public sector innovation

◦ User innovation (Consumers)

◦ Social Innovation

◦ Constrained innovation ‘X innovation for Y’

X = inclusive, pro poor, frugal, ….

Y = development, sustainable development, wellbeing, growth, ….

Introducing more time scales.

Consider: Inclusive Innovation for Sustainable Development

T1: Oslo innovation looks at the last three years

T2: Inclusive requires measurement after the innovation, medium term

T3: Sustainable development requires measurement some time after the

innovation

Then comes the evaluation

With three time scales, how is the evaluation designed?

The policy is to support inclusive innovation for sustainable development◦ Instruments may include

Support for inclusive employment policies

Training of staff to accept the excluded group

Support for locating the firm in a an area populated by the excluded group….

◦ But only if there is innovation which

Includes the excluded group (how is that measured?)

◦ And

Is sustainable (how and when is that measured?)

CCA (2013), Innovation Impacts: Measurement and Assessment: The Expert Panel on the Socio-Economic Impacts on Innovation Investments, Ottawa: Council of Canadian Academies. www.scienceadvice.ca

Gault, Fred (2010), Innovation Strategies for a Global Economy, Development, Implementation, Measurement and Management, Cheltenham, UK and Northampton, MA, USA: Edward Elgar and Ottawa: IDRC.

http://www.idrc.ca/EN/Resources/Publications/Pages/IDRCBookDetails.aspx?PublicationID=45

Gault, Fred (2012), ‘User innovation and the market’, Science and Public Policy, 39, 118–128.

Gault, Fred (ed.) (2013), Handbook of Innovation Indicators and Measurement, Cheltenham, UK and

Northampton, MA, USA: Edward Elgar

OECD (2002), Frascati Manual: Proposed Standard Practice for Surveys on Research and Development, Paris: OECD.

OECD (2012), Meeting global challenges through better governance: International co-operation in science, technology and innovation, Paris: OECD. www.oecd.org/sti/stpolicy/governance

OECD/Eurostat (2005), Oslo Manual, Guidelines for Collecting and Interpreting Innovation Data, Paris: OECD.