Dr. Sanjaya LallDr. Sanjaya Lall

Professor, Professor,

Developmental EconomicsDevelopmental Economics

Oxford University, Oxford University,

UKUK

• Former Senior Economist, World Bank

• Advisor, Technology Policy, Innovation,

Competitiveness, Industrial Technology,

Foreign Direct Investment, Trade, Monitoring

& Evaluation Systems

• Former Senior Research Officer, Institute of

Economics and Statistics, Oxford

Dr. Sanjaya LallDr. Sanjaya Lall

Monitoring and evaluation of innovation projects & programs

Sanjaya Lall

Need for monitoring and evaluation (M&E) All investment proposals need to be

evaluated if they compete for resources All projects and programs have to be

monitored for effective implementation, meeting deadlines and ensuring discipline in spending, purchasing and so on

All projects and programs must be evaluated during and after completion to ensure that direct and social objectives are achieved

M&E is exceptionally difficult for innovation projects & programs

Highly speculative and uncertain nature of innovation results: problem exists even within enterprises in allocating R&D budgets

But public projects face other problems: Public goods nature of many outputs Non-measurable nature of benefits of programs

to Create inter-firm & inter-industry linkages Strengthen linkages between industry and research

institutions Even more diffuse nature of stimulating

‘technology culture’, modernization etc.

Nature of M&E depends on nature of innovation project

Elements of innovation projects

Upgrade S&T

institutions

Stimulate enterprise

R&D

Strengthen linkages of TIs with industry

Improve IPR

regime/ implement

ation

Fund technology based

firms (VCs, etc.)

Two major components

Benefits to the economy Achievement of specific project

objectives

Inherently qualitative and diffuse nature of innovation benefits gives rise to large number of possible M&E measures

Economic benefits: three levels of analysis

Trends at the general economy-wide or industrial level Before and after project International benchmarks

Analysis of enterprises and institutions covered by project

Comparisons of enterprises within project with those outside it

Economic benefits: measures R&D spending Patents, publications Competitiveness (in technology products?) Labour productivity (or TFP) Spin-offs, ‘technopreneurs’ Employment (total or skilled, mfg or R&D?) International R&D collaboration Technology balance of payments Inward and outward FDI (technology

based)

Other possible measures, more industry specific…

Quality standards and reject rates Order to delivery times Customer satisfaction ratings Inter-firm collaboration Cluster effects: local purchases,

joint training and R&D, other links

Economic benefits: sources of data General industrial, trade and R&D

data within country Benchmarks from other countries Industrial census (with additional

data on technology) Questionnaires to firms &

institutions within and outside project

Interviews with firms and institutions in project

Economic benefits: methodology

Benchmarking over time or across countries

Econometric analysis of enterprises within & outside project

Case studies

Project objectives: three components (for implementing agencies)

Inputs Outputs Outcomes

Inputs…

Extent to which implementation meets targets on schedule, e.g.: Disbursement of project funds Purchases of equipment Construction of facilities Other milestones

Outputs, depending on agency…

Speed of processing patent applications Extent of metrology needs met,

conformance to international standards % of budget earned from industry

(contracts or research) Restructuring, international accreditation

etc. of laboratories No. of R&D contracts, commercialization,

no. of publications (impact), patents Number of technology loans, repayment

rates Venture capital funding: extent, success

Outcomes (broad economic impact)…

Part are already covered under analysis of economic benefits

More specific measures of performance by institutions and firms directly involved in project

Specific performance measures Enterprises

Growth and nature of R&D R&D management Access to technology financing Access to MSTQ services Efficiency of IPR regime Extent of interaction with technology

institutions, universities, etc. Export and productivity performance

R&D: Internal institutional culture, responsiveness to and links with industry, spin-offs

MSTQ: Increased efficiency of services, effects on productivity, competitiveness, FDI

VC: Stimulation of hi-tech start-ups, IPOs General: Improvement of R&D performance

and creation of ‘technology culture’, use of new technologies in production & exports, attraction of MNC R&D facilities and contracts, outward FDI, ‘cluster’ effects.

Specific performance measures: institutions

Who does M&E?

Project agencies provide information on project inputs and outputs

Data is collected by statistical & project agencies, questionnaires administered by independent researchers

Analysis carried out by independent researchers

Dr. Sanjaya LallDr. Sanjaya Lall

Professor, Professor,

Developmental EconomicsDevelopmental Economics

Oxford University, Oxford University,

UKUK