the world bank monitoring and evaluation of science, technology & innovation an international...
TRANSCRIPT
The World Bank
Monitoring and evaluation of Monitoring and evaluation of science, technology & innovationscience, technology & innovation
An International PerspectiveAn International Perspective
The World Bank
M&E a core element in strengthening M&E a core element in strengthening national S&T policiesnational S&T policies
AccountabilityAccountability: Demand from elected officials : Demand from elected officials and the public for accountability for the use of and the public for accountability for the use of resourcesresources
Policy-MakingPolicy-Making: Guidance on policy choices, : Guidance on policy choices, reforms and innovations, and knowledge of reforms and innovations, and knowledge of bottlenecks in national innovation systembottlenecks in national innovation system
Program ImprovementProgram Improvement: Information to track : Information to track implementation of S&T policies and to decide on implementation of S&T policies and to decide on improvementsimprovements
The World Bank
Key challenges to developing M&E Key challenges to developing M&E systems in Latin Americasystems in Latin America
Lack of Lack of transparencytransparency in results achieved in results achieved with public resources invested in R&Dwith public resources invested in R&D
FragmentationFragmentation in responsibilities for in responsibilities for collecting and reporting datacollecting and reporting data
Unreliable statisticsUnreliable statistics, notably in regard , notably in regard to private sector R&Dto private sector R&D
Low capacity to Low capacity to process and analyzeprocess and analyze datadata
The World Bank
Monitoring and evaluationMonitoring and evaluation
MonitoringMonitoring: continuous process of : continuous process of collecting and analyzing information to collecting and analyzing information to compare how well a project, program or compare how well a project, program or policy is being implemented against policy is being implemented against expected plans or resultsexpected plans or results
EvaluationEvaluation: systematic assessment of : systematic assessment of planned, on-going or completed planned, on-going or completed interventions to determine their relevance, interventions to determine their relevance, appropriateness, effectiveness, efficiency, appropriateness, effectiveness, efficiency, impact or sustainabilityimpact or sustainability
The World Bank
Responding to the demand for Responding to the demand for informationinformation
Developing theory of change and sector Developing theory of change and sector strategystrategy
Determining what and how to measureDetermining what and how to measure
Organizational arrangements for collecting Organizational arrangements for collecting informationinformation
Capacity to analyze and use collected Capacity to analyze and use collected informationinformation
The World Bank
Developing theory of change and sector Developing theory of change and sector strategy – determining what to monitorstrategy – determining what to monitor
Strategy implies the movement of a sector Strategy implies the movement of a sector from its present stage to a desirable but from its present stage to a desirable but uncertain future stage. Because the uncertain future stage. Because the country/sector has never been to this country/sector has never been to this future position, its intended pathway future position, its intended pathway involves a series of linked hypothesesinvolves a series of linked hypotheses
A M&E module is a method to document A M&E module is a method to document and test the assumptions inherent in the and test the assumptions inherent in the strategystrategy
The World Bank
Measuring the middle groundMeasuring the middle ground
Generally weak Generally weak articulation of the articulation of the mechanisms through mechanisms through which inputs and which inputs and activities are expected activities are expected to result in the desired to result in the desired impactimpact
Milestones and inter-Milestones and inter-mediate measures are mediate measures are important in S&T where important in S&T where final outcomes may final outcomes may take years to emerge take years to emerge InputsIn
terv
entio
n
Outcomes
Outputs
Goal(Impacts)
Res
ults
Activities
??
??
The World Bank
Transition to a knowledge- based
economy
Improved humancapital
Grants forinternationalcooperation
Cooperative research consortia
Insertion of researchers into industry
Sustainedgrowth
State of the art scientific
community
Improved private sector research
capacity
Enhanced public-private linkages and innovation
Coherent strategy for innovation
and human capital
Conducive innova-tive policies and
instruments
Continualmonitoring
and evaluation
Strengthened Chile’s national innovation system
Acquisitionof S&T
equipment
AdvancedHuman Capital
Excellencein
science
Causality treeCausality tree
Represents the best Represents the best guess as to an guess as to an appropriate course of appropriate course of action, given the best action, given the best available knowledgeavailable knowledge
Make relationships Make relationships explicit so they can explicit so they can be monitored, be monitored, validated and validated and managedmanaged
The World Bank
Indicators and incentivesIndicators and incentives
‘‘Indicators’ rather than ‘measures’Indicators’ rather than ‘measures’
Risk: “What you measure is what you get”Risk: “What you measure is what you get”
Academic research: risk of cementing an Academic research: risk of cementing an ‘ivory tower’ mentality by assigning too ‘ivory tower’ mentality by assigning too much importance to bibliometricsmuch importance to bibliometrics
Develop indicators for desirable behavior, Develop indicators for desirable behavior, e.g. collaboration, entrepreneurship, e.g. collaboration, entrepreneurship, mobilitymobility
The World Bank
Apply international standards and data manuals (Oslo Apply international standards and data manuals (Oslo and Frascati) to facilitate international benchmarkingand Frascati) to facilitate international benchmarking
Determining what and how to measureDetermining what and how to measure
Types of indicatorsTypes of indicators Direct measures, e.g. annual PhD Direct measures, e.g. annual PhD
graduation and bibliometricsgraduation and bibliometrics Proxies, e.g. average country citation Proxies, e.g. average country citation
impact factorimpact factor Perception variables, e.g. transparency of Perception variables, e.g. transparency of
funding allocation mechanismsfunding allocation mechanisms
The World Bank
Challenge of measuring private sector Challenge of measuring private sector innovationinnovation
Understanding determinants for turning Understanding determinants for turning science into business essential to ensuring science into business essential to ensuring that investments in S&T translated into that investments in S&T translated into increased competitiveness and growthincreased competitiveness and growth
Company innovation surveys, e.g. Chile Company innovation surveys, e.g. Chile (1994-95; 1997-98; 2000-01)(1994-95; 1997-98; 2000-01)
ChallengeChallenge: To be useful, content needs to : To be useful, content needs to be agreed between several sectors and be agreed between several sectors and agencies, e.g. Ministry of Education, agencies, e.g. Ministry of Education, Ministry of Economy and National Ministry of Economy and National Statistical AgencyStatistical Agency
The World Bank
Marginal effect of company characteristics on Marginal effect of company characteristics on patents, product innovation and process innovationpatents, product innovation and process innovation
Patents
Product innovation
Process innovation
Firm Characteristics Export active **0.34 0.21 0.08 Firm age (ln age) 0.02 0.00 0.02 Firm size (ln size) 0.07 0.10 0.01
Collaboration on innovation with Universities **0.35 **0.30 0.20 Consultants -0.02 0.12 0.16 Suppliers **0.09 **0.32 **0.35 Other firms 0.01 -0.02 0.06
Absorptive capacity R&D department **0.06 **0.19 0.12
Human capital Investment in innovation education **0.09 **0.27 **0.23
Observations 4570 4570 4570 R2 0.23 0.30 0.37 ** Significant at 5% level
InterpretationInterpretation: : Collaboration with universities increases the probability 35 percent of Collaboration with universities increases the probability 35 percent of being involved in patenting, keeping all other factors constantbeing involved in patenting, keeping all other factors constant
The World Bank
Organizational arrangements for Organizational arrangements for collecting informationcollecting information
Assigning responsibilities: Clear Assigning responsibilities: Clear agreement on who does what, when and agreement on who does what, when and howhow
Consider the cost of the M&E system, e.g. Consider the cost of the M&E system, e.g. build on existing reporting structuresbuild on existing reporting structures
Providing an incentive to collect and report Providing an incentive to collect and report valid data by ensuring that requested valid data by ensuring that requested information is useful for decision-making information is useful for decision-making at both central and decentralized levelat both central and decentralized level
Avoiding heavy bureaucratic burdens on Avoiding heavy bureaucratic burdens on researchers taking time away from core researchers taking time away from core activitiesactivities
The World Bank
Systems of performance reportingSystems of performance reporting
Use standardized reporting to increase Use standardized reporting to increase transparency, ensure comparability and transparency, ensure comparability and reduce administrative loadreduce administrative load
Essential to ensure collection of Essential to ensure collection of longitudinal data rather than a number of longitudinal data rather than a number of cross-sectionscross-sections
The World Bank
National capacity for analyzing dataNational capacity for analyzing data
Unit with skills and capacity to process and Unit with skills and capacity to process and analyze data, undertaking studies, and analyze data, undertaking studies, and suggesting policy implicationssuggesting policy implications
Chile: National Observatory for S&TChile: National Observatory for S&T
Technical rigor in designing evaluations Technical rigor in designing evaluations (process, impact): challenge of using (process, impact): challenge of using quasi-experimental designs in small quasi-experimental designs in small innovation systems using competitive innovation systems using competitive resource allocation mechanismsresource allocation mechanisms
The World Bank
8
9
10
11
12
13
14
15
1 2 3 4 5 6 7 8
Year
Dependent
vari
able
Impact evaluation: Impact evaluation: Difference-in-difference designDifference-in-difference design
T1
T2
ΔYC
ΔYTΔYT- ΔYC
Treatment g
roup
Competitively selected
Control group
Cost-effectiveness: Evaluate against best alternative Cost-effectiveness: Evaluate against best alternative investment rather than the no treatment scenarioinvestment rather than the no treatment scenario
The World Bank
8
9
10
11
12
13
14
15
1 2 3 4 5 6 7 8
Year
Dependent
vari
able
Impact evaluation: Impact evaluation: Challenge of selection biasChallenge of selection bias
T1
T2
Treatment g
roup
Selection bias:Controlling for past trends
Control group ΔYC
ΔYT
ΔYT- ΔYC
Ex-ante ΔYT
A baseline is a line, not a pointA baseline is a line, not a point
The World Bank
Summing upSumming up
M&E essential for developing and maintaining M&E essential for developing and maintaining effective ST&I policeseffective ST&I polices
Need to ground M&E in national strategy for ST&INeed to ground M&E in national strategy for ST&I
Indicators shape incentivesIndicators shape incentives
Base M&E systems on international data manualsBase M&E systems on international data manuals
Place increased emphasis on understanding the Place increased emphasis on understanding the determinants for turning science into businessdeterminants for turning science into business
Develop capacity to use M&E for learning and Develop capacity to use M&E for learning and improving ST&I policiesimproving ST&I policies
