Quantifying the Impact of Social Science Development Research: Is It Possible?Kunal SenIDPM and BWPI, University of Manchester

Based on paper: Literature Review on Rates of Return to Research, available on DFID R4D website.

Quantifying the impact of research: the rate of return to research Similar to any other investment by the public sector, research is expected

to yield benefits that are in excess of the costs of funding research. The rate of return to research is one important way that net benefits to

funding research can be measured. To calculate the rate of return to research, the present value of the

current and future benefits of the research is compared to the total costs of the research, and an internal rate of return is calculated to equalise the revenue stream with the cost outlays.

This internal rate of return is the rate of return to research. The higher the rate of return to research, the higher is the expected net

payoffs from research, and the stronger case for investing in research as compared to other types of public investment. Or for investing in one type of research versus another.

TWO QUESTIONS

WHAT DO WE KNOW ABOUT THE RATE OF RETURN TO DIFFERENT TYPES OF SOCIAL SCIENCE DEVELOPMENT RESEARCH?

TO WHAT EXTENT IS IT POSSIBLE TO CALCULATE RATES OF RETURN TO DIFFERENT TYPES OF DEVELOPMENT RESEARCH?

The Causal Chain from Research to Impact

1. Did the research influence policy thinking/decisions/processes (the attribution problem)

2. Did the policy intervention/change/reform lead to the observed outcome (the identification problem)

3. Can the benefits of outcome(s) be quantified? (the measurement problem)

Contextual Factors

Figure 2. The Results Chain for Different Types of Research

Research Programmes/Centres

The Attribution Problem

Policy Change/ Intervention/Reform

The Identification Problem

Outcomes-Tangible-Intangible

The Measurement Problem

Impact

Communication Of Research Findings

The Attribution Problem

The attribution problem can be broken down to the following components:

a) how well defined is the set of research users?

b) the counter-factual: will the policy change have occurred without the research taking place?

c) how important are contextual factors and exogenous events in influencing policy, independent of the research?

The Identification Problem

Since developmental outcomes may occur due to many reasons, and policy interventions is one possible cause of such outcomes among many others, it is often difficult to precisely identify whether the policy intervention can be causally related to the outcome in question.

There are three different aspects to the identification problem: a) selection bias; b) omitted variable bias; c) Reverse causality.

The Measurement Problem

An important requirement in the application of the rate of return approach is that all benefits, past, present and future, can be quantified and expressed in the same unit of value.

This leads to five problems in the measurement of these benefits: a) valuing multiple outputs; b) valuing intangible outcomes; c) time-scale of measurement; d) the degree of uncertainty on the size of the impact; e) measuring effects, where there are macro-changes or strong

spillover effects.

Figure 3. Differential Paybacks to Two Projects in Health Research

Figure 4. Rates of Return to Research in Different Time Dimensions

Time

Cumulative Net Benefit

Project 2

Project 1

Figure 5. The Degree of Uncertainty associated with Returns to Research Type of Research 1 Type of Research 2

Rate of return to research

Number of Projects

Rate of return to research

Number of Projects

20% 20%

Methodologies to quantify the impact of policy change/intervention

Simulation models Regression based methods Case studies Randomised control trials

The Results Chain for Different Types of Research

Type of Research

Attribution Identification Measurement

Agriculture Relatively straightforward

Relatively easy. Relatively easy

Health Relatively straightforward

Easier for clinical trials; less straightforward for multidimensional measures of health outcomes

Relatively easy

Social Policy Complex, depends on contextual factors

Moderate degree of difficulty.

Moderately difficult

Economic Policy

Complex, depends on contextual factors

Moderate degree of difficulty.

Moderately difficult

The Results Chain for Different Types of Research – contd.

Type of Research

Attribution Identification Measurement

Infrastructure Complex, depends on contextual factors

Moderate degree of difficulty –difficult to establish causality

Moderately difficult.

Governance Very complex, depends on contextual factors

High degree of difficulty.

Very difficult.

Climate Change

Very complex, depends on contextual factors

High degree of difficulty.

Very difficul

What do we know about the rates of return to different types of social science research? Usable rates of return to research (RORs) exist

– agriculture and health research Proxy rates of return do not exist, but there are

credible ways to calculate RORs – infrastructure research, economic and social policy research

Proxy Rates of return do not exist, and there are no credible ways to calculate RORs - governance research, climate change research.

So can we calculate the rates of return to different types of social science research? A non-starter for research which lead to

intangible outcomes, where the time-scale of outcomes is very long and where the identification problem is particularly challenging– governance and climate change research.

Possible for economic and social policy research – but the informational requirements for doing so are very high.

Already exists for agriculture and health research.

How to improve our ability to measure the impact of research In general, there is a need for investing in improved methodologies

that tackle the identification problem (but not necessarily a focus on randomised control trials only).

Investing in monitoring and evaluation processes at the start of the research programme to address the attribution problem – creating baselines and using case-studies to track the impact of research.

Looking at best practice on how to address the attribution problem – e.g. Fred Carden’s work in IDRC.

A limited use of methodologies such as willingness to pay where there are clear tangible benefits of research to address the measurement problem.