Aid Effectiveness Literature: The Sad Results of 40 years of Research

Hristos DoucouliagosMartin Paldam

Journal of Economic Surveys (2009) Vol. 23, No. 3, pp. 433–461

Intro

• Aid effectiveness literature (AEL): investigates the effect of aid on growth started in the 1960’s comprised 97 econometric studies by the end of 2004

• Doucouliagos & Paldam (hence D&P): run a meta-analysis on the AEL in order to assess whether aid has been effective or not (the answer is no!)

Table of contents

I. The « aid paradox »

II. The meta-analysis approach

III. Meta-analysis and the AEL: results

IV. An alternative to the AEL

V. Conclusion, Limits and Propositions

I. The « aid paradox »

. Criterion to judge whether aid is beneficial or not: its effectiveness. What does aid effectiveness mean?the effect of aid h on growth g is significantly positive:

μ = ∂g/∂h>0 and significantthe whole question is about the sign and significance of μ in a

empirical model of the form:git =μhit +uit

What is the aid paradox?

. 74% of the estimated μ in the AEL is positive

. Yet we have what D&P call the « zero correlation result »:simple data from WDI over the 1960-2000 period and covering 156

LDCs on growth and aid do not show any correlation between these two variables

1550 4-year averaged observations : this should be enough to find any significant correlation if it existed

zero correlation without imposing a sophisticated structure on the data

Illustration: raw data

Two other evidences of the paradox

• E1:Aid agencies aim at social rates of return of approximately 10% in feasibility studies of their projects. If this is realized, an aid share of 1% (of GDP) should result in a growth rate of 0.1%.

half of all projects succeed, so we expect a growth effect between 0.05 and 0.1.

the average aid share is about 7.5%, so it follows that aid should contribute somewhere between 0.4 and 0.8 percentage points to the growth rate

average LDC growth rate is about 1.6%, aid should explain between 25% and 50% of the growth in the average developing country

These 25 to 50% should be easy to identify

• E2:Both aid agencies and recipients have now had about 40 years of experience

estimate of the learning by doing typically find orders of magnitude of 1%–2% per year (Barro and Sala-i-Martin, 2004, pp. 212–220).

over 40 years: increase in aid effectiveness of roughly 50% or more We should find a clearly visible and positive trend in empirical estimates

of μ=μ(t) but simple regressions show on the contrary a significantly negative trend instead

Illustration: simple regression

The 4 reasons of the paradox

• 1st reason: The ‘why would they’ argumentAid that has continued for at least 40 years. So it must do at least some

of what it seeks. Why else would it continue? average aid share of the donors is actually quite small (only about

0.3% of donor GDP)aid fatigue: this share has even decreased a little in the last decade,

due to donors’ dissatisfaction with this small or nonexistent aid effect

at least some aid is given for non-humanitarian reasons (eg: commercial and security interests)

• 2nd reason: The micro evidenceCassen (1986, 1994): classic survey of aid programs’ evaluation about

50% of all development projects work, and very few of the remaining projects cause harm, even if they fail

macro level ineffectiveness vs micro level effectiveness: micro–macro paradox (Mosley, 1986)

fungible aid: a part of the projects financed by the donor would have been implemented by the recipient anyhow

consequently, the marginal project caused by the aid may differ from the project financed by the aid (opportunity cost reasonment)

• 3rd reason: Standard growth theoryAid projects consists most of the time in investments aid should thus promote growth though capital and human capital

accumulation (the main explanation of growth in standard growth theory)

• 4th reason: Standard macro theoryAid leads to a balance-of-payments improvement and to public spending‘activity effect’ that should run into the recipent economy (ie Keynesian

multiplier)even classical macroeconomics cannot explain why this effect would be

completely crowded out through the mechanism of Ricardian equivalence because development aid contains a gift component, which does not

have to be repaid: EDA vs ODA

• Additional reasons for the paradox. Maybe we should consider EDA rather than ODA measures of aid:- ODA: unilateral transfers with a gift element above a threshold of 25%- EDA: unilateral transfer weighted by its gift component (Chang et al.

(1998))but simple aid/growth regression with EDA do not show either any aid

effectiveness (D&P, 2008). We should weight aid share by population aid share h=H/GDPr (with H the amount of aid and GDPr the GDP of the

recipient country)traditionally, aid share is not weighted by the size of the population and

regressions thus give the same weight to China and Mauritius in the aid/growth regression

whereas h displays huge heterogeneity in population size (larger countries have a smaller h)

but even after this consideration, aid remains ineffective

Why a meta-analysis of the AEL?

• After 40 years of research, we are still left with this “aid paradox”: the effect of aid on growth remains unclear

The meta-analysis of the AEL is aimed at solving this paradox by analyzing the AEL rather than the AE itself: the meta-analysis is

an analysis of the μ previously estimated in the literature, it is not about regressing h on g once again

II. The meta-analysis approach

• Definition: meta-analysis is a quantitative study of a research process

tool to detect whether there is a bias in the research process that leads to overestimated and too significant μ

3 questions that the meta-analysis answers about the AEL research process

• Q1: Do the estimates in the AEL converge to something we might term ‘truth’? yes: the ineffectiveness of aid

• Q2: Can we identify the main innovations, which cause or prevent convergence? yes: three categories of model in the AEL that have all failed to prove the effectiveness of aid

• Q3: Are there biases along the way in uncovering ‘truth’ about aid effectiveness? yes: 3 “perennial problems” that biases the research process (data mining, priors, and incentives)

1st source of bias: data mining

Remark: here ‘bias’ is not used in an econometric sense • Data in macroeconomics is rather limited (in the sense that we

cannot do not find new countries to study everyday)thus, researchers in the AEL use more or less the same dataset over and

over they are “fishing in the common pool of available degrees of freedom”

• Why is data mining a common pool problem in the AEL literature?• Research process: mixture of innovation (ie new research discovery)

and independent replication by other authors seeking to confirm earlier innovations independent replication: it is the replication of a previous result on a different dataset thus in the AEL (on the overall since new data is obviously generated year after year), we

have dependent replications instead of independent replications

Results cannot be confirmed reliably

• What does it mean econometrically?• A few facts on the AEL:

• 97 papers• 1113 estimates• around 1000 observations (4 years averaged) sum of the sample sizes of the 97 papers =

30.516

• 1113 estimates for 1000 observations!Mining ratio = sum sample sizes/data available= 30. A mining ratio of 30 means that each observation of g and h (in a

country i at time t) has served on average 30 timesthis invalidates the interpretation of the t-test in assessing whether an

estimate is significantresearchers tend to believe it is significant while it is no. Another interpretation of the mining ratio is that it increases the

probability of type I error: accepting a result as significant while it is not

2nd source of bias: priors and incentives

• The stopping rule: what determines the results published by researches The stopping rules itself is influenced by the research community’s priors

and incentives

• Explanation (p.435):• "We all know from introspection that when we study an empirical question,

we analyze the data till we are satisfied with the result. Reported results are thus the product of a stopping rule. »

• "We all want to believe that we stop when we have reached some approximation to the truth. However, what we believe to be the approximate truth is influenced by our priors. Also, interests and institutions influence the stopping rule."

5 priors or incentives

5 priors of incentives

• Polishing• Researchers and journals have incentives to show statistically significant

results• Causes researchers to polishing the data (through complicated econometric

tools) until they find a significant effect of aid on growth while this effect is not in fact not significant

• Ideology• 2 ideologies: Marxist/leftwing vs libertarianMarxist: see aid as a form of neo colonialism and believe it hampers groth Libertarian: note that aid is often given to public sectors in the LDC, it would

thus hampers growth through by promoting government expansion Downard bias on the estimated μ

• Goodness• Helping the poor is generally considered to be good in a hard and often

unequal world. So it is a tragedy if it does not work."Reluctance hypothesis" at the micro-level through individual researcher’s

stopping rules: researchers are reluctant to publish negative estimate of μ

5 priors of incentives

• Institutional interests• A dense net of links exists from the aid industry to development

researchersresearchers either work (at least 35%)work for or are financed by the

aid industry another source of "reluctance » at the micro level• Reluctance too at the macro- level through the publication processsome journals receive grants, and may not like to embarrass their

benefactors

• CONCLUSION:• Unbalanced priors and interests

On the overall, they cause to overestimate the effect of aid on growth (and its significance)

Meta-analytic methods

• Combine conventional statistical methods and other unique to meta-analysis

• Meta-significance test (MST) & precision-effect test (PET): test for the existence of a genuine empirical effect

ie. test for the significance of μ

• Funnel asymmetry test: identifies the existence of a publication selection or “reluctance”ie test for the (positive) biasedness of μ examine funnel plots and the distribution of the estimated coefficients as a

function of sample size μ = μ(N)

FAT serves to detect convergence in the AEL process (do results about μ converge something that differs from 0?), asymmetries (reported estimated μ are not symmetrically distributed around the true value of μ when the research process is biased) as well as polishing (if the estimated μ are bigger when results are easier to polish, ie when the SE increases)

Funnel plot

III. Meta-analysis and the AEL: Results

• The authors record 97 AEL papers • Historically, they can be divided into three main families

Statistics of Reported Estimates in the AEL• The authors use a large pannel of studies for their meta-analyses

• Best-set is the regression estimate preferred by the author of the paper

• All-set includes all of the reported regression estimates

III. Meta-analysis and the AEL: Results

The literature process

• Figure 3: Significant rise in the production of empirical estimates of the AEL (1968-2004)• 1. Wave of family A models (savings and then investment models)• 2. Family B estimates• 3. Since 1995 family CNB: data mining is a process that eats degrees of freedom

Production over time of papers in the AEL

Model A: Aid and accumulationDoucouliagos & Paldam, 2006

• Data• 66 studies

• Model• The first AEL models focus on the impact of aid on savings and

investments (Harrod-Domar models)

• Savings rates and balance of payment: key constraints for growth• Critics:

• Griffin & Enos, 1970, Weisskopf, 1972, Fongibility of aid: the marginal activity generated by aid does not lead to increased accumulation

• Boones, 1996: aid leads to an increase in public consumption only

• Governments’ savings tend to decrease

Model A: Aid and accumulationDoucouliagos & Paldam, 2006

• Results• Unclear picture

• Partial crowding-out effect on savings: • Aid increases accumulation by about 25% of the aid• Most of the remaining 75% causes an increase in public consumption and a fall in public

savings

• Aid is likely to be unproductive: public consumption is known to have a negative effect on growth

The Estimated Effect of Aid on Either Savings or Investments

Model B: Aid and growthDoucouliagos & Paldam, 2008

• Data• 543 estimated aid effects

• Model• Literature on cross-country growth models• Barro equation:

• Replacement of the convergence term β lnyit with aid effectiveness term μ hits :

• The Model group B has participated in the idea that aid induces growth

Model B: Aid and growthDoucouliagos & Paldam, 2008

• Results• The variation is falling over time and with the

sample size• The best set is chosen among the highest points

(Funnel plot)• The average line is asymetric• The average result decreases and converges

towards zero• These results confirm the reluctance

hypothesis• Small samples give clearly misleading results• Non significativity is proven in D&P’s explicit

meta-regression

• Funnel asymmetry test: identifies the existence of a publication selection or “reluctance”ie test for the (positive) biaseness of μ examine funnel plots and the distribution of the estimated coefficients as a

function of sample size μ = μ(N)

FAT serves to detect convergence in the AEL process (do results about μ converge something that differs from 0?), asymmetries (reported estimated μ are not symmetrically distributed around the true value of μ when the research process is biased) as well as polishing (if the estimated μ are bigger when results are easier to polish, ie when the SE increases)

Funnel plot & Time series graph

Model C: Conditional effect on growthDoucouliagos & Paldam, 2007

• Data• Limited data

• 10 candidates for z variables• Large studies for 2 of the variables only

• Variable 1: 23 studies, 232 estimates• Variable 2: 16 studies, 123 estimates

• Model• Model C comes from model B• Assumption: Aid helps in some cases and harms in others, depending on

variable z (z>0 causes aide to work and z<0 causes aid to harm)

• 2 models• Good Policy Model

• Burnside & Dollar, 1996: aid to countries with « good » policies helps the countries (hitzit sgnificant and positive)

• The Medicine Model and the effect of Aid squared• μ>0 and w<0• Between h=0 and h=2h* (maximum for excess growth), the marginal contribution of aid to growth is

2wh<0

Model C: Conditional effect on growthDoucouliagos & Paldam, 2007

• Results• Good policy Model

• When the standard tools of the analysis are applied to the different studies, the key coefficient of the model is insignificant

• The estimated coefficient is unusually fragile

• The Medicine Model• When taken together, the different studies fail to prove that the two coefficients are

statistically different from zero• The p-values are near 0,05

• The conclusions of both models don’t show significant results• There is a possibility to test other z variables in the future

IV. An alternative to the AEL

• Resource Rents and Dutch Disease• Empirically, an increase in the income level causes a decrease in the

growth rate• Aid would be less of an advantage in the longer run• Corroboration: The total resource rent amount is twice as important as the aid amount

• The Key Role of the Real Exchange Rent• An unusually high rate of inflation has been noticed in aid-receiving

countries• A rent transfer inevitably leads to a real revaluation of the country’s currency• Expected negative effects on the growth rate, due to competitiveness decrease• Difficulties to verify the data considering the noticed lag (Germany, Greenland)

VI. Conclusion, limits and propositions

• Conclusion• The aid paradox noticed leads to a methodic meta-analysis• All meta-analysis of the AEL studies lead to the conclusion of aid

ineffectiveness, despite the 74% positive results of published aid-growth effects

• Limits• Best-state analysis

• The authors of the article consider equally every result from all analysis, however the authors could have weighted the different methods

• The Dutch disease• We can assume more controls on aid flows than on resource rents

• Propositions• The Dutch disease and the real exchange rate

• Further work can be provided