Evaluating the Effectiveness of OECD Economic Surveillance

John C. Whitehead School of Diplomacy and International Relations

Seton Hall University

PURPOSE• Our objective is to understand the overall impact of

OECD economic surveillance on member countries.

• We analyzed economic surveys for randomly selected OECD Country Members across the three most recent review cycles.

• We determined the extent to which OECD

recommendations were implemented, and we also evaluated the content of these reports.

IMPORTANCE• The OECD is understudied relative to other international

economic organizations.

• Given the global economic crisis, surveillance is important. What are its strengths and weaknesses, and how can it be made better?

• OECD surveillance is conducted every 18 months, and functions by peer review. How well does peer review work?

OUR APPROACH• Two project team members were randomly assigned to each of 24

OECD countries.

• Each team member performed two tasks:• Assess the extent to which past recommendations by the OECD were adopted or

pending in the two most recent Economic Surveys.

• Assess the content of surveillance by completing a survey of the three most recent Economic Surveys.

• Each country team initially worked independently, then collaborated to develop a composite assessment of surveillance effectiveness and surveillance content.

• Project data is available on our website http://oecdproject.wordpress.com/

COUNTRIES IN STUDY

• Twenty-Four randomly selected OECD Member Countries (out of 34 total)

Australia France Italy Slovakia

Austria Germany Japan Spain

Canada Greece Korea Sweden

Chile Hungary Mexico Turkey

Czech Republic Iceland Norway United Kingdom

Denmark Ireland Portugal United States

MEASURING EFFECTIVENESS• Assessment of “Progress in Structural Reform” tables in two

most recent Economic Surveys.

• Coded actions taken by member countries as either • “adopted” (any official policy change in the spirit of OECD

recommendations)and/or

• “pending” (recommendations pending, proposed, or under review)

• Total of 1680 unique recommendations by OECD (average of 70 per country)

OVERALL FINDINGS FOR EFFECTIVENESS

•52% ADOPTED (Median: 53%)• Range: Portugal 75%, Slovakia 25%

•28% PENDING (Median: 27%)• Range: UK 61%, Slovakia 15%

SPECIFIC FINDINGS FOR EFFECTIVENESS• We disaggregated these results in three ways: • Changes in effectiveness over time comparing the two most

recent reviews• Comparing effectiveness in crisis and non-crisis countries• Comparing effectiveness across type of economic issue

CHANGES IN EFFECTIVENESS OVER TIME• Recommendation Adoption• 11 Improved, 12 Worsened, 1 Stayed the Same• Overall worsening with net average of -2%• Range: Turkey +32%, France -29%

• Recommendations Pending• 10 Improved, 14 Worsened• Overall worsening with net average of -1%• Range: Italy +32%, Canada -25%

CRISIS VS. NON-CRISIS COUNTRIES

•We classified countries in crisis based on whether they received IMF adjustment loans:• Greece (2011)• Hungary (2010)• Iceland (2009, 2011)• Ireland (2009, 2011)• Turkey (2008, 2010) • Mexico (2009-2011)

EFFECTIVENESS IN CRISIS AND NON-CRISIS COUNTRIES

% RECOMMENDATIONSADOPTED

% RECOMMENDATIONSPENDING

Greece 2011 62% 36%

Hungary 2010 73% 8%

Iceland 43% 23%

Ireland 51% 30%

Turkey 66% 20%

Mexico 66% 16%

TOTAL: 59% 24%

Non-Crisis Mean 51% 29%

CRISIS VS. NON-CRISIS COUNTRIES

• We compare within-case variations for Greece and Hungary• These countries had Economic Surveys in years preceding crisis as

well as crisis years.• Greece: Adopted 16% more recommendations in crisis years• Hungary: Adopted 14% more recommendations in crisis years

• Implication: Surveillance can make a difference for countries in crisis.

EFFECTIVENESS ACROSS ISSUE AREAS

% RECOMMENDATIONSADOPTED

% RECOMMENDATIONSPENDING

Social Policy* 53 33

Fiscal Policy 53 24

Financial Sector 56 21

Market Competition 48 27

*Social Policy includes education, healthcare, housing, labor, environment, immigration, human capital, pensions, family, aging, and infrastructure.

SURVEILLANCE CONTENT• Assessed three most recent OECD Economic Surveys

• Coded each survey based on four questions:• 1). Is the political and social context of the country explained? • 2) Does the economic survey include a self-contained comprehensive

statement assessing the effectiveness of past surveillance in the country?

• 3) What reasons (if any) were provided where past OECD advice was not followed?

• 4) Are dissenting views by the authorities clearly identified and explained?

OVERALL FINDINGS FOR SURVEILLANCE CONTENT• Question 1• 91% of Surveys had political or social context.

• Question 2• 86% of Surveys had a comprehensive effectiveness statement.

• Question 3• 61% of Surveys contained reasons for past non-compliance.

• Question 4• 23% of Surveys contained dissenting views by state authorities.

SPECIFIC FINDINGS FOR SURVEILLANCE CONTENT• We disaggregated these results in two ways: • Changes in content over time comparing the two most

recent reviews• Comparing content in crisis and non-crisis countries

CHANGES IN SURVEILLANCE CONTENT OVER TIME

Economic SurveyYear 1

Economic SurveyYear 2

Economic SurveyYear 3

Question One (Q1) 91% 87% 96%Question Two (Q2) 58% 100% 100%Question Three (Q3) 58% 62% 62%Question Four (Q4) 16% 25% 29%

Increased transparency over the span of the most three recent reports

Can improve surveillance by including more dissenting views which would provide a detailed explanation why past advice was not adopted

SURVEILLANCE CONTENT IN CRISIS AND NON-CRISIS COUNTRIES

Q1 Q2 Q3 Q4

Greece (2010) 100% 100% 100% 100%

Hungary (2011) 100% 100% 100% 100%

Iceland (2009, 2011) 100% 100% 100% 0%

Ireland (2009, 2011) 100% 100% 50% 0%

Mexico (2009, 2011) 100% 100% 0% 50%

Turkey (2008, 2010) 100% 100% 50% 0%

TOTAL 100% 100% 66% 41%

Non-Crisis Mean 91% 83% 59% 24%

CRISIS VS. NON-CRISIS COUNTRIES

• More transparency about non-adoption of reforms in crisis countries.

• Greece and Hungary had detailed discussions of dissents in pre-crisis years as well• Finding on Q4 for greater transparency about dissents should be

tempered.

FINDINGS ON SURVEILLANCE CONTENT

• Surveillance reports include contextual information.• Surveillance reports provide a benchmark for evaluating prior

recommendations. • Explanations for why past recommendations were not adopted

appear in only 3 out of 5 reports.• Discussions of non-adoption are more anecdotal than substantive.

FINDINGS ON SURVEILLANCE CONTENT

• Surveillance reports include contextual information.• Surveillance reports provide a benchmark for evaluating prior

recommendations. • Explanations for why past recommendations were not adopted

appear in only 3 out of 5 reports.• Discussions of non-adoption are more anecdotal than substantive.

CAN BETTER CONTENT PRODUCE MORE EFFECTIVE SURVEILLANCE?• Adoption and pending percentages in the most

recent Economic Survey by whether the prior survey had positive answers to Questions 3 and 4.

% Recommendations Adopted

% Recommendations Pending

Questions 3 & 4 = NO 57% 23%

Questions 3 & 4 = YES 60% 45%

POLICY RECOMMENDATIONS• Economic Surveys should take a multi-year perspective to their

recommendations.

• The issue of non-adoption of reforms should receive greater attention.

• More consideration should be given to the domestic political context when developing recommendations.

RECOMMENDATION 1: MULTI-YEAR PERSPECTIVE

• Economic Surveys should take a multi-year perspective to their recommendations.• Helps to track whether pending recommendations become adopted in

subsequent years• Helps call attention to potential for reform backsliding

• Could be implemented thematically initially

RECOMMENDATION 2: ADDRESSING NON-ADOPTION

• The issue of non-adoption of reforms should receive greater attention.• Economic Surveys should include a section on why prior

recommendations have not been adopted.• More detail on whether countries dissent with recommendations

would make dialogue more substantive • Consider developing a publicly available dataset (i.e. the IMF’s MONA

data)• Use greater media outreach

RECOMMENDATION 3: TAKE POLITICS SERIOUSLY

• More consideration should be given to the domestic political context when developing recommendations.• Consider prioritized recommendations• Consider tradeoffs between optimal reforms and politically feasible

reforms

PROJECT TEAM MEMBERS

Vanessa Alva-Araya Teruo Katsukawa

Ronald Andrew Jacquie Marte

Richard Armstrong Jordan McGillis

Catherine Baxter Joseph Messina

Paul Bezerra Osman Oztoprak

Gene Bolton Sofia Pantel del Cueto

Jessica Carroll Andrew Prempeh

Kelsey Christianson Samuel Roods

James Einhaus Nick Rosario

Emily Hampton Subarna Saha

Teale Harold Kyle Shong

Jennifer Ijichi Constantina Soukas

Professor Martin Edwards

FOR MORE INFORMATION• Project Website: • http://oecdproject.wordpress.com/

• Email address for Project Team: • whiteheadOECDproject@gmail.com