Getting more bang for the buck, euro, baht, dong, yuan, peso, rupee, kip…

The Challenge of Strengthening Impact Evaluation in the East Asia and Pacific Land Portfolio

World Bank Land Thematic Group Mini RetreatNovember 19, 2007

Background and context of EAP program

Challenges

Role of impact evaluation

Four examples

Going Forward

The Thailand Model has driven most of East Asia’s work in land, at least for land administration EAP has one of the largest land project portfolios in

the Bank, building on the pioneering work in Thailand in the 1970s and 1980s, particularly in land registration

The “Thailand model” has been replicated in Lao, Indonesia, Philippines and Cambodia, and elements are now being extended to Vietnam and China

The Thailand model includes impact evaluation as a core component and has been part of all of the scale ups

The projects have similar development objectives in terms of contributing to poverty reduction:

Increase land tenure security through more efficient land titling and registration to:

Increase access to credit Increase incentives to invest in land

However, the Thailand approach to impact evaluation has not been scaled up systematically… While top Bank researchers were involved in

Thailand, subsequent projects had very limited involvement of Bank impact evaluation specialists to assist design of this component

While Bank researchers worked closely with local researchers to build their skills in Thailand, there has been little or no methodological support to local research institutes in subsequent projects

Despite having a fairly consistent design with similar development objectives, there has not been an effort to develop standard tools and approaches to ensure at least a minimum standard of impact evaluation and provide some basis for comparability

EAP is facing several challenges in scaling up region-wide, as reflected in the portfolio Previous projects satisfactory and generally

sustainable 6 on-going projects (5 land administration and one

land reform) for total of about US$200 million 3 projects are at-risk generally due to fiduciary issues

(legal cov, FM and proct) rather than technical Significant disbursement lags (12 to 24 months) 2 new projects (one land reform, one land

administration) for delivery in FY08 and 3 for FY09

…which has focused us on some common areas for improvement:

Better mechanisms to address systematic corruption at project and sectoral levels

Project designs which are better tailored to country circumstances through simplified design, restructuring and enhanced SPN support

Capitalizing on new opportunities emerging with decentralization and technology innovation

Building effective approaches for improving governance through both supply and demand side approaches for improved service delivery

Defining effective instruments and long term programmatic approaches to the more sensitive aspects of the land agenda, such as land reform and state land issues, to develop stakeholder commitment

Monitoring and evaluation are critical to meet these business and development challenges:

Provide more “real time” feedback to identify needs for adjusting programs

Bring increased transparency to support participation and oversight by local communities and focus attention of politicians

Establish more objective, common information base for stakeholder dialogue on tough issues – M&E beyond projects

Does impact evaluation matter for EAP’s land work? Yes! Scale up has taken place in very different country environments

than Thailand, suggesting that the benefits themselves and the results chain differ across countries.

In the absence of commonly acknowledged, objective evaluation, dialogue space is often occupied by unsubtantiated critics of the Bank’s support

Performance under different enabling environments may inform the options to be presented in policy dialogue.

There are several aspects to program design which could be adopted more to local context but need better understanding of their role and performance in obtaining objectives.

As Bank role evolves with rapid changes in client countries, we need to understand what is working and what isn’t in developing service delivery mechanisms.

What are the key questions to address? Who are the main beneficiaries of land titling programs? Are there losers and why? What is the impact of titling on poverty and growth, and how

does it happen? What are impacts on broader land market development including

rentals? Is full individual title required to obtain the benefits or can

intermediate steps provide most of the benefits for the poor at lower cost?

Alternatively, to what extent do limitations on the property right reduce the poverty reduction impacts?

How do efforts to strengthen gender equity in the legal aspects of titling affect the above?

How do we explain differences in poverty outcomes between regions and groups in a country and across countries?

What are the complementary elements of the enabling environment which are required for titling to contribute to poverty reduction and growth?

Example 1: Indonesia Land Administration Project –Ex-post without baseline but control group-done

Contracted SMERU, a local research institution A combination of quantitative survey and in-depth interview was

used in the study. A questionnaire was developed by SMERU and used in data collection for the quantitative survey, while in-depth interviews used a guideline.

14 districts and cities where project operated were chosen, and then subdistricts were chosen based on degree of urbanization.

There were 1,596 randomly selected respondents from the villages and some key informants interviewed in the study.

1004 were households that received land certificates, 84 were in project villages and were eligible but did not receive certificates, and 508 were outside the project area and were considered the control group.

Example 1: Indonesia - Results

The program had no bias in terms of socio-economic groups receiving certificates

Although significant “additional costs” to titling process, majority felt significant net benefits

70% of certificates in name of husband only for jointly purchased property

12.8% greater mortgaging 5.3% higher investment 1.7% increase in land transactions in areas with

certificates 33% increase in land taxes in areas with certificates

Example 2: Philippines Comprehensive Agrarian Reform Program AAA-ex-post using secondary data Work with Asia Pacific Policy Center – a local research center Focus on analyzing existing data including (i) 1991 and 2002

agricultural censuses; (ii) the Integrated Farm Household Surveys of 1991, 1997, and 2004 conducted by the Bureau of Agricultural Statistics using the agricultural census as a sampling framework; (iii) the Family Income and Expenditure Surveys (1997 and 2002) and the Annual Poverty Indicators Survey (1998 and 1999) from which a set of panel data on 17,897 households is available; and (iv) a set of panel data collected by the Department of Agrarian Reform as part of its CARP impact assessments in 1990, 2000 and 2007.

Attempt to assess poverty impact, consequences of restrictions on sale and rental of reform lands

Other issues addressed through non-statistical approaches

Example 3: China Land Titling Pilot – Ex ante with specific baseline – on-going Development of questionnaire and evaluation strategy by Scott

Rozelle working with Chinese Agricultural University As one small group in one village is being titled in the pilot

project, the following means will be used to choose the sample: a.) select 30 households in the titling small group; b.) select 5 households in each of the other small groups (or a maximum of 20 households); c.) select 10 households in each of two adjacent villages. A total of 30+20+20=70 households will be surveyed. This survey will be repeated each year for the next 5 years.

In anticipation of the upscaling of the project, funding will be sought to do baselines and follow the progress of the upscaling of the project.

It is expected that when the plan is devised for the upscaling, it will maximize the learning about the various ways and options for implementing land titling.

Example 4: Vietnam–Land Registration w/Focus on Gender–Ex Ante integration with other surveys Issues new certificates and changing from man only to joint this will provide us with a unique possibility to test empirically

whether and how joint ownership differs from individual one Plan to put specific questions into upcoming (May 2008) LSMS

on land certificates (in name of male or female), legal knowledge, conflicts, and land market participation

Also Danish TF to have the University of Copenhagen (who are already having a panel survey scheduled for 2008, 10, and 12 to evaluate a DANIDA ag development project) at about the same time to use a survey instrument with identical sections to oversample the intervention provinces and communes

Issues emerging from review of impact evaluation in EAP land projects: Classical results chains but inconsistency in how this is described in terms

of expected project impacts and failure to do ex-post assessment of critical assumptions

Lost opportunities to introduce or utilize elements of randomization to strengthen impact evaluation

Problems in definition of control groups – delays with baseline, non-random selection, etc.

Very different survey instruments limit ability to link to household survey results, compare between projects or develop international comparisons

Limited use of other information in country Absence of cost-benefit or other efficiency measures Limited focus on other impacts including governance attitudes, land use

sustainability and other qualitative data Need for better measures of institutional devpt impact and sustainability Limited interest from clients, particularly if no grant funding, delays start

up and limits use of results Impacts exceed project period and bunching of implementation at end Missed opportunities to build evaluation capacity with local institutions

What we are doing to get ahead of this:

Strong interest from task teams and support from Land TG, including impact specialists as part of peer review process

Complete stock taking and agree on areas for focusing and sequencing strengthening work

Simplified project design should lead to more focused evaluation Integration of consistent land modules in household surveys used for

poverty assessments Highlighting linkages of land to poverty reduction in AAA More integration of process evaluation to capture governance aspects Broaden instruments to include more PSIA, PETS, citizen report cards

and others which can be implemented with local partners Engage govt, NGOs and other stakeholders in evaluation process Develop common instruments and regional community of practice of

local institutions around land impact evaluation Seek funding from Spanish TF for technical & implementation support to

develop approach in one or two countries which can be replicated

Let’s remember that we do this because poor people want tenure security and we want to do it better!