WHAT IS IMPACT EVALUATION? BRIEF IE DESCRIPTIONS

Kenya Patient Safety Impact Evaluation, KePSIE (P149879)

The lack of effective systems to regulate, measure, and improve patient safety and the quality of care places severe limits on evidence-based decision making. For instance, frequent calls for more inspections and greater regulation are not backed by evidence on the impact of such measures on patient safety and quality of care, particularly in the private sector. In Kenya (one of the only low-income countries with a national survey on patient safety), only 2% of health facilities in 2012 were compliant with minimum protocols and systems to assure patient safety (IFC and WHO, 2012).

The Kenya Patient Safety Impact Evaluation is a unique partnership with the Kenyan Government that builds on long-term IFC support for regulatory reform in the health sector. Using health markets as the unit of intervention (defined as clusters of health facilities where no facility is more than 4 KM from the center of the market), we will experimentally allocate all 1,100 private and public health facilities in three Kenyan counties–Kakamega, Kilifi and Meru–into one of three arms: (1) high-intensity inspections with enforcement of warnings and sanctions for non-compliant facilities; (2) high-intensity inspections with enforcement of warnings and sanctions for non-compliant facilities and public disclosure of inspection

i2iDIME

TRANSFORM DEVELOPMENT

CHANGING POLICY THROUGH RESEARCH: DIME IE PORTFOLIO IN KENYA

i2iDIME

TRANSFORM DEVELOPMENT

Departmentfor InternationalDevelopment from the British people

Impact evaluation (IE) uses rigorous experimental and quasi-experimental methods to identify the direct causal impact of a well-defined intervention against a scientifically valid counterfactual. This counterfactual represents what would have happened to program recipients had the program not taken place, or had a different program design been chosen. Thus, IE builds on traditional monitoring and evaluation (M&E) by directly attributing specific changes in outcomes to specific interventions. When undertaken prospectively (i.e. designed and planned prior to the implementation of interventions) and integrated into the operational planning and implementation of a program or policy, IEs are a powerful research and development tool to identify the most effective and cost-effective design options. For this reason, IEs are a management tool for results-based policy planning. At the same time, they are a key instrument for improving aid effectiveness and increasing fiscal accountability.

There are currently eight ongoing DIME impact evaluations in Kenya:

1 Inspections Regimes in Healthcare and their Impact on Patient Safety and Quality of Care in Kenya (P586467)

2 Turning Pipe Dreams into Reality: Understanding the Demand Curve for Sanitation in Nairobi Slums (P144145, IDA16 commitment).

3 Judicial Performance Improvement Project (P105269)

4 HIV Life Skills for In-School Youth (P149162)

5 HIV Sensitization Campaigns for Out-of-school youth (P149168) under the Total War Against HIV/AIDS-TOWA Project (P081712)

6 Nzoia Irrigation Program (P145556, IDA 16 commitment).

7 The Demand for Costs and Socio-Economic Impacts of Rural Electrification in Kenya (P120014).

8 Road Safety in Kenya: Matatu and Insurance Markets

Kenya Impact Evaluations

2 | i2i

results, and (3) “business-as-usual” low-probability inspections (the control group). The results from this study will contribute to understanding the extent to which governance and accountability mechanisms can help improve service delivery in low-income countries, particularly in this case related to improving patient safety and quality of care in the public and private sectors, and if so, how these systems may operate when they are implemented “at scale.” Another important contribution of this project is the development of a set of tools and instruments that can be broadly deployed both in surveys of quality and in the design of inspection systems across diverse low-income settings.

Some results of KePSIE to date include:

• Reform to the Regulatory Framework of the Inspection System for Minimum Patient Safety. The new regulatory framework includes launched by the government in 2015 (a) a refined Joint Inspection Health Checklist with itemized patient safety indicators to make it leaner and less discretionary, easier to deploy and further focused on the fundamentals of patient safety; (b) a scoring system that allows facilities to be categorized according to the level of risk presented to patients; (c) warnings and sanctions to be enforced according to a facility’s level of risk. Additionally, it will be extended to public facilities (only private facilities were covered by the previous legislation).

• Design and Validation of Patient Safety and Quality of Care Measures: During 2014 KePSIE developed a set of tools and instruments in close collaboration with the Ministry of Health and the Boards and Councils that can be broadly deployed in diverse low-income settings, both in surveys of quality and in the design of inspection systems. A combination of methods including surveys for health facilities, patients, and health workers and observation (both known and unknown to the health care provider) were used to develop these tools.

• Collection of Baseline Data in All Health Facilities in the Three Study Counties: Between January and September of 2015, KePSIE collected data in all public and private health facilities of Kakamega, Kilifi and Meru to assess patient safety, quality of care, demand and prices, before the beginning of the intervention. Data collection included around 1,100 health-facility level surveys, 8,000 patient exit surveys, 1,600 health-care worker surveys, and 19,000 observations of healthcare-worker-patient interaction. The scale and scope of this survey makes it the world’s biggest patient safety survey conducted so far.

The team is currently analyzing the results from the baseline data collection, and preparing for the implementation of the inspections regimes, which is planned to start around March 2016.

Turning Pipe Dreams into Reality: Understanding the Demand Curve for Sanitation in Nairobi Slums (P144145, IDA 16 commitment)

The overarching question being asked with this evaluation is how to maximize the development impact of large bulk infrastructure investments in water and sanitation. This evaluation has two main objectives. The primary objective is to understand the demand curve for sanitation services. By offering different levels of subsidies, we will be able to learn what the willingness of poor households is to pay for sanitation connections. By doing so we will identify whether financial constraints are the main underlying reason for low take up rates. Additionally, we will study how take-up rates vary by household structure (compound landlord is resident versus non-resident, etc.) and thus understand if collective action issues and coordination failures are also constraining factors for sanitation services demand.

The second objective of this evaluation is to examine the health and welfare impacts of large infrastructure sanitation services. Different levels of subsidies and the induced variability in sewerage connections that they generate among the population studied, will ultimately be used to assess impacts on welfare and health. Lastly, this evaluation also aims to understand whether the provision of subsidies for sewerage connections, combined with a hygiene campaign that provide awareness, information and hygienic promotion, can increase take up rates and maximize the health and welfare benefits of the intervention.

Judicial Performance Improvement Project (P105269)

The primary goal of this impact evaluation is to test alternative implementation strategies for maximizing the effectiveness of the “Performance Management and Measurement

i2i | 3

Kenya Impact Evaluations

Understandings” (PMMUs). These performance contracts were introduced in the Kenyan Judiciary in 2015 as part of larger reform process to improve the Judiciary, which experiences low public trust and confidence. Several assessment of the Judiciary have been conducted and they all point out to persistent capacity and integrity problems. These reports described the Judiciary’s failure to deliver timely justice, significant case backlogs, ineffective case management systems, poor conditions of service for its officers and deficient physical infrastructure. The PMMUs are seen as an intervention that can potentially improve these issues. They are signed between the Chief Justice and heads of court divisions as well as key administrative personnel and then cascaded down to judicial officers and staff. The PMMUs set targets focused on improving the efficiency of case processing and the quality of judicial services.

The IE will test measures that could enhance the impact of the PMMUs on performance, in particular it will test three of such measures: (i) information that illustrates to PMMU signatories how they are performing against their targets and compared to peers; and (ii) complementing the information with calls from management for supervisory accountability, and (iii) sharing this information with court user committees as a means of local accountability. Given the judiciary’s current emphasis on reducing its backlog of cases, a key secondary concern for the evaluation, is to ensure that performance contracts designed to improve the speed of justice do not undermine the quality of case processing.

To date, the IE has scored some success in affecting policy. Indeed, the IE has supported the development of the Daily Court Returns Template, which was recently gazetted as Legislation. This tool aims to scientifically gather and organize daily court output, which in turn feeds into a monthly report on how each court is doing. This data will be key for the experiments of the IE and will remain as a policy tool beyond the IE.

HIV Life Skills for In-School Youth (P149162), and HIV Sensitization Campaigns for Out-of-school youth (P149168), under the Total War Against HIV/AIDS-TOWA project (P081712)

Young women are disproportionately affected by the HIV/AIDS epidemic. Women age 20–24 are four times more likely to be infected than men of the same age. Gender-based violence, physical or sexual, is common in Kenya particularly in Nyanza, where six in ten women have reported suffered it. Violence against women has serious consequences for their mental and physical well-being, including their reproductive and sexual health.

These two impact evaluations study the effectiveness of two common behavior change communication programs supported by the government of Kenya: A life skills curriculum that targets in-school youth and video screenings that targets out-of-school youth in villages. Both take a multifaceted approach by implementing behavior change interventions in different social contexts and share the objective of reducing risky sexual behavior and gender-based violence among 15–24 year olds in Nyanza province. The baseline survey was collected in 2013. Both the in school and out of school interventions were carried out in 2013 and 2014. The follow up survey is planned for 2016.

Using data and background research for these evaluations, the research team wrote a paper about the determinants of HIV Testing Behavior in Kenya and its implications for ARV scale-up.

Nzoia Irrigation Program (P145556)

The evaluation will explore the impact of the Nzoia irrigation scheme on the ability for smallholder farmers to switch to commercial crops (e.g. flowers) and its role in improving food security. Beyond the infrastructure investment the impact evaluation also plans to unpack the role of credit and financial education for farmers. The concept note has been completed and preparation for the baseline survey is ongoing.

The Demand for, Costs, and Socio-Economic Impacts of Rural Electrification in Kenya

This IE will generate new data on the demand for, costs, and social and economic impacts of national grid electrification in Kenya. It will address the following questions:

1. Demand for household electricity connections at various levels of prices?

2. Economies of scale in costs by connecting neighboring clusters of households at the same time?

3. Effects of electricity access on appliance ownership and monthly electricity consumption for newly connected households over time?

4. Impact of modern energy on social and economic outcomes, such as children’s education, health? 5. impact on small business competition? Evaluation Design: Random assignment at village level

Associated WB Project: Kenya Electricity Modernization Project (P120014)

Road Safety in Kenya: Matatu and Insurance Markets

Road traffic accidents (RTA) take a huge economic toll worldwide. Kenya ranks among the top 10 countries in death rate due to RTAs in the world, with 19 deaths per 100,000 inhabitants and more than 420,000 non-fatal injuries per year. Matatus (mini-buses), which make up roughly 11 percent of registered vehicles in Kenya, are responsible for 73 percent of RTAs in the country. The structure of the matatu and insurance markets has two characteristics that make reducing unsafe driving challenging.

There is a clear need for policy efforts that encourage safer driving and increased accountability in contexts like Kenya. Existing policies, like the 2004 Michuki Rules in Kenya (e.g., speed governors, seatbelt requirements, set routes, inspections and certifications of vehicles, better contract conditions for drivers) have been ineffective in changing the incentives in the current system, as they aimed to address the consequences, but not the causes, of misaligned incentives

and poor accountability. An intervention that addresses these two inefficiencies would increase safety on Kenyan roads, and greatly improve social welfare. The proposed interventions include the construction of accountability systems that can improve information about drivers and their behavior, and the use of these systems to test strategies to strengthen the incentives for safe driving. The proposed IE therefore aims to help us learn more about the existing structures that lead to poor road safety in Kenya, and also aim to provide a platform for developing insights that could directly feed into policy decisions in the Kenyan transport sector. The main ultimate outcome will be road traffic accidents and the intermediate outcome (and behavior we want to incentivize) is safer driving. The main ultimate outcome will be measured through accidents and insurance claims data for driver-caused accidents from our insurance company partner on the ground, and from the police.

i2iDIME

TRANSFORM DEVELOPMENT

Departmentfor InternationalDevelopment from the British people

Kenya Impact Evaluations

IMPACT EVALUATION TEAM

IE Management Team and Operations Counterparts. DECIE: Vincenzo di Maro, Victor Orozco, Aidan Coville, Guadalupe Bedoya, Susumu Yoshida. The management team provides technical and operational inputs at all stages of the IE process, quality control, and overall management of IE-related activities.

AFR counterparts: Wendy Ayres, Patrick Mwangi, Regassa Namara.

Principal and Co-Principal Investigators. Arianna Legovini, Manager, DIME; Jishnu Das, Lead Economist, DECHD; Vincenzo di Maro, Economist, DECIE; Victor Orozco, Economist, DECIE; Guadalupe Bedoya, Economist, DECIE; Daniel Rogger, Economist, DECIE; Ben Daniels, Consultant, DECHD; Josh Graff Zivin, Professor of Economics, University of California–San Diego; Harsha Thirumurthy, Assistant Professor of Health Economics, University of North Carolina; Justin Sandefur, Research Fellow, Center for Global Development; Paul Gertler, UC Berkeley; Sebastian

Galiani, University of Maryland. The principal and co-principal investigators have primary responsibility for research design and analysis, and for ensuring that this builds on existing knowledge broadly applicable to projects with similar objectives and designs. PIs contribute to operational planning, oversee data collection, draft IE-related reports, and participate in dissemination activities.

Field Coordinators. Lacey Ramirez, Amy Dolinger, DECIE, Geoffrey Onyambu (IPA). The field coordinators, based in Nairobi, are responsible for day-to-day IE-related technical assistance to the implementing agencies in relation to all those things concerning the impact evaluation (including IE and intervention design, data collection, implementation support, analysis, and dissemination). Field coordinators work closely with the implementing agencies on a full-time basis, and play a key role in ensuring that IE results feed into policymaking on an ongoing basis and in supporting IE-related capacity-building for implementing partners.

Research Assistants. The research assistant provides support to the impact evaluation investigators with the preparation and processing of data and with IE-related analytical work.