Health Programme Evaluation by Propensity Score Matching: Accounting

for Treatment Intensity and Health Externalities with an Application to Brazil

(HEDG Working Paper 09/05, March, 2009)

Rodrigo Moreno-SerraCentre for Health Economics & Department of Economics

University of Yorkrams500@york.ac.uk

AfrEA-NONIE-3ie Conference, Cairo, 2009

 

Introduction & Motivation

 

Main programme evaluation challenge: ex-post construction of an adequate comparison group, often within a non-experimental setting, to obtain average treatment effects

Key assumption:

Values of treated and untreated outcomes for a given individual are not influenced by the treatment status of other individuals

Usually unrealistic for health programmes: externalities can lead to underestimation of total programme impacts (cf. e.g., Miguel and Kremer, 2004)

Introduction & Motivation

 

Possible solution in non-experimental settings: use availability of a health programme in a given area as the treatment variable of interest

Treated individuals defined as those who live in areas where programme is in place (treated areas)

Methodology already used, normally through an indicator variable for presence/absence of the programme

One mean programme impact that accounts for health externalities to individuals in the treatment areas who did not directly receive the intervention

Yet magnitude of externalities within each locality (and thus the associated average treatment effect) is likely to depend on the number of individuals who actually receive the programme’s services there: intensity of treatment

Suggested empirical methodology at a glance

 

Use a measure of the programme’s population coverage across areas as the treatment variable of interest

Estimate average treatment effects through comparisons between the health impacts of alternative coverage levels vs. reference level (e.g., zero)

Need panel-data or repeated cross-sections (before and after) on coverage levels (phased-in programme) and individual variables

Compare change in outcomes for individuals living in an area with coverage level l (a treatment area) to the change in health outcomes for similar individuals living in the area with coverage level 0 (the comparison area), for a number of l > 0

Implementation: propensity score matching estimators adapted to the case of multiple treatments (Imbens, 2000; Lechner, 2000), coupled with a difference-in-differences approach (PSDD)

Suggested empirical methodology at a glance

 

PSDD estimator of the average treatment effect on the treated (ATT) with repeated cross-sections (Blundell and Costa-Dias, 2000):

Construction of comparison groups through propensity score matching for multiple treatments (generalized propensity score): more than one active treatment, i.e. coverage levels

ATT computed by difference-in-differences

Key assumption: bias stability

ATT of living in area l vs. living in area 0 takes into account health benefits to individuals living in l who did not receive programme services themselves

An example: the Brazilian PSF

 

Family Health Programme (PSF):

MoH initiative with the stated aim of “improving the health status of covered families”

Family Health Teams have to be formed by family doctor, nurse, assistant-nurse and 4-6 community health agents

Monthly household visits: preventive and health promotion actions for all the individuals in a family (adults, children, seniors)

Municipalities make PSF adoption decision: individuals mandatorily covered (visited)

Broadest health programme ever launched in Brazil: 80 million people (2006), yet important variations across regions

Average health gain for resident of a given region likely to increase (non-linearly?) according to coverage, also due to externalities

Data

 

Porto Alegre is the comparison region (~ zero coverage, 98-03)

Household survey data (1998 & 2003): >127,000 individuals (adults and children) and 34,000 households per wave

Individual matching variables include household living conditions, demographics, education, labour and income characteristics

Evaluation question

 

What are the average health impacts of being exposed to each of the eight observed PSF coverage levels from 1998 to 2003, compared to living in the comparison region during the same period (the “no-programme” benchmark)?

Health outcomes: (1) self-assessed health; (2) bed due to illness; and (3) inability to perform usual activities due to illness

One ATT is estimated for each of the eight relevant pairwise comparisons: being exposed to the PSF coverage level observed in region 1, 2, 3… vs. not exposed to the PSF in Porto Alegre

Specification tests: matching successful for construction of similar comparison groups (both samples, adults and children)

Main results

 

Overall, positive levels of PSF coverage in a region tend to lead to improvements in individual health outcomes (small effects for adults, larger estimated impacts for children)

Largest ATT tend to be found for residents of the regions with the three highest median PSF coverage levels during 1998-03 (Belo Horizonte—16%, Recife—23%, Fortaleza—24%)

E.g., children in Fortaleza vs. Porto Alegre: (1) 5-8p.p. higher prob of good SAH; (2) 3p.p. lower prob of bed episode; (3) 4-5p.p. lower prob of inability to perform usual activities due to illness

But no clear pattern of increasing health benefits according to higher coverage levels: too few/low coverage levels available

Concluding remarks

 

Health programme evaluation strategy that can be applied when information on actual impacts is needed to guide resource allocation and roll-out strategies, but only limited (routine, non-experimental) data are available

Of course, validity of assumptions of the PSDD estimator with multiple treatments needs to be assessed case-by-case

Impact estimates account for (i) different health endowments, and the potentially substantial (ii) treatment non-linearities and (iii) externalities from different levels of population coverage

Comprehensive account of the health benefits generated by an intervention—not only its effects on actually “treated” individuals: relevance for policy-makers