evidence to policy initiative (e2pi) to policy initiative (e2pi) ... the challenges of changing...

Gavin Yamey MD MPH, lead, San Francisco hub, E2Pi

Evidence to Policy initiative (E2Pi)

Estimating Benchmarks of Success in the AMFm Phase 1

Presentation to IHME, March 9 2011

• Clinical medicine, journalism, public health• 2009 Kaiser Family Foundation Mini‐Fellow in Global Health Reporting (Sudan, Kenya, Uganda)

Telegraph (UK)5 October 2009“The Net Gains of Keeping Mosquitoes at Bay”

My BackgroundMy background

www.e2pi.org

• GH policy collaboration» GHG at UCSF» SEEK Development, Berlin

• Evidence synthesis & policy analysis to inform discussion & decisions on critical policy and strategic issues

• Two focus areas » GH financing partnerships: Global Fund, GAVI, UNITAID» European donors

• Core funding from Gates Foundation + contracts

AMFm objectivesTerms of referenceEvidence synthesis & key informant interviews3 approaches to estimating success benchmarks

• Pragmatic “mixed methods” approach • Weighted mean approach• Monte Carlo multivariate sensitivity analysis

Estimates of success benchmarksBalancing success measures

Overview

• WHO recommends ACTs for uncomplicated P. falciparummalaria

• BUT <15% children (range 3‐25%) with fever in SSA receive ACTs [World Malaria Report 2009]

• Key reason: about 50‐75% patients with suspected malaria seek Rx in private sector

» ACT course typically costs $6‐8

» 10‐20 times as much as older drugs (e.g. CQ, SP)

Why Was the AMFm Launched?

• Donor‐funded ACT price subsidy managed by the Global Fund

• Donors subsidize the cost of ACTs paid by “1st line buyers”

» buyers who purchase ACTs directly from the manufacturer, or procurement agents buying on their behalf

» private/public/NGO

• Proponents argue: subsidy will be passed along supply chain to the consumer

• Subsidy is combined with “supportive interventions”

AMFm: Global ACT Price Subsidy Scheme

AMFm Technical Design, Nov 2007

donor co‐payment

1. Reduce price of QAACTs compared to price of other anti‐malarials [AMs] (e.g. CQ, SP, AMT)

2. Increase availability of QAACTs in public/private outlets

3. Increase market share of QAACTs and crowd out other AMs

4. Increase use of QAACTs, including by vulnerable groups

Objectives of AMFm

subsidy + supportive

interventionsACT price

fallsACT

availability increases

ACT access and use increase

malaria burden falls

• Pilot launched in August 2010 in 8 countries» Cambodia, Ghana, Kenya, Madagascar, Niger, Nigeria, Tanzania, Uganda

• Independent evaluation (Macro & LSHTM): before‐and‐after design, measures 3 out of 4 objectives (not use)

• Data collection must be complete by Nov 2011 (by then, only 3 countries will have had subsidized drugs for at least 1 yr)

• Global Fund Board will decide at its 2nd meeting in 2012 whether to “continue, expand, suspend, or terminate the program”

AMFm Pilot (Phase 1)




Overview

E2Pi was commissioned to:• Provide “evidence that will help the AHC to reach decisions on how to

judge success of AMFm Phase 1”• Estimate what might be realistically expected at 1 year and 2 years for

the 4 AMFm objectives• Present ways of balancing and judging performance across the 4

objectives, including:• Balanced Scorecard • GF‐type approach• Weighting of indicators

E2Pi’s Terms of Reference

15 days to conduct initial literature review/key informant interviews

Process and Timeline

June 2010: draft presented to AHC

Additional interviews and literature review

Peer review: revised version sent to 9 external peer reviewers

Addressed reviewer comments

Oct 2010: revision presented to AHC

Dec 2010: final paper to AHC & Global Fund Board




Overview

Four major sources of evidence

1. Sub‐national ACT subsidy pilots and national ACT subsidy programs

2. Country‐level data on other national ACT scale‐up programs e.g. funded by GFATM

3. Country‐level data on commodity social marketing programs e.g. zinc, condoms

4. Global‐level data on oral rehydration therapy scale‐up

Literature Review

• Malaria experts: ACTwatch, CHAI, Global Fund, HAI, MMV• Social marketing organizations: PSI, MSI• Drug companies: GSK, Novartis, Cipla, Ipca• Local drug importer/distributor: Surgipharm (Uganda)• Academia: GWU, Johns Hopkins, LSHTM, Karolinska Institute, MIT‐

Zaragoza

33 Key Informant Interviews

ACT Subsidy Pilots: Design and Scale

DESIGN SCALE

Kenya Cluster RCT 3 districts, 18 clusters

Tanzania Quasi-randomized trial 2 intervention districts1 control district

Uganda Non-randomized,controlled trial

4 intervention districts1 control district

Angola Uncontrolled trial 2 municipalities (95 pharmacies)

Only 1 trial is published: Sabot OJ et al (2009) Piloting the Global Subsidy: The Impact of Subsidized Artemisinin‐Based Combination Therapies Distributed through Private Drug Shops in Rural Tanzania. PLoS ONE 4(9): e6857

AVAILABILITY• Proportion of private outlets stocking ACTs rose from 0% to 69‐81% at 1 yr (3 pilots); control data only for TZ pilot (fell from 1% to 0%)

MARKET SHARE• Increased from 0‐1% to 38‐51% at 1 yr, lower among poorest SEQ (3 pilots); control data only for TZ pilot (rose from 0% to 6%)

PRICE• Similar to price of suboptimal AMs at 1 yr (4 pilots) in intervention districts; no control data in 3 controlled trials

USE • Uganda (non‐randomized): Increase from baseline of 16 percentage points in intervention group but outcomes were better in control group

• Kenya (RCT): Increase from baseline of 40.2 percentage points in the intervention arm and 14.6 percentage points in the control arm at 1 yr

ACT Subsidy Pilots: Outcomes

• Pilots are small‐scale, focusing on sub‐national level

• Additional distribution mechanisms (Angola, Uganda)

• Tight monitoring of price violations (Angola)

• Intensive donor‐funded supportive interventions

• Design weaknesses: small samples, lack of randomization, one was

uncontrolled, one trial introduced a new intervention

• Results represent “trial conditions”—not “real world” conditions

“Great care should be taken in trying to learn lessons for the

AMFm Phase 1 from the small pilots” KI INTERVIEW

Why Not Extrapolate Directly From ACT Pilots?

Data from 6 National ACT Subsidy Programs

Country Lead organization

Launch yr Age group Outlet Coverage

Cameroon Govt. 2007 All age groups Public and private health facilities

Countrywide

Senegal Govt. 2006 All age groups Pharmacies Countrywide

Cambodia PSI 2002 All age groups Pharmacies, drug shops

17 of 20 malaria endemic provinces

DRC PSI 2006 Children under 5 Pharmacies Limited to some districts

Madagascar PSI 2003 Children under 5 Pharmacies, private providers, community agents

Countrywide

Rwanda PSI 2007 Children under 5 Pharmacies Countrywide

• Only 1 program has baseline data: Rwanda: 80‐90% at 18 months after launch (baseline 10%)

• Cambodia: Child ACT 6%, Adult ACT 22% at 1 yr• Cameroon: Low availability (key informant interviews)• Senegal: 29% infant ACTs, 43% child ACTs, 11% adult ACTs at 1 yr

Other AMs still very widely available in all countries except Rwanda (e.g. Senegal: SP availability 96‐100% in private outlets)

National ACT Subsidy Programs: Availability

MARKET SHARE • Very few data available; no baseline data• Cambodia: ACT accounted for only 28% of all AM sales in private outlets at 6 yrs; mono‐therapies still accounted for 50% of all sales in commercial private sector

• E2Pi analysis of PSI sales volume data:» It took PSI at least 3 yrs to reach substantial sales volumes for ACT (confirmed by PSI interviews)

National ACT Subsidy Programs: Market Share

Trends in PSI Sales of Subsidized ACT

Jan 06

July 2003Feb 2003

Sept 2006

• Red arrows show when subsidized ACT sales began

• It took at least 3 yrs to substantially increase sales volumes

• No baseline data• Low levels of ACT use in 3 countries

» DRC: 1% at about 1 yr» Senegal: 4% at 2‐3 yrs» Madagascar: 2.4% at about 5 yrs

National ACT Subsidy Programs: Use

• Very few available data (specific data for only 2 countries)• Senegal: mean consumer price for adult ACT at private outlets was $1.34 at 1 yr, similar to RRP, lower than SP price ($2.00)» Private outlets purchased subsidized ACTs at $0.99 i.e. subsidy largely passed on to consumers

• Cambodia: mean consumer price for adult ACTs was $1.07 at 4 yrs, 535% of CQ price ($0.20)» Private outlets purchased subsidized ACTs at $0.42 » Similar mark‐up for RDTs (sold to retailers at $0.10/test; consumers pay mean price of $0.37/test)

National ACT Subsidy Programs: Price

• Pilots found a rapid rise in ACT availability in private outlets, as did one national program

• Subsidies were associated with reduced consumer prices (i.e. subsidies were largely passed along the supply chain to the consumer)

• ACT market share increased rapidly in pilots, crowding out other AMs, but not in national programs

• Pilots found conflicting evidence on ACT use (one twas positive, one was negative) and national programs found very low levels of use at 1‐5 yrs

Summary of Results from Pilots/Programs

Data sources• ACTwatch outlet and household surveys • Surveys conducted in the context of the Global Fund 5‐Year Evaluation• Other national‐level surveys (DHS; MICS; MIS)• Peer‐reviewed studies in public health journals

Results• ACT availability, market share, and use were poor across surveys and

countries at 1‐2 yrs (and beyond), with a few exceptions (see next 2 slides)

• Results suggest great caution in expecting dramatic changes within 1‐2 yrs at national scale in AMFm

Evidence from National ACT Scale‐Up Programs

• Two possible baselines for estimating success of ACT scale‐up:» Yr that ACT was adopted as 1st‐line treatment» Yr of national ACT roll‐out

• Yr that ACT was adopted as 1st‐line is problematic: large time lag between change in national drug policy and actual ACT roll‐out*

• Yr of national ACT roll‐out is more appropriate to help learn lessons for AMFm

• We determined yr of national roll‐out from reports by ACTwatch, Global Fund, PMI, Ministries of Health

Analysis of ACTwatch Outlet & Household Surveys

*Source: Amin, A. et al. (2007) The challenges of changing national malaria drug policy to artemisinin‐based combinations in Kenya. Malaria Journal 6:72.

AVAILABILITY • Ranged from 6.5‐28% (1‐6 yrs)MARKET SHARE• Ranged from 0.9‐24.5% (1‐6 yrs)USE• Ranged from 2.4% to 19.3% (1‐3 yrs)PRICE• Large price differences between ACTs and other AMs

» ACT prices up to 40 x higher than mono‐therapies

Results of ACTwatch Surveys , 7 countries(1st line ACT, national ACT roll‐out as baseline)

“ The Evaluation….indicated very low levels of use of ACT thus far—5% or lower for all countries except Tanzania (20% in 2007) and Zambia (8‐15% in 2004‐2008). This finding is the most perplexing, showing the least improvement in coverage of the four primary malaria interventions.”

Global Fund 5‐Year Evaluation

Literature review on two social marketing (SM) models• NGO model (primarily used in low‐income countries)

• Focuses on population groups that cannot afford to pay commercially viable prices

• Donor subsidy to keep prices low • Additional supply chains set up by the NGO

• Manufacturer’s model (used in middle‐income countries)• Aims to be self‐sustaining without donor support• Uses a commercial company’s existing distribution channels

Commodity Social Marketing Programs

Source: Meekers D, Rahaim S (2005) The importance of socio‐economic context for social marketing models for improving reproductive health: Evidence from 555 years of program experience. BMC Public Health 5:10

PRICE• Price increases are associated with dramatic fall in use

AVAILABILITY• Water purification: 7.5‐13% at 1 yr• Condoms: 25‐39% at 4‐6 yrs

MARKET SHARE• Condoms/oral contraceptives: 10‐15% at 3 yrs

USE• SUZY Project, Bangladesh, a very similar model to AMFm:

» national subsidized commodity SM program with SIs» treatment of a major childhood illness» care‐seeking from private providers» aims to crowd out ineffective drugs (anti‐diarrheals, antibiotics)

Evidence on Outcomes in the NGO model

% Children receiving Zn

Baseline 11‐14 months 19‐23 months

City slum 4% 16% 19%City non‐slum 15% 26% 25%Municipal 7% 18% 17%Rural 4% 11% 12%

Zinc Usage Rates at 1‐2 yrs

Source: Larson CP et al (2009) Impact Monitoring of the National Scale Up of Zinc Treatment for Childhood Diarrhea in Bangladesh: Repeat Ecologic Surveys. PLoS Med 6(11): e1000175.

• Usage leveled off from 1yr to 2 yrs• Usage was higher in higher quintile wealth assets

8‐15% increase from baseline

Manufacturer’s Model of Commodity SM

• Few studies on the manufacturer’s model of commodity SM that can help guide expectations for AMFm Phase 1

• Only 1 study examined impact of model on metrics relevant to AMFm

» Study on socially marketed oral contraceptive pill (OCP) in Morocco» Increase in market share of 3% from baseline at 2 y and 12% at 10 y

• Manufacturer’s model tried without success in Africa (e.g. OCP in Nigeria)

Source: Agha, S. et al (2005) When Donor Support Ends: The Fate of Social Marketing Products and the Markets They Help Create. Bethesda, Maryland: Abt Associates Inc.

Global Data on ORT Scale‐Up

Source: Forsberg BC et al (2007) Diarrhoea case management in low‐ and middle‐income countries—an unfinished agenda. Bull World Health Organ 85:42‐48

• Global usage rate increased from 35% to 41% between 1986‐2003

• Annual increase of 0.39%

Introducing new drugs into developing country and emerging economy markets

» Northern drug manufacturers:» Market share of 10% at 1 yr and 20% at 2 yrs

» Drug manufacturers in India: » Market share of 4‐5% at 1 yr and 10% at 2 yrs » Higher expectations for ACT under AMFm: market share of 10% at 1 yr and 25% at 2 yrs

Results of Key Informant Interviews: Industry Expectations

• Time scale of 1‐2 yrs is short, especially to see changes in use

• Rural areas likely to show less uptake

• Uptake will depend on a range of factors, including:

» quality of supportive interventions and distribution systems, socioeconomic factors, regulatory frameworks, structure of private supply chains, malaria treatment‐seeking behavior, urban‐rural population ratio

• Small countries with supportive government are likely to achieve best results (e.g. see Rwanda’s national ACT subsidy program)

Dominant Messages from Other Key Informants




Overview

Mixed Methods Approach to Estimating Benchmarks

Data collection▪ Literature review▪ Collected unpublished data ▪ 33 key informants

Data appraisal▪ Summarized range of outcomes▪ To what extent do study conditions resemble AMFm?

Initial benchmarks▪ Estimated initial set of benchmarks▪ Discussed these with selected key informants

Input from AHC ▪ Received input at two AHC Meetings (June, Oct 2010)▪ AHC suggestions informed our report

Final estimates

Yr 1 Yr 2

Availability ▪ increase of 20 percentage points from baseline (QA-ACTs)

▪ increase of 40 percentage points from baseline (QA-ACTs)

Market Share ▪ increase in ACT market share of 10-15 percentage points from baseline

▪ fall in market share of artemisininmonoRx (AMT) from baseline

▪ increase in ACT market share of 15-20 percentage points from baseline

▪ fall in market share of AMT from baseline

Use ▪ increase of 5-10 percentage pointsfrom baseline

▪ increase of 10-15 percentage pointsfrom baseline

Price ▪ ACT price < 300% price of dominant non-QAACT (usually CQ, SP)

▪ ACT price <price of AMT (useful but not sufficient to determine success)

▪ ACT price < 150% price of dominant non-QAACT (usually CQ, SP)

▪ ACT price <price of AMT (useful but not sufficient to determine success)

Success Benchmarks (Mixed Methods Approach)

• AMFm Technical Design:

“AMFm….will be measured against its ability to reduce consumer prices of a treatment course of an effective coformulated AM from the current level of USD 6–10 to a far lower level of USD 0.20–0.50 (which is competitive with current retail prices of CQ and SP) for the majority of patients.”

• Thus our estimates are for a price relative to dominant non‐QAACT, usually CQ or SP (we also note that co‐paid ACTs should cost less than AMT—a “useful but not sufficient” benchmark)

• But: there are few empirical data to guide expectations for how quickly ACT prices will fall in the AMFm pilot countries

• Price change was the indicator with the weakest empirical basis for setting expectations at 1 and 2 yrs

The Challenge of Estimating the Price Benchmark

• ACT prices fell rapidly in AMFm pilots, more slowly in 2 national ACT subsidy programs (1 program: subsidy was largely passed on to consumers)

• Key informants: prices of co‐paid ACTs are likely to be high initially, then fall with time, especially with strong SIs (e.g. Daily Nationreports high initial prices in Kenya)

• Our benchmarks: <300% price of dominant AM (usually CQ, SP) at 1 yr, <150% price of dominant AM at 2 yrs (reaching parity beyond 2 yrs)

e.g. Nigeria: SP makes up 52% of all AMs sold/distributed and costs $0.54 (adult course). ACTs cost $6‐8. Our success benchmarks: ACT prices have fallen to under $1.62 by yr 1 and under $0.81 by yr 2

The Challenge of Estimating the Price Benchmark (cont’d)

Two Alternative Approaches to Estimating Success Benchmarks (introduced after peer review process)

Weighted each study:• Extent to which it resembled AMFm• Methodological rigor

Extrapolated outcomes to 1 and 2 yrs

Simple weighted mean approach

Aggregate weighting model approach using Monte Carlo simulations

(Stephane Verguet, IHME)

Step 1: Extrapolation of Available Data

• For both approaches, available data points were extrapolated to estimate the results at 1 and 2 years

• Extrapolation was necessary as these two quantitative approaches require a minimum number of data points—yet few studies measured outcomes at 1 and 2 yrs (e.g. Uganda pilot: usage was only measured at 1 yr)

• Extrapolation assumed a linear scale‐up model

Step 2: Weighting System

• Assigned each study a weight of 1‐4 ‘points’

• 1 point if study was national (rather than local)• 1 point if study examined a treatment (rather than preventive intervention)• 1 point if there was a price subsidy in place• 1 point for rigor (defined as randomization in a trial or random selection in a survey)

• Weighting involved a ‘judgment call’ so we also estimated a standard deviation (SD) for each weight based on our level of confidence in our weighting (SD captured the uncertainty)

Study Result1y

Result2y

Result3y

Result4y

Result5y

Wt(mean µ)

Wt(SD σ)

Pilot: Uganda 16% 32% 0.069 0.00865Pilot: Kenya 40.2% 80.4% 0.103 0.01285Pilot: Benin 4.5% 9.0% 0.103 0.01285ACTWatch: Madagascar 1.2% 2.4% 0.103 0.012875

ACTWatch:Uganda 3.52% 7.04% 10.56% 14.08% 17.6% 0.103 0.012875ACTWatch:Zambia 4.83% 9.65% 14.48% 19.3% 0.103 0.012875Survey (Simba 2010): Tanzania

18.8% 37.6% 0.069 0.008625

Zinc:SUZY trial: Bangladesh 10.3% 10.8% 0.138 0.0345Vitamin A:Survey (Zagré 2002): Burkina Faso

33.3% 66.6% 0.069 0.0345

ORT:Survey of 40 countries (Forsberg 2007)

0.39% 0.78% 0.138 0.01725

Example of Data Inputs: Studies on Usage

Shaded boxes show extrapolations. Data show changes from baseline

Step 3: Estimation of Success Thresholds

Weighted mean approach• We summed the weighted values and calculated a weighted mean for the 4 AMFm indicators

Monte Carlo Multivariate Sensitivity Analysis• Analysis by Stephane Verguet (IHME)• Assumed a normal distribution for the weights (mean = weight, SD = uncertainty)• 95% CI is + 2 SD• 1000 Monte Carlo simulations to estimate the aggregate uncertainty from the model weight inputs

Mean = wt

SD = uncertainty




Overview

Indicator Year 1 Year 2

Availability (%) 22.2 36.6Price (US$) 3.00 3.84

Market share (%) 11.8 21.9

Use (%) 11.8 22.2

Indicator Year 1 (95% C.I.) Year 2 (95% C.I.)

Availability (%) 22.3 (20.7‐23.8) 36.5 (34.3‐38.9)

Price (US$) 3.00 (2.79‐3.22) 3.79 (3.30‐4.26)

Market share (%) 11.8 (10.3‐13.5) 21.9 (19.6‐24.7)

Use (%) 11.8 (10.0‐13.4) 22.1 (18.5‐25.4)

Success Benchmarks Derived From 3 Approaches

Indicator Year 1 Year 2

Availability (%) 20 40

Price (US$) <300% dominant non‐QAACT <150% dominant non‐QAACT

Market share (%) 10‐15 15‐20

Use (%) 5‐10 10‐15

Weighted mean

Modeling approach

“Mixed methods” approach




Overview

• Balanced Scorecard

• Global Fund approach to rating grant performance

• Weighting systems

Approaches to Balancing Metrics of Success

AVAILABILITYbenchmark: 20 percentage point increase from baseline (QAACTs)

COUNTRY RESULT:

USEbenchmark: 5-10 percentage point increase from baseline

(QAACTs)COUNTRY RESULT:

PRICEbenchmarks: QAACT price<300% of price of dominant non-QAACT

and price of co-paid QAACT<price of AMTCOUNTRY RESULT:

MKT SHAREbenchmarks: 10-15 percentage

point increase from baseline (QAACT) and decrease in mkt

share of AMTCOUNTRY RESULT:

AMFm Phase 1ONE YEAR

CONTEXTUAL FACTORS:

Date that grant agreement was signed

Date that co‐paid drugs arrived in country

Date when SIs started, type of SIs

Other contextual factors (e.g. political instability, urban:ruralpopulation ratio,policy on regulation of outlets)

Baseline indicators (availability, use, price, market share)

AVAILABILITYbenchmark: 40 percentage point increase from baseline (QAACTs)

COUNTRY RESULT:

USEbenchmark: 10-15 percentage point increase from baseline

(QAACTs)COUNTRY RESULT:

PRICEbenchmarks: QAACT price<150% of price of dominant non-QAACT

and price of co-paid QAACT<price of AMTCOUNTRY RESULT:

MKT SHAREbenchmarks: 15-20 percentage

point increase from baseline (QAACT) and decrease in mkt

share of AMTCOUNTRY RESULT:

AMFm Phase 1TWO YEARS

CONTEXTUAL FACTORS:

Date that grant agreement was signed

Date that co‐paid drugs arrived in country

Date when SIs started, type of SIs

Other contextual factors (e.g. political instability, urban:ruralpopulation ratio, policy on regulation of outlets)

Baseline indicators (availability, use, price, market share)

The Project Team

Marco Schäferhoff Policy Analyst, E2Pi’s Berlin teamLead author

Dominic MontaguLead, Health Systems Initiative Project consultant

Gavin Yamey

Lead, Evidence‐to‐Policy initiative (E2Pi)

Global Health Group

University of California San Francisco

[email protected]

www.e2pi.org

Thank You

evidence to policy initiative (e2pi) to policy initiative (e2pi) ... the challenges of changing...

Documents