Global Trends in HIV/AIDS Monitoring and Evaluation

Deborah Rugg, PhD Associate Director for Monitoring

and EvaluationHHS/US Centers for Disease Control and

Prevention (CDC)Global AIDS Program

Overview Overview

Background on HIV/AIDS M&E

HIV/AIDS M&E at the National Level

Trends in Global HIV/AIDS M&E

– The need for collaboration

– UNGASS reporting

– Monitoring and Evaluation Reference Group (MERG)

– The "3 Ones" Principle

Background on HIV/AIDS M&E

HIV/AIDS M&E at the National Level

Trends in Global HIV/AIDS M&E

– The need for collaboration

– UNGASS reporting

– Monitoring and Evaluation Reference Group (MERG)

– The "3 Ones" Principle

BACKGROUND ON HIV/AIDS M&EBACKGROUND ON HIV/AIDS M&E

Monitoring and Evaluation for Program Improvement

Monitoring and Evaluation for Program Improvement

ProgramImprovement

ProgramImprovement

Reporting/Accountability

Reporting/Accountability

ShareData withPartners

ShareData withPartners

What is the problem? ● Situation Analysis and Surveillance

What are the contributing factors? ● Determinants Research

What interventions and resources are needed? ● Needs, Resource, Response Analysis & Input Monitoring

What interventions can work (efficacy & effectiveness)? ● Special studies, Operations res., Formative res. & Research synthesis

Are we implementing the program as planned? ● Outputs Monitoring

What are we doing? ● Process Monitoring & Evaluation, Quality Assessments

Are interventions working/making a difference?● Outcome Evaluation Studies

Are collective efforts being implemented on a large enough scale to impact the epidemic (coverage; impact)? ● Surveys & Surveillance

Understanding Potential Responses

Monitoring & Evaluating National Programs

Determining Collective Effectiveness

ACTIVITIES

OUTPUTS

INPUTS

OUTCOMES

OUTCOMES & IMPACTSMONITORING

Problem Identification

A Public Health Questions Approach to Unifying SI/M&EA Public Health Questions

Approach to Unifying SI/M&E

Are we doing the right things?

Are we doing them right?

Are we doing them on a large enough scale?

Most Some Few*All

Monitoring and Evaluation PipelineMonitoring and Evaluation Pipeline

Input /Output Monitoring

Input /Output Monitoring

Process EvaluationProcess

EvaluationOutcome

Monitoring/ Evaluation

Outcome Monitoring/ Evaluation

Impact Monitoring/Evaluation

Impact Monitoring/Evaluation

Levels of Monitoring & Evaluation EffortLevels of Monitoring & Evaluation Effort

# of

Projects

# of

Projects

* Supplemented with impact indicators from surveillance data.

Strategic Planning for M&E: Setting Realistic ExpectationsStrategic Planning for M&E:

Setting Realistic Expectations

Project Level Indicators

Country Level Indicators

GlobalLevel

Indicators

M&E Indicator Pyramid:Levels of Indicators

HIV/AIDS M&E AT THE NATIONAL LEVELHIV/AIDS M&E AT THE NATIONAL LEVEL

Basic Outline for a National M&E PlanBasic Outline for a National M&E Plan

Introduction- overview of programs or interventions

Background Information: e.g., M&E resources—financial, human, other; roles and responsibilities

Logic Model/ Results Framework

– Problem statement

– Expected outcomes/impacts

– Indicators

– Multi-year targets (measurable objectives)

Operational definitions, sources, frequency of indicator data, method of verification/validation

Inclusion of plans for special evaluation studies

Introduction- overview of programs or interventions

Background Information: e.g., M&E resources—financial, human, other; roles and responsibilities

Logic Model/ Results Framework

– Problem statement

– Expected outcomes/impacts

– Indicators

– Multi-year targets (measurable objectives)

Operational definitions, sources, frequency of indicator data, method of verification/validation

Inclusion of plans for special evaluation studies

Multi-agency M&E Logic Model

Intermediate Outcomes

Long-TermImpacts

Outputs Short-Term Outcomes

Other

inputs

USG

inputs

World Bank

inputs

Government

inputs

Program

NAC/NAP

Program

Program

Program

Program

Program

Program

Program

Adapted from Milstein & Kreuter. A Summary Outline of Logic Models: What are They and What Can They Do for Planning and Evaluation? CDC 2000

Fundamentals of M&E Planning at the National Level

Fundamentals of M&E Planning at the National Level

National governments must believe in the value of M&E

Donors’ / development partners’ external assistance efforts are aligned with overall national or local strategies

Donor / partner funding is part of overall development funding

Effective coordination mechanisms between partners are essential

Transparency, trust and consultation between partners are essential

National governments must believe in the value of M&E

Donors’ / development partners’ external assistance efforts are aligned with overall national or local strategies

Donor / partner funding is part of overall development funding

Effective coordination mechanisms between partners are essential

Transparency, trust and consultation between partners are essential

TRENDS IN GLOBAL HIV/AIDS M&E– Current status of HIV/AIDS M&E

– UNGASS reporting

– Monitoring and Evaluation Reference Group (MERG)

– The "3 Ones" Principle

TRENDS IN GLOBAL HIV/AIDS M&E– Current status of HIV/AIDS M&E

– UNGASS reporting

– Monitoring and Evaluation Reference Group (MERG)

– The "3 Ones" Principle

Long-term approach to development planning and funding by key players such as government, donors

Linking of short, medium and long-term frameworks and strategies - including to budgets

M&E systems need to encompass much more complex frameworks and environments

Donor fatigue too many reports, too many terms, too little feedback, too little ownership of interventions

We spend lots of time on indicators, but Evaluation is often neglected; we need to strengthen Evaluation to better understand our programs

Inadequate analysis of results / understanding of what we are actually doing and what is working or is not working –, synthesis of what we learn from M&E, and adapting program practice accordingly

Need for harmonized and yet manageable M&E data systems

Long-term approach to development planning and funding by key players such as government, donors

Linking of short, medium and long-term frameworks and strategies - including to budgets

M&E systems need to encompass much more complex frameworks and environments

Donor fatigue too many reports, too many terms, too little feedback, too little ownership of interventions

We spend lots of time on indicators, but Evaluation is often neglected; we need to strengthen Evaluation to better understand our programs

Inadequate analysis of results / understanding of what we are actually doing and what is working or is not working –, synthesis of what we learn from M&E, and adapting program practice accordingly

Need for harmonized and yet manageable M&E data systems

Current status and Challenges of HIV/AIDS M&E

Current status and Challenges of HIV/AIDS M&E

Prior to this UNGASS report, we only compared country data

on HIV prevalence

Now there are standardized indicators for policies, funding,

services, coverage and risk reduction

Data coming directly from over 100 countries

70% of reports involved civil society, 50% involved people

living with HIV/AIDS

Prior to this UNGASS report, we only compared country data

on HIV prevalence

Now there are standardized indicators for policies, funding,

services, coverage and risk reduction

Data coming directly from over 100 countries

70% of reports involved civil society, 50% involved people

living with HIV/AIDS

United Nations General Assembly Special Session on AIDS (UNGASS)*United Nations General Assembly

Special Session on AIDS (UNGASS)*

*UNGASS Information courtesy Paul De Lay, Director for Evaluation, UNAIDS/Geneva (2004)

2003 UNGASS Survey2003 UNGASS Survey

103 countries responded out of 189

Progress seen in political commitment, improved policies, prevention efforts

Insufficient progress in human rights, human capacity, financial resources

103 countries responded out of 189

Progress seen in political commitment, improved policies, prevention efforts

Insufficient progress in human rights, human capacity, financial resources

Challenges/IssuesChallenges/Issues

Indicators are mainly for generalized epidemics

No indicators for blood safety and infections in hospitals

Indicators for IDU and behavior change in youth need improvement

No indicators for sex workers

Few countries could report on quality of STI treatment

Indicators are mainly for generalized epidemics

No indicators for blood safety and infections in hospitals

Indicators for IDU and behavior change in youth need improvement

No indicators for sex workers

Few countries could report on quality of STI treatment

UNAIDS established and international M&E standards setting group

Members from all UN co-sponsors and international agencies

Develops M&E strategy guidelines and international indicators as well as coordinates international M&E Technical Assistance activities

Meets annually (with sub-committees meeting more frequently) and generates M&E documents and other reports available on the UNAIDS website

Involved in monitoring the implementation of the “ Three Ones Principles”

UNAIDS established and international M&E standards setting group

Members from all UN co-sponsors and international agencies

Develops M&E strategy guidelines and international indicators as well as coordinates international M&E Technical Assistance activities

Meets annually (with sub-committees meeting more frequently) and generates M&E documents and other reports available on the UNAIDS website

Involved in monitoring the implementation of the “ Three Ones Principles”

Monitoring and Evaluation Reference Group (MERG)

Monitoring and Evaluation Reference Group (MERG)

The ‘Three Ones’Key Principles

The ‘Three Ones’Key Principles

1. One agreed HIV and AIDS action framework

2. One national AIDS coordinating authority

3. One agreed monitoring and evaluation framework

1. One agreed HIV and AIDS action framework

2. One national AIDS coordinating authority

3. One agreed monitoring and evaluation framework

The Third “One”Why is it better?The Third “One”Why is it better?

Data based on national needs rather than individual

donors

Production of high quality, relevant, accurate and timely

data

Submission of reports to international bodies under a

unified global effort

Efficient and effective use of data and resources

Allows synthesis of data from multiple sources

Greater transparency, coordination and communication

among different partners.

Data based on national needs rather than individual

donors

Production of high quality, relevant, accurate and timely

data

Submission of reports to international bodies under a

unified global effort

Efficient and effective use of data and resources

Allows synthesis of data from multiple sources

Greater transparency, coordination and communication

among different partners.

Principles for agreementPrinciples for agreement

One M&E unit which coordinates M&E activities

One multisectoral M&E plan

One national set of standardized indicators

One national level information system

Strategic information flow from sub-national to national level

Harmonized M&E capacity building

Collective responsibility and collective achievement (attribution)

One M&E unit which coordinates M&E activities

One multisectoral M&E plan

One national set of standardized indicators

One national level information system

Strategic information flow from sub-national to national level

Harmonized M&E capacity building

Collective responsibility and collective achievement (attribution)

Why isn’t available data used better?Why isn’t available data used better?

Data collection is fragmented

No single unit is responsible for compiling, analyzing and presenting data in a cohesive whole

No budgets for analyzing and presenting data

Underestimate skills and cost needed to present data effectively

Most M&E persons do not know how to use data well

Data collection is fragmented

No single unit is responsible for compiling, analyzing and presenting data in a cohesive whole

No budgets for analyzing and presenting data

Underestimate skills and cost needed to present data effectively

Most M&E persons do not know how to use data well

Key categories of informationKey categories of information

Biologic surveillance

Policy environment

Behavioral surveillance

Resource flows data Tracking commodities

Provision of prevention and treatment services and the coverage of

these services

Mortality and morbidity data

General health service performance

Evaluation research

Biologic surveillance

Policy environment

Behavioral surveillance

Resource flows data Tracking commodities

Provision of prevention and treatment services and the coverage of

these services

Mortality and morbidity data

General health service performance

Evaluation research

Challenges for Monitoring and Evaluating ARTChallenges for Monitoring and Evaluating ART

Need for short-term indicators (first two years of program implementation)

– 3-6 month intervals

– Including equity of access

– Survival at 6, 12, and 24 month time periods

Need to monitor long-term sustainability

– Resource flow tracking (sources, cost per unit services, costs per commodities)

– Staffing patterns

– Facilities capacity

– Systems infrastructure capacity

Need to monitor long-term impact

– ARV resistance patterns

– 5 to 10 year survival and quality of life

– System wide impact

– Impact on incidence

– Impact on economic productivity and social sector services

Need for short-term indicators (first two years of program implementation)

– 3-6 month intervals

– Including equity of access

– Survival at 6, 12, and 24 month time periods

Need to monitor long-term sustainability

– Resource flow tracking (sources, cost per unit services, costs per commodities)

– Staffing patterns

– Facilities capacity

– Systems infrastructure capacity

Need to monitor long-term impact

– ARV resistance patterns

– 5 to 10 year survival and quality of life

– System wide impact

– Impact on incidence

– Impact on economic productivity and social sector services

We need to mainstream M&E at all levels- district, state, provincial, national, and global. We need to do this in a credible way that includes primary users and focuses on outcomes that are meaningful to people on the front lines.

We need to mainstream M&E at all levels- district, state, provincial, national, and global. We need to do this in a credible way that includes primary users and focuses on outcomes that are meaningful to people on the front lines.

ConclusionsConclusions

In building on each other’s strengths, we must identify incentives and opportunities for collaboration, with the fundamental consensus that working together in a harmonized manner is better than going at it alone.

In building on each other’s strengths, we must identify incentives and opportunities for collaboration, with the fundamental consensus that working together in a harmonized manner is better than going at it alone.

ConclusionsConclusions