bss from surveillance to evaluation: advances and new uses

04/22/23Tulane University UTAP 1

BSS From Surveillance to Evaluation: advances and

new uses

Carl Kendall


Presented at the joint meetings of the World Federation of Public Health Associations and the Associação

Brasileira de Pós-Graduação em Saúde Coletiva/Abrasco,

Tuesday, August 22, 2006Rio de Janeiro


Evaluating the impact of the global effort to control AIDS Elizabeth Pisani et al. (BMJ 326:1384-7,2003)

have argued that the reason we aren’t doing well combating the global epidemic is that we don’t have better knowledge of incidence in different subpopulations; therefore the solution is better surveillance and modeling


What is the state of surveillance today? HIV prevalence data most common data

collected But as Ties Boerma argues, no gold standard

for this data, which is of highly variable quality (Lancet,362:1929-1931,2006)

T Diaz et al. argue for the need for enhanced surveillance:

HIV incidence HIV drug resistance Deaths due to AIDS Integrated analysis of data Behavioral surveillance Use of data for action (T Diaz, et al. AIDS 2005)


Behavioral Surveillance

2nd Generation Behavioral surveillance is critical: Where epidemic potential is highest Where risk is occurring Where “combined programs” are

reducing risk


What is second generation behavioral surveillance?

In the early 1990’s surveillance was in its early stages and focused on unlinked anonymous blood testing among sentinel groups. This is referred to as first generation surveillance.

It was soon found that: Measuring the prevalence of HIV does not provide all the

information needed to designing effective policy and programs. e.g.Without understanding HIV risk behaviors we do not know the potential for HIV to spread further

Without data on behaviors HIV prevalence data was difficult to interpret

Second generation surveillance was developed to improve surveillance systems.


Goals of Second Generation Surveillance

Monitors trends in behaviors in addition to HIV (early warning) and understand the behaviors that are driving the epidemic.

Increased focus on sub-populations at highest risk of infection.

Better use of surveillance data to plan prevention and care interventions (including integrating behavioral and biological data).


BSS and impact monitoring Continuous 2nd Generation BSS can:

Provide levels of risk behavior Track seroprevalence Collect some information about exposure

to program And BSS is proposed as the primary

tool for impact monitoring=impact evaluation


Most Some Few *All

Monitoring and Evaluation PipelineMonitoring and Evaluation Pipeline

Adaptation of Rehle/Rugg M&E Pipeline Model, FHI 2001Adaptation of Rehle/Rugg M&E Pipeline Model, FHI 2001

Input/Output Monitoring

Input/Output Monitoring

Process EvaluationProcess

EvaluationOutcome

Monitoring/ Evaluation

Outcome Monitoring/ Evaluation

Impact Monitoring/Evaluation

Impact Monitoring/Evaluation

Levels of Monitoring & Evaluation EffortLevels of Monitoring & Evaluation Effort

# of

Projects

# of

Projects

* Supplemented with impact indicators from surveillance data.

Realistic Expectations for M&E

Slide courtesy of Dr. Deborah Rugg, UNAIDS


BSS and impact monitoring Why no evaluation?

Large, complex, multi-intervention, multi-agency national programs

Traditional impact evaluation questionable:

Program uses proven treatment protocols Design issues: e.g. attribution Expense Ethical concerns


BSS and impact monitoring BSS data can be modeled:

EPP and Spectrum (WHO/UNAIDS) abcDIM (UNPOP) ASSA2002 (South African) AEM (Asian Epidemic model) HIVMM (HIV and TB) SPEHS (Pop. Dynamics) Populate (HIV and fertility) Baggaley (timing of introduction of therapy) GOALS (allocation of resources)


BSS and impact monitoring But, BSS models’ primary focus is:

Projections and estimates for epidemic No single model fulfills all requirements 5 of 11 can utilize some estimate of

program effectiveness, but this is only estimated effect


Limitations of Behavioral Surveillance Many surveys are one-off affairs, no system BSS may be conducted by contractors Linking test results and behaviors still

problematic Substantial proportion of reduction in

prevalence may be due to mortality Populations needing surveillance may not

be included Half of all Asian countries with BSS system do not

include MSM and IDU


Limitations of BSS

Sampling: Convenience samples often used to record high

risk behaviors New methods are available:

RDS Recent demonstration in 8 sites in Brazil

Available methods can produce probability samples if conducted correctly

TLS Cluster sampling


Experience in Brazil

Since 2004 the PN has supported an experiment in BSS 8 sites RDS (Fortaleza, Pernambuco,

Sao Paolo, Porto Alegre, Santos, Campinas, Curitiba and Manaus)

2 sites TLS (Pernambuco, Porto Alegre)


PN supported BSS sites in BrazilCampinas 1 MSM

Campinas 2 IDU

Curitiba Female CSW

Fortaleza MSM

Manaus 1 Female CSW

Manaus 2 DU

Pernambuco Drug Users (DU)

Porto Alegre CSWs

Santos Female CSW


Evaluating the global AIDS epidemic But what if the reasons that the

global program is failing is that we fail to evaluate program interventions?

We need rigorous outcome evaluations of intervention as well as surveillance data


Evaluating the global AIDS epidemic “To be meaningful, this analysis

must include issues of prevention coverage and effectiveness…” (Diaz et al. s5)

But no such system exists


Outcome evaluations

Are interventions working (e.g.): Stigma and discrimination Sexual violence and exploitation (incl. ovc) Sex worker interventions Drugs and risky sex Risk population mixing Condom use Partner reduction Abstinence Harm reduction STI

Not just ABC – can’t be decided politically


Outcome evaluations

Evaluations of programs Evaluation of synergistic effects Understanding community response


Characteristics of outcome evaluations Rigorous designs to measure effects But also rigorous application of

evaluation models Relate design to decisions (Habicht, Victora

and Vaughn, IJE 1999;28:10-18)

Use evaluation theories


Rigorous Study Designs

Experimental Quasi-experimental Ex post facto/ Non experimental Qualitative New methods derived from

econometrics Propensity scoring


Rigorous evaluation designs Example: steps in Utilization-focused

Evaluation: Stakeholder analysis Conceptualize outcomes Design and implement study Analyze findings and involve stakeholders Make decisions and write reports Evaluate the outcome-based management

system


Stakeholder analysis

Identify key actors and leaders Establish leadership group Commit to an outcome oriented

management system Agree on intended use Map out users and uses, set

priorities


Conceptualize outcomes

Select indicators Set targets (e.g. coverage,

effectiveness) Establish work team Establish and involve advisory group Engage line staff/workshops Finalize outcomes


Design and implement study Develop design, including analysis

and dissemination plan Pretest methods, instruments,

including review of available data Train staff and implement data

collection Collect data Monitor and supervise data collection


Analyze findings and involve stakeholders Prepare team:

Review, validate management uses, potential actions, such as decision options

Conduct training with managers on data use Analyze results to compare with baseline/

targets/other sources Identify additional information for interpretation Involve stakeholders in processing the

information With stakeholders, judge performance


Make decisions and write report With stakeholders, make management

decisions Identify audiences, make links between

internal and external use Revise and implement dissemination

plan Prepare versions of report for

audiences


Evaluate outcome-based management system Assemble review team to explore:

Process Use of information to make

management decisions, including policy Intended vs. actual use Expectations for evaluation system Criteria for success of the system Make recommendations


Summary

Not enough to have surveillance data to understand success or failure of programs

Rigorous outcome evaluations of program effects are required

Evaluation models are available that can be combined with conventional research designs to answer questions at the global, national, regional and local levels


Challenges

Contextual/structural factors and intervention synergies appear to influence outcome

A new hermeneutic of program and context/structure

Need to explore new methods/theories to capture these factors


Recommendations

Surveillance needs to continue to improve It is not enough for the community to argue

that impact monitoring is sufficient A major effort – on the scale of surveillance

- needs to be directed to evaluation: Reviewing estimates of program effectiveness Developing tools and training programs Developing new methods to open the black box

of intervention

bss from surveillance to evaluation: advances and new uses

Documents