Assessment of Program Evaluation Activities in Tuberculosis Control

Programs — United States, 2009–2010

Silvia M. Trigoso, MPHFellow, Public Health Prevention Service (PHPS)

TB Program Evaluation Network Annual ConferenceSeptember 21, 2011

National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention

Division of Tuberculosis Elimination

Today’s Presentation

Purpose Objective Methods Results Limitations Conclusions Recommendations Questions

Purpose of the Assessment

To understand the status, progress, and challenges in planning and

implementing program evaluation activities in TB control programs.

Objective

Assessment addressed 3 evaluation questions:

What is the evaluation focus area(s) reported in the CoAg and interim reports and is it in alignment with National TB Program Objectives and Performance Targets for 2015?

To what extent does the TB control program’s evaluation plan incorporate elements of the CoAg outlined evaluation guidance?

What are challenges and opportunities reported in implementing proposed PE activities?

Methods

Variables were created and database developed for data collection and management.

Data sources included grantees’ program evaluation plans and interim reports submitted to FSEB as part of the 2010-2014 CoAg.

Reviewed program evaluation sections submitted as part of the 2010-2014 CoAg FOA and interim reports submitted in August 2010.

Data analysis was conducted and results were obtained.

Results

No Focus Area Reported At Least 1 Focus Area0

10

20

30

40

50

60

70

10

58

Number of Grantees Reporting a Focus Area for Evaluation

in 2010 Interim Report(N = 68 grantees)

# of

Gra

ntee

s

Only 1 Focus Area 2 or More Focus Areas

Outside of Scope0

5

10

15

20

25

30

35

40

45

50

43

15

10

Number of Focus Areas for Evaluation Re-ported by Grantees in 2010 Interim Report

(n = 58 grantees)

# of

Gra

ntee

s

Evaluation Focus Areas Reported by Grantees, 2010 Interim Report (n = 58)

Completion of Treatment 6

TB Case Rate 4

Contact Investigation 24

Laboratory Reporting 1

Treatment Initiation 1

Sputum Culture Conversion 8

Data Reporting 2

Recommended Initial Therapy 2

Universal Genotyping 2

Known HIV Status 13

Evaluation of Immigrants and Refugees 4

Sputum-Culture Reported 3

Program Evaluation/Evaluation Focal Point

1

Human Resource Development 1

Training Focal Point 0

Focus Area # of Grantees

Contact Investigation/Elicitation

(22%)

Contact Investiga-tion/

Evaluation(27%)

Contact Investigation/Tx Initiation

(18%)

Contact Investigation/Tx Completion

(33%)

% Of Grantees Reporting Contact Investi-gation

by Contact Investigation Category, 2010 IR (n = 24 grantees)

Unchanged Focus Area67% (42)

Changed due to NTIP Data 11%(7)

Changed due to program evaluation representative

technical assistance 9% (6)

Changed due to program consultant technical as-

sistance 6% (4)

Changed due to limited staff/less labor intensive 5% (3)

Evaluation plan in development w/PET TA2% (1)

Change in Reported Focus Area for Evalu-ation,

2009 CoAg & 2010 Interim Report (n = 62)

Pre-Planning11% (7)

Planning16% (10)

Implementation45% (28)

Effects2% (1)

Cannot be De-termined26% (16)

Stage of Development for Program Evaluation, 2010 Interim Report

(n = 62 grantees)

Increased in Level Decreased in Level Unchanged0

5

10

15

20

25

30

35

40

6

35

21

Difference in Program Evaluation CoAg Guidance Reported in Interim Report

(n=62 grantees)

Gra

nte

es

Limited staff/resources for evaluation activities

Contacts' interest/ability to come to PH office for evaluation

TB Staff Turnover

Topographical & transportation barriers

TB cases low

Cultural & Linguistic competency

Patients' value and understanding of LTBI tx when asymptomatic

Social factors impacting TB treatment delivery

Discontinuation of TIMS

CI in detention centers (evaluation and tx of contacts)

Demonstrating PE as useful

Outreach to rural areas

Stigma attached to TB disease and LTBI/hinders follow-up & mgmt.

Decentralized PH system/lack of admin or regulatory authority to comply with nat'l objective

0 5 10 15 20 25 30 35

33

External Challenges Reported in Interim Report

(n =40 grantees)

Difficulty with data analysis

Health care provider/lack of TB education

Timely data entry

Record keeping/organization

Lack of experience in survey design

Lack of cooperation in data collection

Lack of experience with survey tools

Lack of participation in survey completion

0 1 2 3 4 5 6

Internal Challenges Reported in Interim Report

(n = 14 grantees)

Limitations

Assessment was limited to what grantees reported on submitted reports.

Not all grantees had program evaluation plans.

Could not obtain interim reports for all grantees.

Grantee’s interim reports did not reflect an update of PE activities conducted between reporting time period.

Conclusions The three most frequently reported focus areas for

evaluation are: (1) contact investigation; (2) known HIV status; and (3) sputum culture conversion.

Over 80% of grantees report at least more than 1 focus area. 10 grantees report a focus area not in alignment with National TB Objectives for 2015.

Over 65% of grantees did not change their focus area for evaluation. For some grantees who did change focus areas it was reported to be because of technical assistance provided by FSEB program evaluation representatives or program consultants.

There is a difference in PE CoAg guidance reported in interim report with most grantees decreasing in level.

The most frequently reported external challenge reported by grantees is limited staff and resources for conducting evaluation activities.

Recommendations

Stronger program evaluation reporting guidance is needed. Reporting formats vary across regions.

Identify areas for program evaluation team (PET) to provide TA.

Understanding challenges grantees face in implementing PE activities (e.g. , limited staff/resources + TB staff turnover) provides PET opportunity to be a valuable and useful resource.

Encourage use of monitoring tools to guide focus areas for evaluation and impacting TB programs (i.e. local data, NTIP, ARPEs).

Acknowledgements

Brandy Peterson & Awal Khan Program Consultants Field Services and Evaluation Branch Tom Chapel, CDC Chief Evaluation Officer

Contact Information

Silvia M. TrigosoPublic Health Prevention Service Fellow

Email: kje0@CDC.gov

For more information please contact Centers for Disease Control and Prevention

1600 Clifton Road NE, Atlanta, GA 30333Telephone: 1-800-CDC-INFO (232-4636)/TTY: 1-888-232-6348E-mail: cdcinfo@cdc.gov Web: http://www.cdc.gov

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

National Center for HIV/AIDS, Viral Hepatitis, STD , and TB Prevention

Place Division name here