using m&e to guide scale-up of xpert mtb/rif in tanzania - m and e to guide... · using m&e...

Using M&E to guide scale-up of Xpert MTB/RIF in

Tanzania

What data, how to collect it, and what to do with it?

Heidi Albert, Victoria Harris, Saidi Mfaume, Welile Sikhondze, Basra Doulla

Global Forum of Xpert MTB/RIF Implementers, 2 May 2014

1

What is M&E?

Planning

Monitoring

Evaluation

Using the findings

Routine data

collection

Analysis of routine data

Collection and analysis of

additional data

Data analysis

Interpretive skills

Programme review process

Set objectives and targets

Standardised tools to collect

relevant data

Data analysis

Interpretive skills

2

2010 or earlier

2011

2012

2013/2014

Xpert testing sites – Tanzania

3

Some questions ….

How many instruments are in the country? Where are they?

Who put them there? When?

Where to place new instruments?

Have Xpert users received training? What curriculum? Were they certified as competent?

Who is responsible for conducting site assessments, training, site monitoring, collecting lab

indicators? Has it been done according to plans?

How do we ensure uninterrupted cartridge supply and avoid expiry of cartridges?

Who is responsible for equipment maintenance and calibration? When does it need doing?

What level of unsuccessful tests? What causes?

Has the rate of errors reduced since mentoring and refresher training was given?

Has use of Xpert increased TB and drug resistance detection rate compared with microscopy?

What proportion of patients diagnosed with Xpert are put on appropriate treatment within 7

days?

Are data reported to MOH?

5

Xpert M&E framework

Framework for Gene Xpert Programme Implementation

Overall objective: Set up and implement quality-assured routine use of GeneXpert in [x number] of sites in line with national strategy.

Intervention Objectively verifiable Sources and means of Assumptions Documentation

logic indicators of achievement verification

Specific Objective 1 MOH-led technical working group guiding

Xpert implementation

Technical Working Group operating. Terms of Reference & mandate Members Commitment

Expected Results MOH/CTRL steers implementation and

monitors operations

No. of meetings attended. Advice and

guidance actively used.

Minutes of meetings; Populated

Stakeholder Coordinated Sheet

Cooperation amongst stakeholders and tight

coordination and follow up.

Activities Means: Sources of Info/Costs Preconditions

MOH/CTRL convenes, chairs meetings &

provide manage the TWG

Meeting Venue, Participant

Accomodation, Refreshments

Country Consultant Salary/Fee Meet with Country MoH/CDC to establish

working programme

Provide tools for monitoring stakeholder

meeting progress

Human Resources or Technical

Assistance to develop tools or steer the

TWG

Monthly report Appoint Country Consultant

TWG TOR

Specific Objective 2

Review Country Lab Strategic Plan, prepare

or revise xpert implementation plan

Gene Xpert implementation and

operation is included in the national plan.

Record of discussions and the text

document

Expertise is available to draft the needed text.

Expected Results

Updated Country National Lab Strategic

Plan includes Xpert Implementation

Updated document finalised, printed and

distributed to stakeholders

Stakheholder meetings; MoH Cooperation from relevant stakeholders

Annual implementation plan drawn up

including M&E plan and budget

Annual Plan document MoH and Stakeholder consultations

reports

Phased implementation plan per site No of checklist items checked per site Site checklists, transport, Human

Resources

Diagnostic algorithm included in plans Plan containing Xpert TB algorithm Guiding documentation

Eligible suspects criteria determined No. of eligible suspects in site area Report

Activities Means Sources of Info/Costs Preconditions

Compile Xpert Implementation text to be

inserted into Strategic Plan

MOH/CTRL personnel, Strategic Plan

sample text

Meeting minutes MoH agreeing & committing to implementation

of Xpert Strategic Plan Sample Text

MOH/CTRL hold meetings for the process Venue, personnel, meeting specific

materials

Strategy document development

The National Lab develops the annual

implementation plan including m& e plan

and budget

National Lab Personnel Monthly reports Availability of funds

Draw up phased implementation plan per

site TB Lab strategic planning template doc

Personnel Country, Site Readiness TB Lab Strategic Planning Template

Doc

Specific Objective 3 Conduct site assessment in preparations of

[Number of sites] for Xpert installation

No. Sites passing minimum criteria on

Site assessment checklist

Site Visit Reports Availability of technical Assistance

Cooperation of all needed partners.

Expected Results [Number of sites] assessed, ready and

prepared

No. of sites ready for Xpert

implementation.

Checklists; Reports Xpert is procured and ready to be installed.


Develop site assessment schedule Site Assessment Checklist, Site Readiness

Checklist

Individual site assessment checklist Tools developed and ready for use Site Assessment Checklist, Site

Readiness Checklist

Estimate annual lab supplies and drugs

needed

lab supplies, drugs Site assessment report & schedule Willingness of personnel to participate, learn and

apply learning

Check and follow up on progress of

procurement according to plans for project

year

XpertInstrument and catridges Receipts, procurement docs Data is available on which estimates will be based

Check and follow up on progress lab

renovations and provide TA if necessary

Materials and supplies Follow on site assessment report Procurement process of instruments instigated

Conduct follow on site assessment after

renovations

National Lab personnel Report/Checklist fully complied with First Assessment results and recommendations

followed.

Develop Maintenance, Waste

Management, Supervision, Training plans

and tools for monitoring progress

National Lab personnel, guiding

documents

Discussion/Meeting records,

Developing & Completed plans

First Assessment results and recommendations

followed.

Prepare technical SOPs & clinical protocols National Lab personnel, guiding

documents


Developing & Completed docs

Skills and resources available.

Provide TA to develop registers & request

forms updates


documents


Developing & Completed docs


Conduct training and customisation of

SOPs/protocol for FIND supported sites.


documents

Training Evaluation,Participants

feedback, Completed docs


Provide TA/tools if needed for the Catridge

Supply System


documents


Developing & Completed protocols


Specific Objective 1: MOH-led technical working group guiding

Xpert implementation

Specific Objective 2: Review Country Strategic Plan, prepare

Xpert implementation plan

Specific objective 3: Conduct site assessment in preparation for

Xpert installation

Overall objective: Implement rational and quality-assured routine use of Xpert MTB/RIF in line with appropriate

national strategy

6

M&E framework

Specific Objective 4 Install Xpert Instruments and train a minimum of 2

persons per site

No. of personnel completing traning successfully

and working in the lab.

Instruments installed and in working order; Machines Procured and all preparations completed.

Expected Results [Number of Xpert instruments] installed and

operating in [Number of sites]

No. of sites where Xpert instruments have been

successfully installed and integrated into the

routine operation of the lab.

Reports Laboratory is ready

[Insert no. of people] people are skilled in the use

of Xpert and can operate it, in the context of the

lab routine.

No of people using Xpert in the laboratories. No.

site at which the data

collection/processing/analysis procedure has

been established. No. of staff understsanding

the data collection/processing/analysis

procedure

Training register, Lab reports Key people who will run the instruments are the ones

doing the training

Trainee mentoring and independent operation No. of staff being shadowed showing marked

progress towards becoming independent. No. of

staff members becoming independently skilled

in the process

Staff reports.


Install GeneXpert instruments and routinely used Gene Xpert instruments; Installation Checklist Reports, Checklists Instruments procuredInstallation Checklist

Conduct training of trainers Technical know how on how to install Training report/ participant feedback Labs are ready for use

Conduct site specific lab training Training Materials and packages; lab monitoring

checklist

Training report/ participant feedback Training Material prepared

Training Materials and packages; lab monitoring checklist

Conduct site specific clinician training Training Instructors; Clinicians Training Package Training report/ participant feedback Trainees identifiedClinicians Training Package

Implement maintenance & waste management

plans

Venues for training Monthly reports Plans finalised and circulated

Plans finalised and completed ready to be used Plans document

Establish Data Collection/processing /analysis

procedure or routine supervised by lab

advisor/expert per site.

Lab manuals, Standard Operating Procedures,

M&E tools

Lab reports Lab personnel has capacity

Lab manuals, Standard Operating Procedures; M&E tools.

Channel feedback, back to site Site Visit Reports, Gene Xpert Generated Error or

Results report

Trainees Participate in at least 3 data

collection/analysis/feedback cycles with

consultant/external advisor taking lead in activities

No. of staff being shadowed showing marked

progress towards becoming independent.

Report, Checklist Trainees complete training successfully

Carry out independently at least 3 data

collection/analysis/feedback cycles (3 months)

Xpert Lab Manual/Monitoring Checklist

Report, Checklist Trainees Commitment

Xpert Lab Manual, Monitoring Checklist

Train a minimum of 2 people in data

collection/analysis/feedback and MNE from MoH

No. of people trained Training register

Specific Objective 5 Conduct Monitoring No. of labs being visited Reports Training has been applied

and Supervision Visits No. of staff(ex trainees) being supervised Reports

Expected results Routine operation with regular supervision visits No. of visits conducted Visit/Monthly Reports Routine has been established


Implement supervision plan Supervision checklist Supervision plan finalised and ready for use

Conduct monitoring and supervision visits regularly Country FIND Consultant Monitoring visit reports Operations started already

Check that all sites participate in external quality

assurance programme

EQA guidelines Site Schedule and checklist Knowledge of EQA

EQA guidelines, Site Schedule and Checklist

Check that data from sites are collected and

analysed monthly

National Lab Personnel; Lab Indicators Reports Process started

Lab indicators

Monitor the use of new tools and templates

introduced

Templates and M&E tools Reports on usage Tools available

Templates, M&E tools

Specific Objective6 Programme Evaluation

Conduct reviews and evaluation at the relevant

project progress periods

Evaluation conducted; Sustainability Plan

developed

Reports Stakeholder cooperation

Expected Results

Evaluate quality of Xpert use and impact in country Number of items on quality checklist ticked off. Checklist Implemenation goes according to plan

Future perspectives for Xpert use in country are

determined and formalized


Conduct end of project evaluation or impact

assessment.

External evaluator Terns of Reference Evaluation consideration of ways to cover costs of treatment due to

improvements in diagnostics.

Monitor implementation plan. National Lab Personnel

Specific Objective 7 Programme Sustainability

Develop sustainability plan (including costing and

knowledge transfer)

Sustainable indicator/checklist achieved Sustainability Indicators; Sustainability

Checklist

Project has been implemented successfully

Sustainability indicators, Sustainability Checklist

Expected Results Develop Sustainability plan for key competency for

sustainable xpert implementation.

FIND/National Lab personnel Sustainability Plan document


Develop country or site specific sustainability

checklist

Country Consultant, Site handover checklist;

Sustainability Checklist Reports

Site has successfully installed GeneXpert and operating it

successfully ready to be independent. Site Handover Checklist, Sustainability Checklist

Specific Objective 4: Install Xpert Instruments and training

Specific Objective 5: Routine testing and conduct Monitoring

and Supervision Visits

Specific Objective 6: Programme evaluation

Specific Objective 7: Programme sustainability

7

Lab indicator monitoring (1) MTB and RIF resistance rates

� Overall test

consumption

� Instrument over or

under-utilised?

� Rate of MTB positive

� Rate of RIF resistant

� Variation in number of

tests per month

(possible stock out?)

� Variation in rates of

MTB and resistance?

8

Lab indicator monitoringUnsuccessful test rates

9

What data to collect?

Monthly

GeneXpert Reporting - MONTHLY / QUARTERLY STATISTICS

Site name site

District

Person compiling report

Reporting period (Month/Year)

Xpert results

# Xpert tests MTB- 123

# Xpert tests MTB+ RIF sens 52

# Xpert tests MTB+ RIFres 0

# Xpert tests MTB+RIF indeterminate 8

# error results 4

# invalid results 2

# no result 0

Total numbers

# total Xpert tests 189

# total successful Xpert tests 183

# total unsuccessful Xpert tests 6

# total Xpert tests MTB+ 60

# GeneXpert modules in use (GX4 = 4

modules etc)

Analysis

Error rate 2.1%

Invalid result rate 1.1%

No result rate 0.0%

Total unsuccessful test rate 3.2%

Xpert MTB positivity rate (Xpert MTB

positive/all successful Xpert tests)32.8%

Xpert RIF resistant rate (Xpert MTB+RIF

resistant/all Xpert MTB positive)0.0%

Maximum Xpert testing capacity per month at

this site (no. tests)320 assumes 4 instrument runs per day x average 20 working days per

month

% Maximum instrument capacity being utilised 59.1%

4

Fill in the yellow-highlighted cells. The other cells will populate

automatically.These data are the minimal recommended data to be collected from all

sites implementing GeneXpert. Additional data may be recommended Data should be collected and analysed on a monthly basis at each

laboratory. Data may be reported and analysed at the national level on a

Combined data for all instruments at one

site. All TB suspects tested

0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8 9 10 11 12

0

20

40

60

80

100

120

140

Month

Xpert MTB positivity rate (Xpert MTB positive/all

successful Xpert tests)

Xpert RIF resistant rate (Xpert MTB+RIF resistant/all Xpert

MTB positive)

Total unsuccessful test rate

# total Xpert tests

Stratified by HIV+, MDR-TB risk etc.

10

What data to collect?

# Xpert tests MTB- 1251 module error rate % errors Staff tests done error number % errors

# Xpert tests MTB+ RIF sens 434 A1 33 27 A 742 50 7

# Xpert tests MTB+ RIFres 28 A2 25 20 B 830 53 6

# Xpert tests MTB+RIF indeterminate 3 A3 49 40 C 195 9 5

# error results 124 A4 17 14 D 0 0

# invalid results 3 E 2 0 0

# no result

28

F

6 0 0

Total Tests 1871 G 1 0 0

% errors 7 TOTAL 1776 112 0

During on-site supervision visits

Collection of all data from archive since last visit – more in-depth analysis

11

Correct staff – trained and competent

Qualification requirements for testing personnel determined by MOH or other governing body

Standardised and documented M&E training tailored to responsibilities

• Test operators

• Advanced/Super-users (country experts)

• Regional/district supervisors

Competency assessment

• Including refresher training at pre-defined intervals

Ongoing mentoring and monitoring for M&E

SOPs up to date and available at point of testing

12

Standardised tools

Assessors trained in use of tools

Direct observation important

observing users, checking registers and documents, comparing

registers with data from instruments

Internal and external M&E

Integrated data management tools

Reliable data collection

13

Need for basic data skills at sites and district/regional level

Need for more advanced data skills at national level

Critical analytic skills

Interpretation of findings, review and framing of recommendations and corrective actions

Statistics and critical analytic skills

14

Ref: Gapminder World. www.gapminder.org/world-offline

Connectivity – cell phones and low but growing internet use (limited broadband access)

15

� Establish mechanisms for flow of information between users to supervisors to

regional and central focal persons to NTP management

� Systems approach – integrated data collection, automated basic analysis across

different tools, option for advanced data analysis

� Integrate into existing NTP/lab M&E activities

� Practical training in basic data skills

� Advanced data skills training for supervisors/ national focal persons/planning

� Partner coordination – develop and use single set of tools, report near real time

to MOH

� Leveraging connectivity to improve data collection

� Donors providing whole package for implementation, not just instruments and

cartridges

� Ongoing mentoring in M&E

Moving towards systematic M&E

16

On-site supervision

17

Ongoing routine monitoring and evaluation provides the evidence base to inform planning for scale up and

quality of Xpert MTB/RIF programme

M&E is not an end in itself, it must be linked to planning and using the findings

Summary

Planning

Monitoring

Evaluation

Using the findings

18

“I have been impressed with the urgency of doing.

Knowing is not enough; we must apply. Being willing is not enough; we must do”

Leonardo da Vinci

Acknowledgements

Centers for Disease Control and Prevention – funding and technical agency

NTLP

WHO

USAID

Laboratory and clinical staff at sites

Partners: AIDS Relief, Apopo, ICAP, Ifikara, KCRI, Medical Mission Institute,

MMRC, PATH, Pharmaccess, UMB/MGIT, World Bank, WRP

using m&e to guide scale-up of xpert mtb/rif in tanzania - m and e to guide... · using m&e...

Documents