harmonized monitoring and evaluation indicators for … · 7 where to focus in the in-depth...

Harmonized Monitoring and Evaluation Indicators for Procurement

and Supply Management Systems

Tracking the performance of PSM systems for ARVs, TB and Malaria medicines

Working Document for Field Testing

May 2009

2

AMD Partner Network

Table of Contents

Acknowledgement .....................................................................................................................3

1. Introduction...........................................................................................................................5

2. Rationale...............................................................................................................................5

3. Purpose .................................................................................................................................5

4. Development process............................................................................................................6

5. Target group..........................................................................................................................6

6. Key Supply Chain Issues and Related M&E Indicators......................................................6

6.1 Introduction ........................................................................................................................... 6

6.2 Components of the Medicine Supply Cycle and related M&E Indicators ........................... 7

6.2.1. Selection .............................................................................................................................................. 7

6.2.2. Prescription and rational use ........................................................................................................... 8

6.2.3. Quantification and Forecasting ...................................................................................................... 10

6.2.4. Procurement and Ordering: Prices and delivery time.................................................................. 12

6.2.5. Quality Assurance ........................................................................................................................... 13

6.2.6. Inventory control (manual or computerized) ................................................................................ 14

6.2.7. Availability of ARV medicines (or malaria & TB medicines) ...................................................... 15

6.2.8. Distribution....................................................................................................................................... 16

6.2.9. Human Resources ............................................................................................................................ 18

6.2.10. Funding Sources and Budgeting ................................................................................................... 19

6.2.11. Organization and Management .................................................................................................... 19

6.2.12. Policies, Legislation and Regulations ........................................................................................... 19

7. Annexes .................................................................................................................................. 21

Annex 1: Core Indicators for Monitoring the Performance of PSM Systems ...................................... 22

Annex 2: Supplementary Indicators for M&E of PSM Systems ........................................................... 27

8. References ...................................................................................................................... 36

3

Acknowledgement This document was developed with the active involvement of many national and international institutions over the past three years. We are grateful to the following participants of the WHO consultation on Harmonization of M&E requirements for ARVs procurement and supply management (PSM) systems, held in Geneva, 10-11 October 2005: • Hélène Degui and Alassane Ba (CHMP), Luca Li Bassi and Steen Stottrup (Global

Fund), Xenophon Santas (PEPFAR/OGAC), Henk W.A. den Besten and Charles Chiedza Maponga (IDA Solutions), Yasmin Chandani (JSI), Douglas Keene (MSH/RPM Plus), Bechir N'Daw (UNAIDS), Helene Moller (UNICEF), Bert Voetberg (World Bank), Wilbert Bannenberg (Consultant), Andrew Loke (Pharmaniaga Solutions), Ernest Rwagasana (CAMERWA), Martin Olowo Oteba (MOH Uganda), Nguyen Van Kinh (MOH Vietnam), Cao Thi Than Thuy (WHO Office - Vietnam), Emma Nelson Msuya (National AIDS Control Programme – Tanzania), Deus Bazira Mubangizi (Catholic Relief Services), Carole Presern (UK Mission), and the following WHO staff: Teguest Guerma, Jos Perriëns, Mary Couper, George Loth, Amolo Okero, Vincent Habiyambere, Cyril Pervilhac, Helen Tata, Jeff Sinden and Marco Vitoria.

We specifically would also like to thank: • the participants and facilitators of the IDA Solutions training workshops held in 2006,

2007 and 2008 on M&E of ARVs PSM systems. They provided valuable input on the indicators which were reduced and categorized into 12 core indicators and 30 supplementary indicators: Rumen Andreev, Viviane Leu, Hailu Tadeg, Abiy Andargachew, Akaki Lochoshvili, Naana Frempong, Sookdeo Singh, Sharmini Khalikaprasad, Konan Jules Yao, Clement Kouakou, Matebele Sefali, Masoko Nts’ekhe, Lerato Kholokholo, Doris Tshabalala, Remi Olaitan, Liliana Caraulan, Eric Nyiligira, Ivan Makumbi, Vladislav Volchkov, Bamikale Feyisetan, Thomas Wushe, Daniel Taddesse, Yohannes Tsegay Berhe, Mulusew Lijalem Belew, Abdulmalik Hassen Ebro, Nesrane Senbet Sahlemariam, Mehamed Feleke Tessema, Jurgen Hulst, Alemu Shiferaw Damassa, Grace Waiharo, James Batuka, Laurentiu Ionesii, Sushil Kumar Koirala, Ivana Lohar, Ishaya Dawha, Loic Aubry, Galina Bolshakova, Ludmila Reutskaya, Tifenn Humbert, Karin Wiedenmayer, Angela Taylor, Aminu Abubakar, Yakubu Adamu, Kgosiemang Atamelang Sidney, Olivia du Moulin, Elfatih Elamin, Amjad Idries, Jennie Lates, Okumu Morris, Mohamed Motwaly, Frederick Nicolaus Rubanga, Gashaw Shiferaw Mekonnen, Hendro Supaat, Jansen Susanna, Seyoum Temenit, Araia Berhane, Tesfaselase Ghedel, Habte Desbele, Elina Sverdlova, Albert Ntiringaniza, Be El Hassan, Félicienne Trévant, Jenny Jasmine Akondja Yandja, Carl François, Jean Claude Lundu Deka, Michèle Razanamparany, Tantely Rakotomalala, Marthe Everard, Jean-François Saint-Sauveur, Chiedza Maponga, Peter Graaff, Vincent Habiyambere, Cees Tuinenburg and Clarisse Morris.

• The members of the JSI/MSH/SCMS working group (Dana Aronovich, Timothy Williams,

Yasmin Chandani, Youssouf Ouedraogo, Naomi Printz [all from JSI], Bannet Ndyanabangi, Helena Walkoviak, Leila Akhlaghi [all from MSH] and Gary Bettger, Jennifer Mboyane, Niall Shanbhag and Sarah Levine [all SCMS].

• The following people who, during the French adaptation process allowed to improve,

both the English and the French versions: Carinne Bruneton (ReMeD), Charles Rambert (ReMeD) and Caroline Damour (GIP-ESTHER).

• Daisy Carandang, Dennis Ross-Degnan, John Chalker, Cyril Pervilhac, Yves

Souteyrand, Chika Hayashi, Dongbau Yu, Boniface Dongmo Nguimfack, Jean-Michel Tassié, Camellia Falcon, Loïc Aubry, Phillip Savio and Priya Emmart.

4

This document has been written by the following persons to whom WHO extends its acknowledgement: Henk den Besten, Cees Tuinenburg, Clarisse Morris, Jos Perriens, Peter Graaff and Vincent Habiyambere. The present version is a working document ready for field testing. Any comment to improve the next version should be sent to Dr Vincent Habiyambere at [email protected].

5

1. Introduction Tracking key aspects of procurement and supply management (PSM) and taking corrective action when required is a means to continuously improve the effectiveness of a programme or a system. This monitoring should cover different components of the PSM system. It must be noted that just monitoring does not improve the PSM system. M&E shows the strengths and the weaknesses of the system. If the measurements are below the target this should trigger the responsible person to investigate and correct the problem. This is how monitoring can assist in ongoing quality assurance of the PSM system. Trained human resources in sufficient number, financial and other resources for the effective implementation of the M&E system should be available. 5-10% of the programme costs are recommended to be allocated to M&E activities. 2. Rationale Monitoring and Evaluation (M&E) can be used for two main purposes. Firstly, M&E is used to regularly check and evaluate the processes and the results (outputs, outcomes and impact) of a programme and to find out whether progress is being made towards the targets and defined objectives. When M&E detects that the performance of any programme component is below the expectations, actions to prevent and/or to correct the problems should be initiated. Used this way, M&E is a tool for continuous improvement of a programme’s performance while at the same time facilitating reporting, accountability and transparency. Secondly, M&E is used to inform donors on the programme’s effectiveness and efficiency. Donors require information that justifies how the funds were spent and what were the achievements. M&E is also used to compare programmes in an objective manner. Programmes funded by various donors have specific objectives, which call for specific indicators. Indicators to measure the performance of various programmes may be different given the specificity of each programme and given the requirements of each donor. However, indicators to monitor the national PSM system should be common for all PSM programmes. This way, various programmes would contribute to the national M&E system with their data which are relevant to the national system. The indicators proposed in this manual are for monitoring and evaluation of the national system and should be part, as much as possible, of the M&E indicators of PSM programmes of different partners. Although focused on M&E of ARVs due to their importance in the treatment of HIV, these indicators can be used to monitor other essential medicines such as TB and malaria medicines. During various consultative processes, national programme managers actually recommended that we include these medicines. 3. Purpose The purpose of this document is to harmonize M&E indicators to measure the effectiveness and the performance of the national PSM system: national M&E system and donors are encouraged to use the same indicators to increase complementarity and additionnality on the national M&E agenda. This is in line with the Three Ones principle. As much as possible, preference will be given to indicators that can be measured using information that is already being collected or could be collected on a routine basis so that the burden on human resources is limited.

6

4. Development process In 2004 the AMD Partner Network and the AMD Secretariat in WHO/HIV Department recognized the need for the different donors funding HIV/AIDS programmes to harmonize their reporting requirements at country level in order to increase transparency, productivity and efficiency in various programmes. Since then WHO/AMDS consulted IDA Solutions to perform a desk study on reporting requirements including M&E indicators for ARV procurement and supply management. The results of the desk research were discussed at the harmonization meeting organized by WHO/AMDS in October 2005. Participants included representatives from central medical stores, ministries of health, WHO regional offices, and several international donors. Some of these organizations submitted their M&E indicators for different PSM aspects. Following the meeting, JSI and MSH as part of SCMS, were requested to compile key PSM indicators which could be used for continuous programme tracking as well as for donor reporting. This working group produced the initial list of 50 indicators which was used as a basis for further development and simplification In June 2006 and March 2007 the draft M&E indicators for ARV PSM system were discussed during IDA Solutions training sessions on Monitoring & Evaluation of Antiretroviral Procurement & Supply Management Systems. This training was attended by health staff involved in M&E and in ARV procurement and supply management for low and middle income countries. The participants and facilitators reviewed the 50 original indicators and made valuable suggestions on deletion of a few items and on selection of priority indicators based on their experience in the field. WHO, guided by the above input and feedback from other experts, finalized a list of 12 Core Indicators. After deletion of a few items, the remaining 30 non-prioritized indicators, here called "Supplementary Indicators" can be measured if more information is needed. Core and Supplementary Indicators are found in Annexes 1 and 2 respectively. 5. Target group The indicators described in the present document provide a practical tool for staff in charge of planning, management, implementation, monitoring and reporting on national PSM systems. This document can also be used by institutions and donors who wish to monitor the performance of the PSM programmes for which they are providing technical and/or financial support. 6. Key Supply Chain Issues and Related M&E Indicators 6.1 Introduction In order to guide the reader for a better use of the indicators, the difference between monitoring and evaluation needs to be clarified: Monitoring is the routine tracking of the key elements of program/project performance through routine record keeping, routine regular reporting or continuous established surveillance systems. Monitoring helps managers to take a decision in a timely manner. There is a need for consistency and harmonization in the data being collected and indicators being used for measuring trends over time and comparisons between programmes, regions, or countries. Monitoring and evaluation are related: monitoring contributes to evaluation and can inform the evaluator

7

where to focus in the in-depth evaluation. It is important to select a limited number of monitoring indicators that will be actually used by implementers and managers for decision making. There is a tendency to collect to many data which are not used at all and do not provide any added value for decision-making. This tendency should be avoided as it is time and money consuming and put unnecessary burden on scarce human resources. Evaluation is the episodic assessment of the changes in targeted results related to the programme. It is more difficult and more time and money consuming than monitoring due its methodological rigor required to avoid wrong conclusions. Evaluation attempts to relate programme outputs to outcomes in behaviors or disease impacts after a certain period has passed. Even if the programme is part of a collective effort, this kind of evaluation is very important. It helps the managers to determine the added value of the programme investment. Monitoring and evaluation take place at regular intervals: the interval is shorter for monitoring and longer for evaluation. Results are used for reporting from peripheral operational levels to central level or from central level to donors. Given the confusion met in categorizing indicators under input-process-output-outcome-impact framework, we will not use these terminologies in the indicators described below. It was found however that some definitions are useful to assist those who want to use this framework. Inputs refer to any resources (e.g. money, staff, time, vehicle, equipment and other resources) required to achieve results; processes are activities (e.g. training, supervision, etc.) implemented by the programme using these resources; outputs are direct results of the programme(e.g. number of staff trained, number of supervision visits, number of patients on treatment, and any other coverage measure); outcomes are direct positive effects of the programme such as change in behavior (condom use, appropriate prescribing, increased performance of trained staff, increased productivity, etc.) which ultimately would lead to long term disease impacts as reduction in HIV incidence, reduction in opportunistic infections, improved quality of life and reduction in mortality. Below you will find a description of each component of medicine supply cycle and the key aspects that could be monitored or evaluated. The most important aspects are covered by 12 core indicators. If potential problems in a particular PSM component are identified through the core indicators, supplementary indicators can be used to get more information about the problem and corrective measures can be implemented. Both lists of core and supplementary indicators are found in Annexes 1 & 2 respectively. The present document is ready for field testing. After the field testing, the current indicators listed below and in annexes 1 & 2 will be revised and finalized for publication. 6.2 Components of the Medicine Supply Cycle and related M&E Indicators 6.2.1. Selection Core indicator 1: "Percentage of procured ARVs (TB, malaria medicines and other suppliers) that are in the national ARV Standard Treatment Guidelines (STG)"

What it measures: this indicator measures the extent to which the ARV procurement is in line with ARVs recommended in national standard ARV treatment guidelines (STG). Rationale: ARVs which are not in the national STG are still found in some countries, particularly where the procurement activities are not integrated in the national system. Treatment recommendations are updated regularly given the development of new more effective and safer medicines, the availability of more generic ARVs on the market, the occurrence of resistance and severe side-effects. The World Health Organization

8

Department of HIV/AIDS updates the antiretroviral treatment guidelines every two years. It is important that national HIV/AIDS programmes refer to updated WHO ARV treatment guidelines when selecting medicines for their national ARV standard treatment guidelines in order to guarantee the best possible ARV treatment. In addition, the efficiency of procurement is improved if programmes only buy a limited number of nationally recommended medicines in sufficient quantities. Tools: The list of ARVs recommended in the national STG and the list of ARVs procured. Source of information: The information is collected at central level where the procurement of ARV is done: the list of ARVs procured is compared to the list of ARVs recommended in the STG. Calculation: Number of procured ARVs found in the national ARV STG ------------------------------------------------------------------------------------------------------------------------- x 100

Total number of procured ARVs Target: 100% Although this indicator is easy to measure, most countries may find it not useful as no ARV is procured outside the national STG. However, where discrepancy between procured ARVs and STG exists, this indicator is still relevant. This indicator can also be measured at facility level during supervision visits or surveys in countries where some facilities receive ARVs from several agencies which carry out parallel distribution activities outside the national distribution system. If it turns out that procurement is not consistent with the national STG, supplementary issues could be monitored such as the availability of updated national ARV STGs at central medical stores and all health facilities to ensure that staff order the right products. Another issue is the regular revision of the national STG and the number of ARVs in the latest national ARV STGs included in the national EML. The national STG should be regularly updated and the national EML should be synchronized with the STG. Recommendation should be made to regularly update the national STG and to harmonize both documents (STG & EML) to ensure that ARVs recommended in the STG are considered as essential medicines which are tax free in several countries. 6.2.2. Prescription and rational use Core indicator 2 : "Percentage of patients receiving treatment regimens which are in line with the national ARV STG" What it measures: this indicator measures the extent to which patients receive treatment regimens which are in line with the STG. Rationale: irrational ARV prescription/use may lead to the development of medicine resistance, with the consequence of shifting procurement to more expensive ARVs. Furthermore, irrational prescribing for instance a wrong quantity of ARV per prescription or unjustified second line ARV treatment can distort the quantities, creating the risk of stock out as consumptions will be disconnected from the quantification. In addition this irrational prescription could increase the cost of medicines to be procured. This is the reason why prescribing ART should be in line with national ARV Standard Treatment Guidelines.

9

Caution: due to several reasons, particularly ARV shortage, ARV prescribed are not necessarily ARV dispensed. Dispensing records are important to know the actually dispensed medicines. It could be interesting to measure the consistency between ARV prescribed and those dispensed by comparing medical records and dispensing records. Tools: requisition forms: when reordering ARVs at the central medical stores, health facilities report on their requisition forms, the number of patients by treatment regimen for quantification and procurement purposes. From this information, the most used ARV treatment regimen can also be measured. In some countries, the number of patients on the waiting list is also reported to show the unmet demand (the demand is real data, unlike the estimated need which is an estimate, sometimes not reached in several countries where the number of people tested for HIV is still low. Source of information: the information is collected at central level where the requisition forms are submitted by health facilities. Calculation: Number of patients with ARV regimens which are in line with the national STG ------------------------------------------------------------------------------------------------------------------------- x 100

Total number of patients on ARV treatment Target: 100% This indicator could be desegregated by treatment regimens to see the most commonly used treatment combination. Another indicator which looks at rational medicine use is the "Percentage of ARV prescriptions that were dispensed in full at health facility level in the last 12 months" measures the extent to which ART prescriptions are dispensed in full. Normally, all prescriptions should be dispensed in full. The targeted percentage for this indicator is 100%. This indicator is very difficult to measure as the required information for its measurement is not available and is difficult to collect. Reasons for not fulfilling a prescription are numerous and include ARV shortage, poverty of patients unable to afford the price of the whole prescription and irrational prescribing which would lead the pharmacist to modify the prescription by reducing the number of prescribed medicines if over-prescription is detected. This indicator requires a survey to collect data and investigate the reasons and disaggregate the results into groups according to reasons (ARV shortage, poverty, irrational prescription). If the prescription is not dispensed in full, treatment adherence is affected and resistance may occur. It is difficult to monitor treatment adherence as it requires survey. However, this should be possible if dispensers (and prescribers) are involved in the data collection provided that related information is recorded. Treatment adherence can also be monitored by recording patients who collect their refill according to the schedule and by the comparing the number of pills remaining to the expected number of pills (use of pill counts of remaining ARVs when patients return for refills). The adherence can be improved by informing patients on the importance of respecting the schedule for ARV re-supply, correct dose and frequency of taking each ARV, and the use of appropriate containers for ARV dispensing. The related indicator, which requires survey studies, is "the proportion of patients who take their ARV treatment without interruption 6, 12, 18, 24, 30, 36 …months after they started ART". Individual patient dispensing forms or an electronic dispensing software could be useful tools for such analysis. They should be recommended in all treatment sites. Dispensing forms can be used to measure for instance the percentage of patients who failed to collect their medication at the next due date during the last 12 months. These two indicators are much better than "proportion of patients who are still on ARV treatment 6,

10

12, 18, 24, 30, 36… months after initiation of their treatment" as this indicator may omit the frequently observed treatment interruptions which have several reasons sometimes independent from the patient (ex. ARV stock out). N.B. The results can be disaggregated in % of patients on 1st line treatment, 2nd line treatment, % females on treatment (data collection by sex) and % of children on treatment (data collection by age group). 6.2.3. Quantification and Forecasting Core indicator 3: "Ratio between procured ARV quantities and forecasted ARV quantities" What it measures: this indicator measures compliance to needed ARV quantities. As a target, the total quantities procured should be as close as possible to quantities forecasted unless there is evidence that forecast was not accurate or the budget to order the forecasted quantities and the needs have changed since the last forecast exercise. Rationale: Stock out with risk of interruptions of antiretroviral treatment should be avoided since this could lead to the development of resistance. Updated forecasts ensure that medicine quantification and procurement activities are based on accurate data and the risk of stock outs is avoided or minimized. The reverse, overstocking leading to expiry of products in the warehouse, should also be prevented by keeping forecasts as close as possible to the needed quantities of medicines. Otherwise funds will be wasted on medicines that cannot be used. During recent M&E workshops, participants mentioned that health facilities request quantities of medicines based on the real demand (quantification). Quantification is a calculation of ARV quantities based on the real number of patients who need ARVs (demand). Forecasting is an estimate of needed quantities of ARVs performed at national level based on various assumptions, such as the estimated needs, the capacity of the health system and the available financial resources. Tools: quantification/forecast report; annual quantities procured. Source of information: the information is collected at central level where forecasting and procurement are carried out. Calculation: Quantities procured for each ARV in the last 12 months -----------------------------------------------------------------------------------------

Quantities forecasted for the same ARV for the same period Target ratio: as close as possible to 1 Interpretation: a ratio which is >1 means that quantities procured were higher than needs and a ratio <1 means that the procured quantities were below the needs. In any case the causes and the consequences on stock levels need to be investigated. Core indicator 4: "Consumption rate: % of quantities consumed out of quantities procured" What it measures: this indicator measures if quantities of ARV were overestimated compared to the ARV demand (actual ARV needs). This leads to overstock with high risk of expired ARVs observed in several countries.

11

Rationale: assuming that procured quantities were consistent with forecasted quantities, it is useful to assess if the consumption was consistent with the quantities procured. Health facilities order quantities in line with the real number of patients who need ARV treatment. If the quantities procured by the central medical stores were overestimated, consumption will be very far below the quantities purchased, leading to overstock with high risk of expired medicines. Flexibilities and regular communication between treatment sites and the procurement programmes may increase a procurement which is tailored to the real demand. Exchange of ARVs between lower consumption rate treatment sites and higher consumption rate treatment sites is another strategy used in some countries: this flexibility of exchange of medicines between treatment sites allows to prevent stock outs in high volume treatment sites(high consumption rate) and to prevent expired medicines in low volume treatment sites (low consumption rate). If the quantities procured were underestimated at national level, then there will be stock outs in several treatment sites, or a high number of patients without treatment on the waiting list. Tools: annual consumption of ARVs quantities; annual quantities procured Source of information: the information is collected at central level where annual consumed ARV quantities and procured ARV quantities are registered. Calculation: quantities consumed for each ARV in the last 12 months ----------------------------------------------------------------------------------- x100

quantities procured for the same ARV in the same period Target %: as close as possible to 100%. It is also possible to measure the ratio between the procured quantities and the consumed quantities. In that case, the target ratio is 1; if the ration between the procured quantities and the consumed quantities is 2 for instance meaning a consumption rate of 50%, this would mean that the quantities procured were twice the quantities consumed. Interpretation: a consumption rate far below 100% means that quantities procured were higher than the demand, leading to overstocking and a high risk of expired ARVs in the stock. The number of patients on the waiting list and stock outs are indicators of unmet demand. Core indicator 5 : "Percentage of quantities (or quantities) of each ARV (TB, malaria medicines) lost per quantities procured in the last 12 months" What it measures: this indicator measures the quantities lost for various factors (expiry, damage, theft, diversion, etc.) compared to quantities procured. Rationale: the loss of medicines should be avoided as much as possible. It is essential that the loss is monitored to minimize it as much as possible over time. Tools: reports from central medical stores and from the treatment sites submitted to the central level. Source of information: the information is collected at central level where reports on ARV quantities consumed and lost are submitted. Calculation:

12

Total quantities(# of units: ex. pills) of each ARV lost in the last 12 months ----------------------------------------------------------------------------------------------------------------- x 100 Total quantities(# of units: ex. pills) of each ARV procured in the last 12 months Target: should be minimized up to a maximum of 5-7% This indicator could be desegregated by cause: expiry, damage, theft, diversion, etc. This desegregation is useful to assess the most important cause of loss. Comments from some field professionals indicated to us that it is sometimes difficult to know the denominator as the losses recorded this year may belong to quantities which have been procured more than one year ago, which makes the denominator difficult. In this case, it is essential to assess the losses in absolute value (quantities and monetary value). 6.2.4. Procurement and Ordering: Prices and delivery time Core indicator 6: "Ratio between the median price paid by the country for each ARV in the last 12 months to the median international price (GPRM)" What it measures: this indicator compares the obtained prices to those paid on the world market and measures the efficiency of ARVs procurement practices. Rationale: key issues in procurement of pharmaceuticals are product pricing and supplier delivery times. The prices obtained for ARVs and other commodities have a significant influence on the efficiency of the PSM system. In turn, the choice of a procurement method e.g. competitive bidding or direct procurement has an effect on the obtained prices. An efficient programme manages to get the best prices and reliable deliveries from suppliers. Tools: median GPRM price for each ARV per patient per year; prices obtained from several orders for each ARV during the year in a country, calculate the price per unit (ex. pill), calculate the annual cost treatment per patient for each order from each supplies, from all these annual treatment cost per patient, find the median price for all orders including emergency orders and patented medicines. As the unit price is too small to be meaningful, it is advised to use the median treatment cost per patient as this is what is found in GPRM by level of economy (low income, middle income and high income countries). Example from Country A for a fixed dose combination INN of ARV Strength

& Dosage

Form

Supplier Quantities per order

in 12 months

Value in US$

per order

Unit (ex. pill)

Unit price per order

Annual Treatment cost per patient

D4T30+3TC+NVP 80 D4T30+3TC+NVP 95 D4T30+3TC+NVP 100

D4T30+3TC+NVP 150 D4T30+3TC+NVP 200 Source of information: the information is collected at central level where procurement is carried out. Calculation: National median treatment cost per patient for each ARV based on various prices obtained for each ARV in the last 12 months

Comment [vh1]: Median price to be compared to the GPRM median price � this is the value to compare to the GPRM median price per treatment patient per year

13

-------------------------------------------------------------------------------------------------------------------------

Median GPRM treatment cost per patient for the same ARV during the same period Target ratio: <or= 1 Interpretation: if the ratio is > 1, this means that the national median price is much higher than the median international price (e.g. ratio=2 meaning that the country is paying twice the international price) for a particular ARV medicine (single or Fixed Dose Combination). The causes should be investigated e.g. were several quotations sought for a product? was the international competitive bidding used for multiple source ARVs as the main procurement method? or were there a lot of emergency orders for which price negotiation is not an option? In any case, corrective measures should be undertaken. Core indicator 7: Percentage of orders delivered to the central medical stores in full and on time as

stated in the procurement agreement in the last 12 months for each supplier. What it measures: this indicator measures supplier's performance in term of complying with the agreed delivery time and delivery of all quantities ordered. Rationale: If delivery times are longer than the agreed delivery times, or if the delivered quantities do not match the ordered quantities, this may lead to ARV shortage or to emergency order. Penalties for not complying with the agreements (lower quantities, late deliveries) should be included in the contractual agreement with the supplier. The delivery times can also be applied at district and facility levels if appropriate. In that case, this indicator measures the performance of the distribution system. The reasons for late deliveries need to be identified in order to prevent the problem in future deliveries. Tools: list of orders from each supplier during the year Source of information: the information is collected at central level where procurement is carried out. Calculation: Number of orders from each supplier delivered on time and in full as stated in the agreement in the last 12 months ----------------------------------------------------------------------------------------------------------------------------- x 100

Total number of orders from the same supplier in the same period Target : 100% of orders delivered in full and within the stated delivery time. 6.2.5. Quality Assurance Core indicator 8: Percentage of QC tested ARV batches which met the quality requirements or if no QC tests performed: % of ARV batches which met the national & international defined quality standards. What it measures: this indicator measures the extent to which procured ARVs meet the quality requirements. During a recent IDA Solutions M&E course, several countries reported that QC tests are performed on each ARV batch before release for consumption. Therefore, this indicator on quality control is easy to measure in countries where QC tests are performed on a regular basis before ARVs are released for consumption, otherwise quality standards such as WHO prequalification or USFDA/EU/stringent DRA approval will be applied.

14

Rationale: medicines dispensed to patients should be effective, safe and of good quality. In order to ensure that the procured products are safe and effective, certain quality standards should be met: Good Manufacturing Practice (GMP) certification, WHO prequalification, or FDA, EU, stringent DRA approval, registration by the national regulatory authority, correct packaging, label information and remaining shelf-life upon receipt, and other quality standards defined in the quality assurance policies. National drug regulatory authorities (DRA) will need to set their own quality standards and filter out the products that do not meet them. Tools: ARV products procured; list of WHO prequalified/USFDA/Stringent DRA approved ARV products Source of information: the information is collected at central level where quality standards are assessed by the national DRA. Calculation: Number of QC tested ARV batches procured which met the quality standards ------------------------------------------------------------------------------------------------------------------------- x 100

Total number of QC tested ARV batches

Target: 100% products supplied (tested) met the quality standards.

Products that failed the quality standards in particular those failing the QC tests need to be sent back to the supplier/manufacturer or destroyed on site and replaced at the supplier's cost. This should be reflected in the contractual agreement with the supplier. Quality control is done in addition to other nationally or/and internationally defined quality standards used to assess the extent to which procured products are of good quality. In addition to the QC test any observation made on product quality during inspection activities should be reported (Supplementary indicator No. 12). This could for instance pertain to product packaging and labeling or any other abnormalities observed during inspection activities of the pharmaceutical products delivered. A supplementary indicator on this topic has been included in Annex 2. If the remaining shelf life of received products is below the set standard, this can be measured by using another supplementary indicator in the QA section (Supplementary indicator No. 13). If there are problems with reporting product quality issues and it is unknown whether facilities are using a systematic approach, this could be measured by using a qualitative supplementary indicator i.e. percentage of facilities that have procedures in place to report product quality issues (Supplementary indicator No. 14). 6.2.6. Inventory control (manual or computerized) Core indicator 9: Percentage of health facilities that submit complete inventory control reports according to an established schedule in the last 12 months What it measures: this indicator measures regularity in reporting. Rationale: inventory control reports on stock management are useful management tools and important sources of information for monitoring the PSM system. It is important that inventory control reports are sent from the different levels to central level in a timely fashion to facilitate data analysis, reporting and decision-making. Tools: list of health facilities having submitted their inventory control reports

15

Source of information: the information is collected at central level where health facilities submit their inventory control reports. Calculation: Number of health facilities which have submitted complete inventory control reports according to the schedule in the last 12 months ------------------------------------------------------------------------------------------------------------------------- x 100

Total number of health facilities expected to submit their reports

Target: 100% of facilities submitting their complete inventory reports on time. This indicator is easy to measure as information is available and is useful for effective M&E system. It is also important that information registered in the system is consistent with the actual situation in the warehouse. "Percentage of products for which the physical counts and inventory records (stock cards or computerized data) match" measures the accuracy of inventory control and the quality of reported data i.e. whether inventory control data and actual physical stock data match. This indicator was not considered as a core monitoring indicator as it is only available in countries where supervisory visits are done on a regular basis and systematically. The indicator is measured using supervisory report information as the information is collected through physical counts carried out during supervision visits unless on site facility survey is conducted. The expected target is 100%. 6.2.7. Availability of ARV medicines (or malaria & TB medicines) Core indicator 10: " Percentage of health facilities dispensing ARV that experienced one or more stock-outs of at least one required ARV in the last 12 months.(to be adapted to other medicines: ex. TB medicines and ACTs) What it measures: This indicator measures the performance of the PSM system in term of ARV availability. Rationale: Stock out can lead to interrupted ART with a consequent treatment resistance. Therefore product availability is a key factor for any PSM system and must be ensured at all levels of the health system. Tools: ARV stock inventory control reports from health facilities; ARV requisition forms. Source of information: the information is collected at central level where health facilities submit their inventory control reports or their ARV requisition forms. Calculation: Number of health facilities dispensing ARVs that experienced one or more stock-outs of at least one required ARV drug in the last 12 months ------------------------------------------------------------------------------------------------------------------------- x 100 Total number of health facilities dispensing ARVs

Target: 0% facility experiencing ARV stock out ( or 100% of health facilities with no ARV stock out). This indicator is easy to monitor as this information is provided regularly to the central medical stores in order to receive new ARV supplies. It is useful indicator for assessing the performance of a PSM system. ARVs should be always available at the health facility level to ensure that all prescriptions are filled when patients need ARVs.

16

The objective of any PSM manager is to avoid stock out. It is essential to monitor on regular basis the stock in hand and the stock level of the treatment centre should never fall below the security stock in order to prevent the stock out. The stock level is being discussed further under Indicator 12 below. A related supplementary indicator is the duration of stock outs since this will show the severity of the stock out problem. The longer the duration of the stock out, the more severe is the situation (Supplementary indicator No. 15). 6.2.8. Distribution Core indicator 11: "Percentage of facilities which received all ARV orders in full and on time in the last 12 months" (to be adapted to other medicines: ex. TB medicines and ACTs) What it measures: this indicator measures the reliability of distribution system. Rationale: Timely distribution is important since it is related to product availability and the need for uninterrupted ART supply at health facility level. Tools: requisition forms from health facilities. Source of information: the information is collected at central level where health facilities submit their requisition forms. Calculation: Number of health facilities which received their ARV orders in full and on time in the last 12 months ------------------------------------------------------------------------------------------------------------------------- x 100 Total number of health facilities which received ARV orders in the last 12 months

Target: 100% of facilities. This indicator is similar to core indicator 7 discussed above but it measures two different issues: indicator 6 measures the performance of the suppliers while indicator 11 measures the performance of the distribution system. A deeper analysis of which facilities received all deliveries in full and on time, and which ones did not receive them on time can give more information on causes: for instance 10 facilities are expecting 10 deliveries each. 5 facilities received all deliveries in full and on time (100% deliveries received in full and on time for 5 sites) and 5 facilities had only 8 deliveries in full and on times (80% deliveries on time). The characteristics of these facilities for which deliveries did not arrive on time could be useful for instance treatment sites in remote areas with bad roads and during rainy season could explain this problem. In the other words, late deliveries could be internal e.g. the order was received too late compared to the standard delivery schedule or external e.g. bad roads or heavy rains. Core indicator 12: "Percentage of health facilities with all ARV orders made while the ARV stock on hand was within the minimum stock level in the last 12 months." (to be adapted to other medicines: ex. TB medicines and ACTs)

17

What it measures: this indicator measures effective use/application of inventory control tools in decision-taking. Rationale: the minimum stock refers to the lowest stock level you can allow to have, taking into account the delivery time and the average monthly consumption (AMC). If the ARV order is done when the stock level of an item is below the established minimum stock level, the risk of stock out is high. This indicator shows whether inventory control is being used properly or if product requests are sent to the procurement department too late, which can be a possible cause of stock outs. Tools: requisition forms from health facilities showing quantities requested, consumption and stock in hand. Source of information: the information is collected at central level where health facilities submit their requisition forms. Calculation: The security stock is a defined stock level for security purpose and varies with the size of the facility and the number of the patients. The minimum stock corresponds to security stock plus the quantities required to cover the patients needs during the delivery period. The maximum stock corresponds to the minimum stock level plus the quantities required to cover the patients' needs between 2 orders. The longer the period between two orders the higher the maximum stock level. As mentioned above, the objective of any PSM manager is to avoid a stock out. It is essential to monitor on regular basis the stock in hand and the stock level of the treatment centre should never fall below the minimum stock in order to prevent the stock out. Indicator 12 is crucial to ascertain the risk of drug shortage as this indicator assumes the regular monitoring of the stock level. In fact, in order to know which treatment sites have ordered when their stock levels were below the security stock, the treatment sites have to report the average monthly consumption (AMC) and the stock in hand (in quantity & in months). AMC= Total consumption during a defined period

------------------------------------------------------------- Total number of months for this period

The numerator should be corrected for any shortage which occurred during the defined period. The minimum stock varies between treatment centres and depend on the total number of patients on treatment and delivery time: the longer the delivery time, the higher the quantity and the higher the number of patients, the higher the needed quantities of medicines. Based on the length of delivery time, the minimum stock should be defined for each treatment centre. It is advised to monitor the stock level in term of "months" rather than in term of "quantities" because quantities vary with the AMC Stock level (stock in hand) in months = Current quantity stock in hand for each ARV

--------------------------------------------------------------

AMC for each ARV

The stock in hand should exclude all expired medicines. If the defined minimum stock is 2 months, and the value above is 1 month, the risk of stock out is high unless an emergency delivery is done. Having a lot of months in stock is not better neither: for instance if the stock level is 12 months and the ARVs expiry date will occur within 6 months, the risk of loss due to expired products is high and should be corrected by redistributing some quantities to other

18

treatment centres which need the medicines. This is the reason why to monitoring the maximum stock when medicines are received is as useful as monitoring the minimum stock level why the stock is about to finish. Calculation of Indicator 12:

Number of health facilities which had all ARV orders made in the last 12 months while the ARV stock level was within the minimum stock level for each ARV ------------------------------------------------------------------------------------------------------------------------- x 100

Total number of health facilities which made ARV orders in the last 12 months

Target: 100% of health facilities with orders done within the defined minimum stock levels. Finally, appropriate storage conditions e.g. warehouse temperature, cool storage for certain diagnostics, first expiry first out policy (FEFO), and secured storage for ARVs need to be maintained. This can be measured using the supplementary indicator on acceptable storage conditions, which need to be defined beforehand (Supplementary indicator No. 17). The indicators mentioned in the following sections are all supplementary indicators and are found in annex 2. 6.2.9. Human Resources An important issue in HIV/AIDS-related human resources is the presence of sufficient and appropriately qualified staff to enroll, treat and counsel patients on ART. Human resource shortage is a significant bottleneck of ART scale up in most developing countries. The first supplementary indicator of this section measures the percentage of facilities that exceed the recommended standard of ART patients per dedicated person to dispense ART prescriptions (Supplementary indicator No. 19). Another important aspect of human resource is the number of qualified staff allocated for M&E, completing various reports. When the number of staff allocated to these activities is inadequate, the burden becomes high and the quality of reported data and reporting rates or timely reporting becomes low. A second important issue in HIV/AIDS-related human resources is the level of staff training. Staff should be trained in various areas related to their responsibilities. Pre-service and in-service training in ART PSM is essential to staff. Refresher courses should be provided regularly e.g. once a year to maintain training impact and to provide updated information. The second supplementary indicator of this section measures the percentage of staff responsible for HIV/AIDS services trained in HIV related commodity management (Supplementary indicator No. 20).

19

6.2.10. Funding Sources and Budgeting Funding for ART can originate from national and international sources and should be used in efficient and transparent manner. Developing a national procurement plan for HIV/AIDS that has been approved by all partners is a first step in ensuring efficient use of financial resources and also in preventing the duplication of efforts. This plan should contain the budget, the quantities of products to be procured and the sources of funding among other things. Secondly, a mechanism for annual audits should be set up to ensure that funds are used appropriately and transparently. These aspects can be measured using the first two supplementary indicators of this section: the existence of a national procurement plan (Supplementary indicator No.21) and the existence of an annual audit of a Central Medical Store (Supplementary indicator No.24). Other issues that can be monitored include percentage of late release of funds from ministry of finances to procurement services (e.g. Central medical stores) and percentage of late payments to suppliers (Supplementary indicator No.22). Late release of funds by the government or partner may jeopardize the supply system. Late payments can result in penalties from the suppliers and the cause should be investigated after which preventive action can be taken. Finally, the level of operational costs can be monitored as an indicator of efficiency. For this supplementary indicator the total commodity value needs to be divided by the total cost of warehousing and distributing the commodities (Supplementary indicator No. 23). As with other quantitative indicators a target needs to be set so that an investigation is started if the measurements are below this target value. 6.2.11. Organization and Management Written standard operating procedures (SOP) and guidelines for the activities of the PSM cycle are required to ensure quality assurance throughout the PSM system. If standard procedures are followed there is little room for errors due to personal interpretation of the required actions and their frequency. Supervision will be required to make sure that SOPs are actually adhered to. Finally, guidelines to ensure that supervision is executed appropriately are also required. Supervisory visits should be conducted in a standardized way rather than them being entirely dependent on the person who undertakes them. Therefore the three supplementary indicators in this section pertain to the existence of written management procedures and tools (Supplementary indicator No. 25), the number of supervisory visits made compared to the number of scheduled visits (Supplementary indicator No. 26) and the existence of guidelines for supervisory visits (Supplementary indicator No. 27). 6.2.12. Policies, Legislation and Regulations The policy, legal and regulatory environment is meant to ensure safety of pharmaceutical products and equitable access to health services and treatment. Compulsory registration of pharmaceuticals is a way a government ensures that the products used in the country meet certain predefined standards. However, registration can take several months to several years, depending on the country. In view of the HIV/AIDS treatment urgency in many developing countries, a fast track registration system for ARVs should be set up. The existence of such a system is one of the supplementary indicators of this section (Supplementary indicator No. 28). Since the provision of ART services requires specific standards including human resources with specific knowledge, skills and capacity for effective ART PSM, it is important that a facility starting the provision of ART can refer to nationally agreed quality standards.

20

Therefore, supplementary indicator No.29 measures the existence of criteria for ART preparedness or a national accreditation programme so that the minimum requirements for quality ART services are guaranteed. Another supplementary indicator in this section is related to financial barriers that prevent equitable access to health services and treatment. If these barriers exist the government can opt to set up a policy on user fees and exemptions so that no one is excluded from HIV/AIDS services and treatment in the public health sector. The related indicator which measures the existence of such a policy is Supplementary indicator No. 30. Detailed information on the core indicators and on the supplementary indicators are found in Annex 1 and in Annex 2 below.

21

7. Annexes

Annex 1: Core Indicators for Monitoring the Performance of PSM Systems No. PSM stage Core Indicator Purpose/

When to use How to measure Target &

LEVEL Frequency Linked

Supplementary Indicators

Required information And Data source

1. Product selection

Percentage of ARVs procured that are found in the national ARV Standard Treatment Guidelines (STG).

To measure whether ARV procurement is in line with national ARV STG

Numerator: No. of ARVs procured found in ART STG Denominator: Total No. of ARVs procured

100% at central level

As often as necessary for intermediate reporting)

Annex 2, # 1, 2.

Required Info: National ARV treatment guidelines, EML, procured ARV list. Data source: Ministry of Health, National AIDS Committee, CMS or other national ARVs procurement agency if different from the CMS

2. Prescribing & Use

% patients receiving treatment regimens in line with the ARV STG.

To measure whether ARV prescriptions are in line with national ARV treatment guidelines

Numerator: No. of patients with ARV regimens in line with national ARV STG Denominator: Total No. of patients on ART

100% at Facility level but data can be found at central level in the periodic requisition forms

Annually (and as often as necessary for intermediate reporting)

None Required Info: National ARV treatment guidelines, ART prescriptions. Data source: Health facility treatment records, MOH

3.

Forecasting

Ratio between procured ARV quantities and forecasted ARV quantities for each ARV.

May be useful to measure accuracy of forecasting in countries where expired ARVs due to overestimation is an issue.

To measure the extent to which the quantities of ARVs ordered are consistent with the estimated needs (quantities of ARVs forecasted).

Numerator:

quantities procured for each ARV Denominator:

quantities forecasted for each ARV

Ratio = 1 at central level


Annex 2, # 7 Required Info: Ordered ARV quantities and forecast ARV quantities. Data source: MOH, National AIDS Programme, CMS and other places where can be found ARV Forecasts and procurement records

23

No. PSM stage Core Indicator Purpose/ When to use

How to measure Target & LEVEL

Frequency Linked Supplementary Indicators


4.

Consumption (Demand)

Consumption rate: % quantities consumed out of quantities procured

To measure the discrepancy between consumption and supplies: the extent to which the quantities of ARVs procured match the demand (consumption) i.e. quantities of ARVs correspond to the number of patients who need ARVs in the health facilities.

Numerator:

quantities consumed for each ARV Denominator: quantities procured for each ARV

100% at central level; a consumption rate below 100% Means a ratio>1 means overestimation or oversupply.

5. Loss Percentage of quantities of each ARV lost per quantities procured for each ARV in the last 12 months

To measure the loss of ARVs for various reasons(mismanagement): (1) expired products (2) damage, (3) theft and other types of diversion.

Numerator: Total quantities (number of units: pills) lost for each ARV in the last 12 months Denominator: Total quantities procured

5-7% Central level

6. Price Ratio between the median price paid by the country for a specific ARV to the median international price (medicine by medicine basis).

To measure the efficiency of the procurement practices: the national median price is compared to the median international price.

Numerator: Median price paid for an ARV over several orders. Denominator: Median international price for same ARV. GPRM median price for a given ARV medicine is the reference standard.

Ratio<or=1 meaning lower or at most equal to the median international price (e.g. GPRM, MSH) Central level


Annex 2, # 8, 9

Required Info: Prices of ARVs ordered ; procurement records; international Median ARV prices from the GPRM data base. Data source: MOH, National AIDS Programme, CMS and other places with ARV order records, WHO GPRM database.

24





7 Delivery time Percentage of (non-emergency regular) orders delivered in full and on time as stated in the procurement agreement in the last 12 months for each supplier

To measure supplier's performance in terms of complying with the agreed delivery time. The delivery times should be less or equal to the stated delivery times.

Numerator: number of non-emergency orders delivered in full and on time according to the agreement in the last 12 months for each supplier. Denominator: Total number of non-emergency orders from each supplier in the last 12 months.

100% orders Central level, District level and Facility level


None Required Info: procurement records; warehouse receipt notes. Data source: MOH, National AIDS Programme, CMS and other places with ARV order receipt notes.

8 Quality Percentage of products tested (or ARV procured) that met the quality control tests (nationally/internationally quality standards).

To measure product quality.

Numerator: Number of products that met quality tests performed by a WHO prequalified QC lab. Denominator: Total number of products tested by a WHO prequalified QC lab.

100% products meet the quality standards or 0% failure. Central level

Each supply none Required Info: national pharmaceutical standards; QC reports. Data source: QC Labs, CMS

9. Inventory control

Percentage of facilities submitting complete inventory control reports according to an established schedule.

To measure regularity and completeness of reporting.

Numerator: Number of facilities submitting complete inventory control reports according to schedule. Denominator: Total number of facilities required to submit reports.

100% Central, regional, provincial, district levels (all levels receiving reports)


none Required Info: Number of facilities submitting complete reports on time, total number of facilities in reporting area, reporting deadlines. Data source: MOH or CMS reports on reporting schedule,

25





inventory control reports and their submission dates.

10. Availability

Percentage of health facilities dispensing ARVs that experienced a stock out (or that had no stock out ) of one or more ARVs in the last 12 months

To measure the performance of the PSM system in term of ARV availability.

Numerator: No of facilities dispensing ARVs that experienced a stock out (or that had no stock out) of one or more ARVs in the last 12 months Denominator: Total No of facilities dispensing ARVs surveyed.

0% health facilities with stock out or 100% facilities without stock outs Central, district levels


Annex 2, # 16, 17, 18, 19

Required Info: Stock levels Data source: LMIS, stock cards and other stock management tools.

11. Distribution Percentage of facilities which received all ARV orders in full and on time in the last 12 months .

To measure the reliability of the distribution system: all health facilities should receive all deliveries on time to avoid ARV stock out.

Numerator: number of facilities that received all ARVs orders in full and on time within the last 12 months Denominator: Total number of facilities

100% Central, district levels


None Required Info: Minimum stock levels per product, stock levels at time of each supply was received, total number of products re-supplied during the last 12 months. Data source: stock cards and other stock management tools.

12. Security stock and inventory control

Percentage of health facilities with all ARV orders done at CMS while the ARV stock level on

To measure effective use of inventory control: ordering respecting the security stock is

Numerator:

number of health facilities which had all ARV

100% Central, district levels

Annually (and as often as necessary

Annex 2, # 19 and number of emergency orders in a

Required Info: Delivery schedule Data source: LMIS,

26





hand is within the minimum security stock level in the last 12 months

recommended to prevent stock out.

orders made while the ARV stock in hand was within the security stock level for each ARV Denominator:

Total number of health facilities submitting requisition forms

for intermediate reporting)

year. ARV receipt reports.

Annex 2: Supplementary Indicators for M&E of PSM Systems

No. Level in

the supply chain

Supplementary Indicator Definition Frequency Required information

Data source

Product Selection

1 Facility Latest copy of national STGs available and seen at dispensary at ART site.

Latest copy of national STGs (or facility formulary, if relevant) reported to be available and seen at health facility


STGs, EML, product list.

Observation at health facilities.

2 Central Percentage of listed ARV medicines in latest national STGs included in latest national EML

Numerator: Number of listed ARV products in latest national ARV treatment guidelines included in latest national EML Denominator: Total number of ARV products included in latest national guidelines


STGs, EML, product list.

National selection committee, Ministry of Health

3 Central National ARV resistance surveillance system exists and has reported results to national selection committee in last 12 months

National ARV resistance surveillance system exists and has reported results to national selection committee in last 12 months


Components of a functioning ARV resistance surveillance system.

National selection committee, Ministry of Health

Prescription, Use and Treatment adherence

4 Facility Percentage of patients who failed to collect their ARVs on due date Percentage of patients overdue in collecting their ARVs who were identified and appropriate action taken for follow-up. This indicator measures patient's treatment adherence

Numerator: Number of patients who failed to collect their ARVs in the last 12 months Denominator: Number of patients overdue in collecting their ARVs Numerator: Number of patients overdue collecting their ARVs who were identified and followed up promptly as


Definition of when a patient is classified as overdue. Procedure for action to be taken at pharmacy if patient is overdue in collecting ARV prescription, including timeline.

Patient-focused records, diary, dispensing records.

28

No. Level in the supply chain


Data source

and corrective action taken. It is an interesting indicator but very cumbersome. A less cumbersome way of looking at this issue but gives different information: "The proportion of facilities that track overdue patients routinely or the proportion of facilities reporting overdue patients"

"The proportion of patients who take their ARV treatment without interruption 6, 12, 18, 24, 30, 36 …months after they started ART".

per procedure. Denominator: Number of patients overdue in collecting their ARVs. Numerator: Number of facilities tracking overdue patients or reporting overdue patients. . Denominator: Number of facilities surveyed.

5 Facility Percentage of encounters where pharmacy staff members explained the dose and frequency of each ARV medicine to the patient

Numerator: Number of encounters where pharmacy staff members explained the dose and frequency of each ARV medicine to the patient/caregiver Denominator: Total number of encounters where pharmacy staff members issued ARVs to patient/caregiver


Prescribed dose and frequency

Data collection tools, observation of pharmacy staff members; supervisory check list

29



Data source

6 Facility Percentage of prescribed ARVs dispensed in an appropriate container with clearly labeled identity, with clearly labeled dosage instructions

Numerator: Number of prescribed ARVs that are dispensed in appropriate and clearly labeled containers Denominator: Total number of prescribed ARVs dispensed


Standards for appropriate containers; standards for clear labeling

Data collection tools, observation of dispensed ARVs

Forecasting

7 Central Percentage of forecasts and updates completed as per standard operating procedures

Numerator: Number of forecasting exercises completed in a time period Denominator: Number of forecasting exercises that were scheduled to be conducted in the time period


Forecasts Forecast records

Procurement

8 Central Percent of emergency orders issued in the last 12 months (emergency order is any unplanned order done to solve an urgent imminent shortage.

Numerator: Number of emergency orders Denominator: Number of total orders


Number of purchase orders issued; number of emergency orders issued

Transaction records

9 Central Proportion of the value of emergency orders issued in the last 12 months

Numerator: Total value of emergency orders (in money and quantities). Denominator: Value (in money and quantities) of total orders.


Total value of purchase orders issued; total value of emergency orders issued

Transaction records

30



Data source

10 Central Proportion of orders which were fully filled in the last 12 months

Numerator: number of orders which were fully filled in the last 12 months (orders where quantities received = at least quantities ordered) Denominator: total number of orders in the same period


Requisitions and issues at all levels (i.e., quantities of products ordered and quantities received) for a specific time period

Requisition and issue records; LMIS reports

11 Central Proportion of ARV products found in the stores which are listed in the latest national ART STGs

Numerator: Number of ARV products found in the stores which are listed in the latest STGs Denominator: Total number of ARV products found in the stores.


List of products in STGs

Inventory records

Quality Assurance

12 Central General product quality issues Any observation made on product quality outside of the QC test.

Each supply Product quality observations

Products upon arrival. contract with supplier

31



Data source

13 Central Percentage of items received where the minimum shelf life equaled or exceeded the shelf life specified on the purchase order.

Numerator: Number of items received where the minimum shelf life equaled or exceeded the shelf life specified on the purchase order. Denominator: Total number of items received.


Shipment invoices; shelf-life regulations per country

Purchase orders; transaction records

14 Central Percentage of health facilities that have procedures in place to report product quality issues.

Numerator: Number of health facilities that have procedures in place to report product quality issues. Denominator: Total number of health facilities.


System available to raise issues

Warehousing and Inventory Control

15 Central, District, Facility

Stock out duration: average percentage of time out of stock for ARV products during a specified time period.

Numerator: Sum of all days out of stock for each ARV product during a specified time period (e.g. 180 days or 365 days). Denominator: Total number days for the specified time period x total number of items Or Numerator: sum all days of stock out and divide by the number of items (=average stock out days) ex. (10+20+30)/3=20 Denominator: Total number days for the specified time period (ex. 180 days)


Designated stock levels, stock out data, review of stock cards, consumption data.

LMIS, stock cards, consumption records,

32



Data source


Product losses due to expired drugs, damage, theft and other diversion: Value lost per value received

Numerator: Value (quantities and financial) of product lost due to: (1) expired products (2) damage, (3) theft and other types of diversion. Denominator: Total value (quantities and financial) of product received.


Value of product lost due to theft, damage, diversion; total value of products received

Stock cards, transaction records, inventory records, any other stock management records


Percentage of storage facilities meeting acceptable storage conditions and handling procedures

Numerator: Number of storage facilities meeting acceptable storage conditions and handling procedures. Denominator: Number of facilities visited.


Checklist of acceptable storage conditions, SOPs.

Checklist of acceptable storage conditions, SOPs.

Distribution


Emergency order lead time Average duration of the time between date emergency order placed and when received.


Dates emergency orders were placed and received.

LMIS, goods receipt notes.

33



Data source

Human Resources

19 Facility Percentage of facilities that meet or exceed the recommended standard of ART patients per trained staff available to dispense ART prescriptions.

Numerator: Number of facilities which meet or exceed the recommended standard of ART patients per dedicated staff Denominator: Number of facilities surveyed


Number of staff, number of ART patients, national standard of ratio of staff/ART patients (ex. 10 qualified staff/1000 ART patients).

Dispensing records, staff schedule.

20 Facility Percentage of staff responsible of PSM of ARVs who have been trained in PSM

Numerator: Number of staff responsible of ARV PSM who have been trained in ARV procurement and supply management. Denominator: Total number of staff responsible of ARV PSM


Number of staff responsible of PSM of ARVs and other HIV-related commodities who have been trained in ARV/HIV related commodities procurement and supply management.; Number of staff responsible of PSM of ARVs and other HIV-related commodities

Ministry of Health, training officer, national training center, training reports

34



Data source

Funding Sources and Budgeting

21 Central A national consolidated procurement plan exists and has been approved/funded by all partners

A national consolidated procurement plan exists and has been approved/funded by all partners.


Fund allocation records; accounting books and records

Ministry of Health

22 Central Percentage of late payments to suppliers by the central medical stores

Numerator: Number of payments made late to suppliers. Denominator: Total number of payments to suppliers.


Procurement contract terms; invoices

Accounting records

23 Central Proportion of total commodity value (costs in money) to total cost of the central medical stores in the last 12 months

Numerator: total commodity value (costs in money) in the last 12 months Denominator: Total costs of the medical stores in the same period.


Operational costs: warehousing, distribution, transport

Medical store finance department

24 Central Existence of an annual audit of the Central Medical Store.

Annual audit conducted and results available.


Daily financial statements; accounting books and records

Ministry of Health

Organization and Management

25 Facility District Central

Existence of appropriate organizational and management procedures and tools

Written standard operating procedures and guidelines (e.g., manuals, job aids, standards) exist to carry out logistics responsibilities such as using LMIS forms, forecasting/ordering commodities, inventory management, storage, distribution, product selection, budgeting, supervision and staff development, etc.


Written Standard Operating Procedures for specific logistics responsibilities

Observation of written procedures and guidelines (e.g., manuals, job aids, standard operation procedures) at central level; observation of existence at lower levels

35



Data source


Percent variation between the number of supervisory visits made and the number of supervisory visits scheduled

Numerator: total number of scheduled supervisory visits - total number of supervisory visits made. Denominator: total number of scheduled supervisory visits.


Records of supervisory visits. Established schedule of supervisory visits

Supervisory records


Existence of guidelines for appropriate supervision system to support the operationalization of logistics activities

Written guidelines exist for routine supervision to support logistics operations at facilities.


Guidelines for routine supervision system that covers logistics activities

Observation of guidelines for routine supervision at central level; observation of existence at lower levels

Policies, Legislation and Regulations

28 Central Existence of a system for fast-track registration of ARVs in countries where national registration is required for HIV/AIDS commodities.

A system for fast-track registration of ARVs exists.


Components of a functioning fast track registration system.

Ministry of Health, Regulatory authority

29 Central Existence of criteria for ART preparedness or of a pharmaceutical facility accreditation programme for ART at national level.

ART preparedness criteria or an accreditation programme exist at the national level for ART programmes.


Criteria for ART preparedness or for national accreditation programme

Ministry of Health

30 Central Existence of a policy on user fees and exemptions for HIV/AIDS treatment and services.

A national policy on user fees and exemptions for HIV/AIDS treatment and services exists.


Components of a policy on health system user fees and exemptions.

Ministry of Health

8. References Presentations made in the meeting (available on http://www.who.int/3by5/amds/geneva_oct10/en/index.html):

� 4 WHO presentations: o Indicators for monitoring ARV treatment outcomes (URL) o George Loth: Patient monitoring (URL) o Marco Vitoria: Pharmacovigilance (URL) o Cyril Pervilhac: HIV Drug Resistance (URL)

� Xenophon Santas, U.S. President’s Emergency Plan for AIDS Relief Reporting Requirements and the Country Operational Plan and Reporting System (COPRS)

� Maponga CC et al. Presentation on "Desk Study Conducted in Preparation for

Discussion on Harmonization of National Reporting Requirements for ART Medicines Supply Management". IDA Solutions, Desk Study for AIDS Medicines and Diagnostic Services, October 6, 2005

• Yasmin Chandani. Presentation on "Reporting Requirement and Indicator

Harmonization for HIV Medicines and Diagnostics: a Review of PEPFAR, GFATM and Country Specific Requirements and Indicators". Prepared by JSI for AIDS Medicines and Diagnostics Services, JSI 28 September 2005

� Douglas Keene, Overview of Ethiopia ARVs and other Medicines Supply

Management Issues � Ernest Rwagasana, Donor reporting requirements for ARV’s and related drugs in,

Rwanda � Emma Nelson Msuya , Harmonization of Reporting Requirements: Tanzania � Martin Olowo Oteba , Harmonization of donor reporting requirements for

antiretrovirals and related drugs: Uganda (URL) � Nguyen Van Kinh , Accessing to ARV in HIV/AIDS care and treatment programme in

Vietnam (URL) Reference documents:

1. Maponga CC et al. Desk Study Conducted in Preparation for Discussion on Harmonization of National Reporting Requirements for ART Medicines Supply Management. IDA Solutions, Desk Study for AIDS Medicines and Diagnostic Services, October 6, 2005 http://www.who.int/3by5/amds/geneva_oct10/en/index.html

2. JSI. Reporting Requirement and Indicator Harmonization for HIV Medicines and Diagnostics: a Review of PEPFAR, GFATM and Country Specific Requirements and Indicators. Prepared by JSI for AIDS Medicines and Diagnostics Services, JSI 28 September 2005. http://www.who.int/3by5/amds/geneva_oct10/en/index.html

3. WHO Indicators for monitoring national drug policies: a practical manual. Second edition, WHO/EDM 1998

37

4. WHO. HIV Patient ART Monitoring Meeting. Report of 29-31March 2004 meeting at

ICC Geneva http://www.who.int/hiv/strategic/me/artmonitoring/en/index.html

5. The Global Fund to Fight AIDS, Tuberculosis and Malaria: Monitoring and Evaluation Toolkit: HIV/AIDS, Tuberculosis and Malaria. Annexes: Selected indicators for HIV/AIDS, Tuberculosis and Malaria. June 2004 http://www.theglobalfund.org/en/links_resources/library/evaluation_framework/

6. The President’s Emergency Plan for AIDS Relief: Indicators, Reporting

Requirements, and Guidelines for Focus Countries. Revised for FY2006 Reporting. July 29, 2005.

7. The WHO M&E Webpage, a good concentration of already done M&E work: http://www.who.int/hiv/pub/me/en/

8. World Health Organization. Monitoring and Evaluation Toolkit HIV/AIDS, tuberculosis and malaria June 2004 http://www.who.int/hiv/pub/me/me_toolkit2004/en/print.html

9. WHO, 2005: NATIONAL AIDS PROGRAMMES A guide to indicators for monitoring and evaluating national antiretroviral programmes http://www.who.int/hiv/pub/me/naparv/en/index.html

harmonized monitoring and evaluation indicators for … · 7 where to focus in the in-depth...

Documents