independent mid-term evaluation report...may 2016 – final version independent mid term evaluation...

May 2016 – Final Version

Independent Mid Term Evaluation Report

Health Education and Literacy Project for Women - (HELP for Women project) - in Narowal, Pakistan

i | P a g e M i d T e r m E v a l u a t i o n

Figure 1: Focus Group Discussion in Communities Bhattian Kalan

Picture on the Cover Page: Focus Group Discussions with Women’s Health Committee in Katho

Wali, Narowal District.

ii | P a g e M i d T e r m E v a l u a t i o n

AUTHORS

This independent mid-term evaluation report was commissioned by Feed the Minds and was supported by its partner National Rural Development Programme. It was produced by GLOW Consultants Pvt Limited Pakistan. For further information in regard to this evaluation, please contact Mr. Saeed Ullah Khan ([email protected]).

The contact point for Feed the Minds is Ms. Albha Bowe ([email protected]).

GLOW Consultants Private Limited

4th Floor, Software Technology Park-1, F 5/1, Islamabad, Pakistan Tel: +92 51 282 89 48 Fax: +92 51 282 89 49 Email: [email protected]

mailto:[email protected]



iii | P a g e M i d T e r m E v a l u a t i o n

PREFACE

The evaluation team would like to thank everyone who participated in and supported

the undertaking of this evaluation. This includes the communities the team visited as

well as the management and field staff of National Rural Development Programme.

EVALUATION TEAM

Mr. Saeed Ullah Khan (Lead Consultant)

Mr. Amjad Ali Khan (Consultant)

Ms. Azra Kausar (Consultant)

Mr. Zaki Ullah Khan (Consultant)

Dr. Ms. Salma Khalil (Technical Expert)

Mr. Muhammad Haroon (Map Designer)

Mr. Muhammad Ismail (Field Coordinator)

iv | P a g e M i d T e r m E v a l u a t i o n

Table of Contents

Authors .........................................................................................................................................................ii

Preface ........................................................................................................................................................ iii

Evaluation Team ......................................................................................................................................... iii

List of Key Acronyms ................................................................................................................................... vi

List of Figures ............................................................................................................................................. vii

List of Tables .............................................................................................................................................. vii

Executive Summary ................................................................................................................................... viii

Section 1: Background ................................................................................................................................ 1

1.1 Background Information ................................................................................................................. 1

1.2 project goal and specific outputs .................................................................................................... 3

1.3 Purpose of the Evaluation ............................................................................................................... 5

1.4 Overall Scope .................................................................................................................................. 6

1.5 Evaluation methodology ................................................................................................................. 6

1.5.1 Process and Evaluation Criteria ......................................................................................... 6

1.5.2 Evaluation Area and Techniques ....................................................................................... 8

1.5.3 Evaluation Questions ......................................................................................................... 10

1.5.4 Sample Size for FGD .......................................................................................................... 10

1.5.5 Limitations .......................................................................................................................... 11

SECTION 2: Overview of Findings .............................................................................................................. 12

SECTION 3: Progress Towards Outcomes ................................................................................................. 13

3.1 Women’s Health Committee .................................................................................................... 13

3.1.1 Selection & Composition: .................................................................................................. 13

3.1.2 Training & support received: ............................................................................................ 13

3.1.3 Activities: ........................................................................................................................... 14

3.2 Community Midwives ............................................................................................................... 17

3.2.1 Selection & Training: ......................................................................................................... 17

3.2.2 Feedback from Communities: ........................................................................................... 18

3.3 Health Literacy Facilitators ....................................................................................................... 19

3.4 Health Literacy Classes .............................................................................................................. 20

3.4.1 Learners and attendance .................................................................................................. 20

3.4.2 Coverage of Health Literacy Classes ................................................................................. 21

v | P a g e M i d T e r m E v a l u a t i o n

3.4.3 Learning from Health Literacy Classes .............................................................................. 21

3.4.4 Health Literacy Classes Impact on Vaccination and Diarrhoea ......................................... 22

3.5 Community Advocacy and Awareness ...................................................................................... 23

3.5.1 Advocacy Activities ........................................................................................................... 23

3.5.2 Awareness Raising Activities ............................................................................................. 24

3.6 Men’s Support Groups .............................................................................................................. 24

3.7 Other Programmatic Aspects: .............................................................................................. 27

Section 4: conclusions and recommendations ......................................................................................... 29

4.1 Conclusions ............................................................................................................................. 29

4.2 Recommendations .................................................................................................................... 31

4.2.1 Women’s Health Committees ............................................................................................... 31

4.2.2 Community Midwives ........................................................................................................... 32

4.2.3 Health and Literacy Classes ................................................................................................... 33

4.2.4 Registration of MSGs and WHCs: .......................................................................................... 34

4.2.5 Men’s Support Groups .......................................................................................................... 34

Annex 1 Evaluation Matrix .................................................................................................................. 35

Annex 2 LIST OF VILLAGES FOR COHORT 1 ........................................................................................ 39

Annex 3 Focus Group Discussions Guide ............................................................................................ 42

Annex 4 KEY INFORMANT INTERVIEWS ............................................................................................. 46

Annex 5 Evaluation Questions and Related KII/FGD Questions ......................................................... 49

vi | P a g e M i d T e r m E v a l u a t i o n

LIST OF KEY ACRONYMS

Antenatal care ANC

Basic Health Unit BHU

Civil Society Organisations CSOs

Community Management Skills Training CMST

Community Midwives CMs

Development Assistance Committee DAC

Feed the Minds FTM

Focus Group Discussions FGDs

Health Education and Literacy Project HELP

Health Literary Facilitator HLF

Information, Education and Communication IEC

Integrated Community Development Initiative ICDI

Key Informant Interviews KIIs

Leadership Management Skills Training LMST

Men’s Support Groups MSG

Millennium Development Goals MDGs

Multi Indicators Cluster Survey MICS

National Rural Development Programme NRDP

Non-governmental Organizations NGOs

Organisation for Economic Co-operation and Development OECD

Pakistani Rupees PKR

Postnatal Care PNC

Punjab Health Sector Reforms Program PHSRP

Tetanus Toxoid TT

Terms of Reference ToRs

Women’s Health Committee WHC

World Health Organization WHO

vii | P a g e M i d T e r m E v a l u a t i o n

LIST OF FIGURES Figure 1: Focus Group Discussion in Communities Bhattian Kalan ................................................ i

Figure 2: Evaluation Process ................................................................................................................... 7

Figure 3: Map of FTM Project Evaluation District ............................................................................... 8

Figure 4: Female / Male Ratio in FGDs .............................................................................................. 11

Figure 5: Interview with BHU staff in Narowal ................................................................................ 15

Figure 6: Picture Illustrating Fund Box and Record Registers ........................................................ 16

Figure 7: Receipt Book for Donation ................................................................................................... 16

Figure 8: Discussion with MSG in Katho Wali .................................................................................. 26

Figure 9: Discussion with MSG and Beneficiaries in Sarjal .............................................................. 27

LIST OF TABLES Table 1: List of Reviewed Villages in Narowal .................................................................................... 8

Table 2: Type and Number of Key Informants .................................................................................. 10

Table 3: Number of Different Evaluation Tools Used During Evaluation .................................... 11

viii | P a g e M i d T e r m E v a l u a t i o n

EXECUTIVE SUMMARY This midterm evaluation report covers the Big Lottery Fund supported programme - HELP (Health Education and Literacy Project) for Women with a total cost of £477,575. The project will take place over 3 years and started in April 2014. It is implemented in Narowal District in Punjab Province, Pakistan by Feed the Minds and its Pakistan based partner National Rural Development Programme. Overall this project provides a strong base for improving mother and child healthcare in Narowal. Of the five evaluation criteria, the project scored high on all five; relevance, effectiveness, efficiency, impact and sustainability. The project focuses on improving health conditions for mothers and children in 150 villages and supports access to health services. These villages are divided into three cohorts. At the time of the midterm evaluation, the first stage of activities in cohort 1 villages was concluded and activities in cohort 2 villages were beginning. The project selected 50 programme villages for cohort 1 in consultation with the government. 50 Women’s Health Committees (WHCs), each with seven members, were then selected with the active participation of the community. The Women’s Health Committees include representation from marginalised groups such as people with low literacy levels, minorities and the landless but until now with the exception of people with disabilities. These committees were provided with training on relevant issues. During the mid-term evaluation, the evaluation team found all 50 WHCs to be active and meeting at least once in a month. They had also collected approximately Pakistani Rupee (PKR) 80,000 and distributed PKR 45,000 amongst 20 poor pregnant women so enabling them to access health care. A second important feature of the programme is Community Midwives (CMs). In year 1, the WHC members, with the support of NRDP, selected 50 CMs. The 50 CMs were enrolled in a 30 day training programme split in 15 days training in two groups followed by additional training and follow-up. For cohort 1, the training was conducted by an NGO called Integrated Community Development Initiative that also trained NRDP staff as Master Trainers. All Community Midwives are given a midwifery kit, but until now no provision of regular supplies. The medical equipment is manually operated, creating challenges for the CMs. In all the evaluation villages, the team found CMs were providing advice and support to women and families. In total, according to NRDP data, CMs conducted 1,009 deliveries. Besides providing advice to the patients, the CMs were actively referring women to health professionals in hospitals and Basic Health Units. CMs were using smart phones to seek advice and conduct emergency referrals. A challenge identified was the software using English, since CMs were unable to write complex messages in English. Based on NRDP data, in total, CMs sent a total of 327 messages and had a response rate of just over 80% from doctors. According to the CMs they normally receive a response within thirty minutes to two hours after sending a text message. CMs are an effective and efficient way of providing medical service to mothers and children in the communities. The WHC members also selected 50 Health Literacy Facilitators (HLFs). These facilitators were provided with 27 days training and health education materials taking low literacy levels into account. Each health literacy class had 32 women students, seven of whom were Women’s Health Committee members, while the remaining 25 were women and girls from the wider community. These classes focus on teaching basic literacy skills and health knowledge. Using participatory teaching methods, the learners gain basic knowledge about mother and child

ix | P a g e M i d T e r m E v a l u a t i o n

health related issues and other health issues relevant to their communities. These measures will help in improving overall health conditions of women and children. Women’s Health Committees have until now led 155 advocacy efforts in the villages to promote mother and child health related issues. Advocacy efforts have led to improved mother and child health care services in villages. Further, WHCs have been actively engaged in awareness raising activities using information, education and communication materials. Government officials, students from health literacy classes and Men’s Support Groups participated in these events. Beside Women’s Health Committees, in the first year of the programme, 50 Men’s Support Groups (MSGs) with 20 members were formed in each village. The MSGs were provided with orientation on the project, their roles and responsibilities and coordination with WHCs. Further, they were provided with training on different topics relating to mother and child health. These MSGs also contributed financially to the work of WHCs. The contribution of MSGs is important as it facilitated women’s and children’s access to health services in the targeted communities. Research for the evaluation took place in December 2015 and January 2016. The mid-term evaluation is based on criteria developed by the Development Assistance Committee of the Organization for Economic Co-operation and Development. Along with a desk study/literature review and interviews with project staff, visits to project sites were conducted. Project sites were independently selected. Of the fifty villages in cohort 1, five villages were initially selected using stratified sampling. During the course of field work, an additional village was selected thus bringing the total sample size to six villages. A total of 19 Key Informant Interviews and 13 Focus Group Discussions (with 188 participants, 137 of whom were women) were conducted. Summary of Recommendations:

It is recommended to actively include women with disabilities in the WHCs;

The development of written criteria for the selection of vulnerable women to receive financial assistance will bring increased transparency to use of the communities funds;

Introducing smaller class sizes during training of Community Midwives will lead to increased hands-on training opportunities for CMs during the training;

Reviewing and supplementing the contents of medical kits for CMs (automated medical instruments pyodine and medical gloves) will facilitate the work of the CMs;

Introducing refresher training for trained CMs throughout the project will further enhance their skills and will enable them to exchange lessons learned with each other;

Establishing a delivery station / birth station within a village will help CMs both during medical check-ups and delivery processes;

Stronger engagement with doctors who are based in District Headquarter Hospitals and involved in referral will lead to improved follow-up of NRDP referred cases;

x | P a g e M i d T e r m E v a l u a t i o n

Resolving issues relating to use of the smart phones such as having English as the software language, difficulties charging phones and lack of phone credit would facilitate CMs referral work;

Extending the health literacy programme by introducing a second class in cohort 1 villages and in villages in subsequent cohorts;

Introducing flexible timing of classes to minimise drop out from health literacy classes and follow-up on individual students to resolve any issues that prevent them from attending;

Supporting the registration of MSGs and WHCs with government as CSOs to contribute to their long term sustainability;

Providing additional training in the area of mother and child health and other related issues for MSG members, equivalent to that received by WHC members.

1 | P a g e M i d T e r m E v a l u a t i o n

SECTION 1: BACKGROUND

1.1 BACKGROUND INFORMATION The maternal mortality rate in Pakistan has been reduced by 48% since 1990 from 490 deaths per 100,000 live births, supported by the introduction of a national Maternal, New-born and Child

Health Policy in 20051. The 2015 Save the Children report “State of the World’s Mothers puts Maternal Mortality Ratio (MMR) for Pakistan at 276 maternal deaths per 100,000 live births placing it at number 149 amongst 179 countries. The under-five mortality rate is 89 per 1,000

live births2. The Millennium Development Goals target of 140 maternal deaths per 100,000 live

births by 2015 was not met in Pakistan3. There are multiple causes of high maternal mortality in Pakistan such as haemorrhage (excessive bleeding), eclampsia (the onset of seizures in a woman with high blood pressure or other organ disorders), hypertensive diseases (a group of diseases including gestational hypertension and chronic hypertension), sepsis (a potentially life-threatening complication of an infection) and abortions (the process of ending of pregnancy by removing a foetus or embryo). Of these, haemorrhage is the most common. It is also important to note that haemorrhage often is preventable through timely intervention. Limited financial resources, lack of knowledge about health issues and services available, lack of health facilities and availability of skilled health personnel prevent women from seeking timely advice and assistance thus contributing to high maternal mortality. According to the business plan developed by FTM and NRDP for the HELP for Women - Health Education and Literacy Project, the figures for maternal and under-five deaths in Narowal District are 25% higher than the national figures (320 and 120 respectively). Lack of knowledge on health issues, distance to health facilities from the villages and availability of services in hospitals are amongst the reasons that contributed to higher mother mortality in Narowal. According to the World

Health Organisation4 (WHO) is lack of skilled care the main obstacle to better health for mothers. Lack of health education and low levels of literacy are also recognised as key factors in Pakistan. Narowal District has experienced several floods over the past few years making it one of the worst affected districts of Punjab Province. Heavy rains in Kashmir and other parts of Pakistan result in significant increases in the volume of water in the Chenab river and Nullah Dekh canal – two key water sources passing through Narowal district. For example during the rainy season in 2014, the volume of water in Nullah Dekh increased from its normal capacity of 31,500 cusec to 65,000 cusec. This resulted in damage to a dyke, flooding in Narowal city and 55 other villages5. Similarly, in 2013 and 2015, Nullah Dekh, exceeded its maximum water carrying

1 UNFPA Pakistan Country Profile 2010 2 Punjab Health Sector Reforms Program (PHSRP) – Rolling Plan 2010-2013 – Narowal 3 http://www.undp.org/content/dam/pakistan/docs/MDGs/MDG2013Report/UNDP-Report13.pdf

4 WHO 10 Facts on Maternal Health 2013 who.int/features/factfiles/maternal_health/en/index.html 5 http://www.dawn.com/news/1130395


capacity. As a result, dozens of villages were flooded in the district.67 In 2012 did the water level in Nullah Deik canal increased to three times its normal size causing floods in neighbouring villages8. These reoccurring disasters have severely affected the general population of Narowal District and created particular challenges for women, children and other vulnerable groups such as villagers losing their crops and have to rely on their savings and/or get loans for seeds and fertiliser for subsequent crops. In the case of reoccurring floods where villagers are unable to get produce from their lands for successive years and are therefore repay their loans they slowly move into a poverty trap. Secondly, floods affect public health infrastructure, especially at community level. According to the 2011 Health Facility Assessment – Punjab, Narowal 9 has one District Head Quarter Hospital, seven Rural Health Centers and 56 Basic Health Units. This is for a population of 1.42 million people with 88% of the population based in rural areas. Of the total population, 2.7% are under the age of 1 year, 13.4% are under 5 years and 41.97% are under 15 years. The same report indicates that in Narowal around 22% of women are between 15-49 years, 16% of married women are of child bearing age and 3.40% women are pregnant at any given time. 10 According to the Three Years Rolling Plan (2011-13) for the district, acute (upper) respiratory infections are the most common form of disease comprising 52% of the reported cases followed by scabies at 13%, diarrhoea / dysentery at around 11%, peptic ulcer disease at 6.5%, urinary tract infections and suspected malaria at around 2%. According to 2010 data, from the National Programme for Family Planning and Primary Health Care, the district has at least 1,208 Lady Health Workers who are covering over 72% of the total population in the district. Tetanus Toxoid (TT) vaccine coverage of pregnant women under this programme is at around 91%. At the same time, TT vaccine coverage is 93% and 50% amongst general population and pregnant women respectively, covered under the Expended Program on Immunization which is also a major government health initiative. The same plan puts the Infant Mortality Rate for Narowal district at 82/1000, Under 5 mortality at 120/1,000 Live births and Maternal mortality Rate at 320/100,000 deliveries. According to the Punjab Health Sector Reforms Program

(PHSRP) – Rolling Plan 2010-2013, in Narowal access to Antenatal care (ANC) and Postnatal Care (PNC) was 64% and 43% respectively. At the time, only 45% of the deliveries were attended by a skilled birth attendant. Recent data from 2014 Multi Indicators Cluster Survey (MICS) do however show some overall improvement both in Punjab as well as in Narowal. The low rate of deliveries attended by skilled birth attendants, despite the significantly high coverage by Lady Health Works Programme, can be linked with expecting women in the villages prefer Traditional Birth Attendants for their deliveries which highlights low awareness about alternatives.

6 http://nation.com.pk/national/12-Jul-2015/flood-plays-havoc-in-sialkot-narowal-villages

7 http://tribune.com.pk/story/591472/land-routes-to-narowal-inundated-as-nala-dek-gives-way-under-

flood-waters/ 8 Pakistan: Flood/Rains 2012 http://suparco.gov.pk/downloadables/Flood%20series-4(for%20web).pdf

9 pspu.punjab.gov.pk/system/files/HFA-Narowal.pdf

10 www.phsrp.punjab.gov.pk/downloads/3yrp/narowal.docx


To respond to these needs, Feed the Minds (FTM) and its Pakistan based partner National Rural Development Programme (NRDP) developed a project “Health Education and Literacy Project for Women’ (HELP for Women project) in Narowal, Pakistan”. This project is funded by the Big Lottery Fund UK and began in late April 2014. It will be completed by the end of April 2017 (implemented over a 3 year period). This project centres on the accomplishment of three of the United Nations Millennium Development Goals (MDGs). Particularly it concentrates on enhancing the health and survival rates of pregnant women (MDG5) and that of their children (MDG4) in Narowal District. The project also promotes gender equity and women’s empowerment (MDG3) as women beneficiaries will have improved social, economic and political participation in the society through access to resources and improved advocacy. By accessing improved resources through this project which are part of different activities, women would be in a better position to realise their rights thus leading to gender equality and women’s empowerment.

This independent mid-term evaluation was commissioned by FTM and was conducted in accordance with the Organisation for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) Criteria.

1.2 PROJECT GOAL AND SPECIFIC OUTPUTS

The NRDP/FTM project in Narowal is executed in a total of 150 villages in Narowal District. The project extends support to 8,100 community members to improve access to health services and make positive changes in health for a further 134,150 men and women. The overall project outcome is to increase access to skilled midwifery care and also improve women’s knowledge about their health and the health of their children, thus enabling them to adopt safer health practices and avoid life-threatening preventable conditions (i.e. anaemia, dehydration and sepsis) and diseases such as malaria and tuberculosis. The project-specific outcomes and outputs, along with corresponding indicators, are outlined below. Outcome and Output Indicators: 1. 4,800 Women and girls (aged 14-40) will have increased their access to existing health

services and will have adopted safer practices in terms of sanitation, nutrition, vaccinations and clean water, as a result of increased knowledge of reproductive, maternal and infant health through integrated health and literacy training. 1.1 Number of women and girls gaining knowledge on health education (disaggregated by

age, disability, HIV status) 1.2 Number of girls and women able to read basic health information and write own

personal information of self and family as a result of the health literacy classes 1.3 Increased family nutrition, preparation of clean water and sanitation 1.4 More effective management of preventable diseases among community members (e.g.

malaria, tetanus, scabies)


2. A network of trained local community midwives is established and actively supporting women and new born infants through anti-natal, delivery and post-natal care.

2.1 Number of villages with trained Community Midwives (CMs) 2.2 Number of deliveries undertaken by CMs in the community 2.3 Number of cases identified of high risk symptoms during pregnancy or childbirth

resulting in referral through Smart Phone technology linked to District Health Authority Doctors

2.4 Increased survival and well-being of pregnant women 2.5 Increased survival and well-being of new born infants

3. Self-supporting Women’s Health Committees (WHCs) coordinating health education and ensuring access to these by the most marginalised in the community 3.1 Number of WHCs established and actively coordinating Health Literacy Classes 3.2 Number of sensitization events in communities disseminating health education 3.3 Number of ‘hard to reach’ women able to access CM services through targeted support by WHCs 3.4 Number of new families practicing family planning

4. Through the Women’s Health Committees (WHC) advocacy activities; Civil Society Organisations (CSOs), local decision/policy makers will gain knowledge of the importance of maternal and child health contributing to an improved integration of these issue into existing services, policy and community life, leading to improved access to and a greater understanding of the need for maternal and child health care provision.

4.1 Number of local decision/policy makers actively supporting dissemination events on maternal and child health 4.2 Number of women and girls accessing existing health services with trained staff for non-emergency conditions 4.3 Number of villages receiving improved or additional health services from Government and Non-Governmental Organisation (NGOs) as a result of lobbying by WHCs and MSGs

Over three years, the project will establish 150 Women’s Health Committees (WHCs) across 150 villages: 50 WHCs will be selected each year (three cohorts in total). Each of the committees has seven local women as members. Three of these women are from the most marginalised groups like religious minorities (i.e. Christian communities), widows and women with disabilities amongst others. These WHCs receive specialised training such as project management, advocacy and lobbying skills. These committees are responsible for recruiting 150 Health Literacy Facilitators (one from each of the local communities) and coordinating health literacy classes. The facilitators receive 27 days training along with follow up and support from a master trainer engaged by NRDP. Each of the facilitators teaches a class of 32 women basic literacy skills and health knowledge for six months. The Women’s Health Committees will also engage 150 Community Midwifes (CMs), one from each of the village. The CMs are trained in groups of two groups of 25 students. They attend an initial training course of 15 days with two follow up training courses of 10 and 5 days, giving a total of 30 days training over a year. The training of Community Midwives was completed with active engagement of Integrated


Community Development Initiative (ICDI). ICDI is an NGO with experience in training Community Midwives and it was envisioned at the design stage that the project should capitalize on their experience and avail their services for the training of CMs. The CMs provide advice and support to women in their local communities on MNCH issues, challenge local myths about pregnancy, childbirth and infant health and encourage women to access existing services (pre- and post-natal). The CMs also refer cases that they assess as high risk to a doctor. Besides promoting mother and child health care related messages, WHCs are coordinating events to raise funds to support marginalized women in accessing health services. They are also lobbying local government leaders and NGOs for additional health services for their villages. In addition, 150 Men’s Support Groups will be established through the project. Their role is to assist WHCs in their activities and promote the role of men in improving health and access to health services for pregnant women and infants. They actively support the engagement of other men and boys in the community in fundraising and lobbying for women’s health. Further, community meetings were conducted WHC and MSG to promote issues around mother and child health care amongst the communities. Other project activities included knowledge sharing workshops by NRDP to disseminate best practise, successful techniques and learning from the project with external stakeholders. An example of such workshop was attended by District Coordination officers, District Officer Health, District Officer Population, District Social Welfare Officer, District Officer Finance, and local political leaders amongst others.

1.3 PURPOSE OF THE EVALUATION The consultants evaluated performance against intended results for each of its outcomes and outputs at the mid-term stage. The review questions were formulated in line with OEDC DAC criteria. These questions covered the core components of the project such as the activities of Women’s Health Committees, Health Literacy Facilitators, Community Midwives, Men's Support Groups and the Smart Phone Referral system amongst others. Specific evaluation questions are included in the evaluation matrix, which can be found in Annex 1. The learning questions highlighted in the Terms of Reference (ToRs) for the review were also considered in the design of the evaluation methodology. These questions included:

To what extent has the project met its objectives?

What are the lessons learned (works well and what does not work so well)? And why?

Are there any improvements in approach, methodology or activities that can be suggested within the project budget to improve the impact?

Are there any unexpected outcomes? Do they need to be addressed? If so, how can these are addressed within the project resources?

What is the level of stakeholder engagement?

How does the project rate in terms of relevance, sustainability, impact, community contributions and ownership, effectiveness and efficiency? What improvements could be suggested within the project budget?


1.4 OVERALL SCOPE

The consultants assessed the level of change/improvement achieved overall in terms of access to and quality of health services for pregnant women and girls. Some of the specific issues that were assessed were whether or not qualified health services and medical opinion is available to pregnant women and girls through CM’s and whether or not they have better information on health related issues. Further, the study assessed whether or not women have access to emergency funds established through the project by communities to avail access to health services and procure medicine in public and private hospitals. Similarly, the evaluation looked at ways in which men are supporting women in getting better health services and if these efforts are leading to better health for pregnant women and their children in rural communities in Narowal district. To make an informed opinion, the evaluation team used baseline data from the project Logical Framework and/or other available documents as part of its analysis to measure what improvements the programme has brought in the lives of target population. Data collected during the evaluation was used to identify and document the major gaps, challenges and lesson learned to-date.

1.5 EVALUATION METHODOLOGY

1.5.1 Process and Evaluation Criteria In line with FTM specifications, the evaluation team addressed the following OECD Development Assistance Committee (DAC) criteria:

Relevance Efficiency Effectiveness

Sustainability Impact The process or sequence of activities followed for the mid-term evaluation is outlined in the diagram below.


Figure 2: Evaluation Process

Briefing

• Discussions/Meeting FTM/NRDP Network Partners in Islamabad/Over skype.

Desk Review

• Final Review of Proposal, Beneficiary Data, Progress Reports, Other relevant Reports etc (already shared with GLOW team)

Inception Meeting

• Agree on Final List of Key Stakeholders and Explain any remaining questins on Evaluation Methodology and data Collection Tools

• Finalise the Timeline

Field Travel and Interviews

• Travel to the field and conduct interviews with relevant team members

Field Data Collection - FGDs Interviews

• Focus Group Discussions

• Interviews with government officials, partners and/or cluster/working group leads

Debriefing

• Debrief workshop with FTM and/or NRDP

Draft and Fianl Report

• Draft Report

• Final Report based on FTM Feedback


1.5.2 Evaluation Area and Techniques The evaluation took place in Narowal District of Punjab Province in Pakistan. Villages visited during the evaluation are listed in Table 1.

Table 1: List of Reviewed Villages in Narowal District Narowal

Sr# Village Name Cluster Union Council Tehsil

1 Bhattian Kalan Hakeem Pura Langah Shakargarh

2 Katho Wali Goralla Goralla Shakargarh

3 Sarjal Mehlowala Ghona Shakargarh

4 Khara Megah Jasar Nedoky Narowal

5 Sahdoky

(Mandihali)

Murali Dongia Narowal

Please refer to map below indicating the district’s geographical location. Figure 3: Map of FTM Project Evaluation District


The methods used to collect data for this review were as follows:

Meetings / interviews with FTM and NRDP staff.

Desk study / literature review, review of secondary data and documents, including proposals, logical framework, beneficiary data etc.

Field visits – The consultants visited 12% of the project villages where activities were implemented in Year 1/Cohort 1. (Refer to Annex 2 for a complete list of cohort 1 villages). A list of the 50 project villages was provided by the NRDP team. These villages were divided into five different clusters based on their geographical location and one village was selected from each of the clusters. Each of the villages in the sample size had an equal chance of been selected for the survey. The 5 villages selected and visited during the review were: Bhattian Kalan, Katho Wali, Sarjal, Khara Megah and Sahdoky (Mandihali). One additional village, Purana Wahla in Jasar cluster, was also visited by the evaluation team and one male and two female FGDs were conducted there. This was because some government officials were not available to meet during the day. The team therefore carried out the additional field work in the day time and met with government officials in the evening. In all six villages, the evaluation team met with and interviewed core beneficiary groups including WHCs, MSGs, HLFs and CMs.

Semi-structured interviews were held with 19 key informants as follows:

Women’s Health Committee (WHCs) members (4),

Men’s Support Groups (MSGs) members (2),

Health Literacy Facilitators (HLFs) (3),

Community Midwives (CMs) (3),

NRDP staff (2),

Government Officials from District Offices and Health Units (3),

Other community representatives including a local political leader and a local landlord (2).

More information is provided about the key informants in Table 2 below. Focus Group Discussions (FGDs) were conducted with members of the WHCs and MSGs, CMs and other members of the communities. Most of the FGD participants were active participants in the project and engaged in its different activities. Some indirect beneficiaries (around 10% of the total participants) were also included in the FGDs: even though they were not members of the committees established through the programme their views and experiences of the programme were still valid. More information is provided about the FGDs in Section 1.5 below.


Table 2: Type and Number of Key Informants

S.No Type Number Village Names

1 HLF 3 Bhattian Kalan Katho Wali Khara Megah

2 WHC 4 Bhattian Kalan Katho Wali Khara Megah Purana Wahla

3 MSG 2 Katho Wali Sarjal

4 CMs 3 Khara Megah Katho Wali Purana Wahla

5 Health Department 1 District Health Office – Narowal

6 Basic Health Unit (BHU) – Staff (doctors/paramedics)

1 Paramedic - BHU Union Council Mandihali

7 District Administration 1 District Social Welfare Officer

8 NRDP Field Staff 2 Project Staff

9 Community representative 2 Sarjal Katho Wali

Total 19

1.5.3 Evaluation Questions Please refer to the evaluation matrix in Annex 1 for more information about the data collection methods used and type of questions covered with different stakeholder groups.

1.5.4 Sample Size for FGD

13 FGDs were held in total. The breakdown of FGDs by location is: Bhattian Kalan (3), Sahdoky (Mandihali) (1), Katho Wali (3), Khara Megah (2), Purana Wahla (3) and Sarjal (1).

Both male FGDs and female FGDs were held in three villages : Bhattian Kalan, Katho Wali, Purana Wahla. In two villages, Sahdoky (Mandihali) and Khara Megah, FGDs were not held with men as the overall target was already met. In Sarjal, an FGD was not held with women as the overall target for women FGDs was already met. In the two villages where male FGDs did not take place, the evaluation team instead conducted interviews with key stakeholders and other participants in the programme. Similarly, in Sarjal where no female FGD took place and the female enumerator spent time with a community representative to further understand the project. In total 4 male FDGs and 9 female FGDs were held. The inception report had anticipated that 3 male and 7 female FGDs would be held.

188 people in total participated in the FGD’s of whom 137 were women and 51 were men. Each FGD took approximately one to two hours. Please refer to table 3 for additional details about FGDs per village.


Table 3: Number of Different Evaluation Tools Used During Evaluation

Village No of FGDs No of FGDs participants

Men Women Total Men Women Total

Bhattian Kalan 1 2 3 13 27 40

Sahdoky (Mandihali) 0 1 1 0 15 15

Katho Wali

1 2 3 13 34 47

Khara Megah 0 2 2 0 32 32

Purana Wahla 1 2 3 12 29 41

Sarjal 1 0 1 13 0 13

Total 4 9 13 51 137 188

Figure 4: Female / Male Ratio in FGDs

1.5.5 Limitations

As such there were no major limitations during the evaluation stage which could have affected overall validity of the findings. One challenge faced was meeting with government officials in health departments as they were engaged in other official activities. However the team still managed to meet with them in the evenings with the help of the NRDP team. It is important to mention the evaluation team did not experience any issue related to security or lack of cooperation from the communities.

Men 27%

Women 73%

Female / Male Ratio in FGDs


SECTION 2: OVERVIEW OF FINDINGS The HELP for Women - Health Education and Literacy Project for Women programme by FtM and NRDP positively contribute to the health and wellbeing of women and children in the 50 targeted communities in Narowal district in cohort 1 (the remaining 100 villages will be targeted in years 2 and 3). These project villages were selected with the active participation of the local government administration. Priority was given to villages that were relatively far from government health facilities. The positive contribution of the programme was achieved through active engagement of women in maternal, new born and child health related activities. For this purpose, the programme formed 50 WHCs in 50 villages with 350 members. This programme enabled marginalised women to have a say in their lives as the majority of the 350 members in WHC were from marginalised communities. Training provided through the programme enable WHC members to organise as a group and increase their understanding of conditions like Hepatitis A, B and C, malaria, cholera and the importance of hygiene amongst other health topics which affect their daily lives. The WHCs fundraised amongst themselves to cover health related costs for children and women from poor households. They engaged Community Midwives who were trained by the programme to provide medical advice and service in the 50 communities. These services were previously not readily available for women in the targeted villages. An example of the contribution of these CMs is the 1,009 deliveries they conducted and 868 at risk patients they referred to hospitals. No case of mother or child mortality was reported in the evaluation villages by the FGD participants during this time. The establishment of Health Literacy Classes with 1,600 leaners improved literacy skills amongst women in the communities and provided them with knowledge about hygiene and health issues of relevance to them. During FGDs, participants demonstrated their newly acquired knowledge and were able to provide examples of how they are applying it. Women also mentioned they are better informed about care needed during and following pregnancy. There is greater awareness of critical health related issues among project communities in general. This was in considerable part due to the efforts of the Women’s Health Committees. The Cohort 1 WHCs conducted 155 community workshops and awareness-raising events on health issues, using information, education and communication materials from the project. One concrete output of these workshops was the vaccination of some 4,500 children in connection with measles and polio campaigns. All components of the programme combined resulted in an increase in immunization amongst women and children. In light of the findings from FGD and interactions with the BHU team, immunisation coverage for cohort 1 villages was almost 70%, up from 61% before the intervention. In the first year of the programme, 50 Men’s Support Groups (MSGs) were also formed with 1,000 members. The MSGs members are contributing financially to the work of WHCs and also promoting their efforts to improve health conditions for women and children in the communities.


SECTION 3: PROGRESS TOWARDS OUTCOMES The midterm evaluation focused on the first cohort of 50 villages where activities were completed. These villages were selected for the programme with the active engagement of the government District Health Officer. All these villages were in rural areas and were chosen from a list of some 70 longlisted villages where consultations were held with communities. A key consideration was the availability of existing health services. As a result of the Government of Punjab’s integrated health programme, Lady Health Workers and Expanded Immunisation Programme were present in almost all population centers of Narowal. The main criteria for selection therefore were the long distance from the nearest health facility and problems with general wellbeing within the community. It can thus be safely concluded these selected villages are marginalized as compared to other parts of the Narowal and their selection involved the active involvement of both government officials and communities.

3.1 Women’s Health Committee

3.1.1 Selection & Composition:

To facilitate the selection of WHC members, NRDP staff held meetings in all of the project communities. At the meetings, community members were informed about the overall purpose of the committees, their structure and the selection criteria for members. In collaboration with the communities, seven women were then selected in each village thus totalling 350 members. During the evaluation, the team concluded that the majority of the members were from farming communities. From a social status perspective, they ranged from lower middle class to poor households. Christian community members were part of these committees in at least two of the villages which the evaluation team visited. The evaluation team did not come across any person with disabilities as a committee member. Each of the committees the evaluation team visited included at least two to three illiterate women even if committees had a higher ratio of literate women amongst its members as compared to the overall village population. A key characteristic of the committee members were their active engagement in the communities’ lives. They, as a whole, also enjoyed community trust.

3.1.2 Training & support received:

WHC members were provided with training in project management, scope of the project, roles and responsibilities, reporting skills, the role of literacy in health education awareness, book keeping, resource mobilization and advocacy. Two basic skills training per month for nine months (August 2014-April 2015) were additionally carried out covering Community Management Skills Training (CMST), Leadership Management Skills Training (LMST), and on health issues like dehydration, Hepatitis A, B and C, malaria, cholera and the importance of hygiene amongst other topics. All these trainings took place at the community level and lasted for around two to three hours and were provided by NRDP team members. An example is training on "Hand Washing" which took place in May 2015 in which 329 WHC members


participated. Another example is Training on "Dengue Fever" which took place in June 2015 which 358 members of WHCs participated. A training with the topic "Clean Drinking Water" and was provided to all 50 WHCs with participation of 350 members. All the above mentioned trainings were provided individually to each of the 50 WHCs by NRDP staff and had same training content. During interactions with the WHCs, they confirmed that they found the training they received useful. They mentioned that the skills they learnt were new for them. When asked by the evaluation team about key messages were they able to provide specific examples like practical methods to purify or maintain clean drinking water.

3.1.3 Activities:

Background discussions with WHCs confirmed these committees were meeting regularly (at least once in a month), and were making an active contribution to bring improvements to the health situation of women and children in their communities. The WHCs had run community workshops to disseminate health related messages, with a particular focus on women and children. They used information, education and communication materials at these events as well as role play and drama. WHCs identified Health Literacy Facilitators and Community Midwives. WHCs engagements in identifying HLFs and CMs ensured selection of the best persons for the job – persons whom the community trust. These activities were organised with the support of the Men’s Support Groups and wider community. The WHCs coordinated the organisation of Health Literacy Classes and followed up on students who were dropping out. They also successfully fundraised to support women in accessing health services who otherwise would not be able to do so due to lack of financial resources. These 50 WHCs collected around PKR 80,000 over the last nine months and distributed PKR 45,000 to some 20 poor pregnant women who otherwise would not be able to avail health services. The remaining funds are still held with WHC for future cases. WHCs are continuing to contribute to the fund monthly. Even though there were clear book keeping records with regards to money collected and money distributed, there was no reference information as to how the beneficiaries were selected by the WHC. It is an important consideration as this programme is implemented in rural areas with high poverty rates thus it is important to have clear guidelines to ensure prioritising the poorest of the poor. These WHCs were engaging local community elders, government officials and other stakeholders like NGOs to lobby for additional health services for their communities. An example of such advocacy efforts is the meeting of the WHCs with at least four Basic Health Unit teams regrading vaccination, deliveries and medicine issues. Another example is the application from the community of Mundayki to Tehsil Municipal Administration Department for repair of clean drinking water plant, which is now functional after close follow up by the communities including MSG members. As mentioned on multiple occasions by the different MGS members and WHCs and also confirmed by NRDP team members, NRDP strong network with Mr. Ahsan Iqbal (currently one of the federal ministers and Member of Parliament from Narowal and is leading national development planning including Pakistan Vision 2025)


provided active support to the project beneficiaries. His role was especially instrumental when additional funding was required to carry out follow on activities e.g. repair of the drinking water plant in Mundayki. As Narowal is Mr. Ahsan Iqbal’s constituency, is he keen to listen to input from the people in the area. This project has enabled people to make specific and concrete requests to him, making it more likely for their demands to be met. Women including WHC members, participants from the health literacy classes and other women from the community participated in bi-annual knowledge sharing workshops to disseminate best practise and learning from HELP. These events were organised by NRDP staff and participants included officials from the District Health Department and Social Welfare Department as well as the wider communities. An example is March 2015 seminar on “Healthy Women Leads to Healthy Society” in which some 250 women and 85 men participated.

Figure 5: Interview with BHU staff in Narowal


Figure 6: Picture Illustrating Fund Box and Record Registers

Figure 7: Receipt Book for Donation


3.2 Community Midwives

3.2.1 Selection & Training:

WHCs, with the support of NRDP, selected 50 Community Midwives, one in each village in July 2014. (Selection of the second batch of 50 CMs, from Cohort 2 villages, took place in July 2015). There was an upper age limit of 40 years for the selection of CMs. Traditional Birth Attendants, or their close relatives, were given priority for selection as CMs if they were meeting the minimum selection criteria and were interested in joining this project. This was done with the assumption that these women already have some hands-on experience in mother and child health related issues and with new knowledge and learning will be in a far better position to positively contribute to their communities. The Cohort 1 CMs were split into two groups of 25 CMs and were given 15 days training in September and October 2014. Additional training as provided in the beginning of 2015 thus giving a total of 30 days training over a year for the cohort-1. The curriculum for this CM training was developed in consultation with government and included some components from the National MNCH Programme curriculum for government trained CMs. The training included use of professional manikins (a life-sized anatomical human model used in education) from Punjab District Health Authority and was conducted by Integrated Community Development Initiative (ICDI). ICDI had previous experience in training Community Midwives. ICDI has also trained a group of five master trainers who are NRDP team members. This led to a development of in house training capacity within NRDP for CMs. In year two, for cohort 2, the training was led by NRDP’s master trainers with support from ICDI. This is a good measure and will increase sustainability of the programme. A key aspect of the training was making the project’s CMs aware of their limitations and encourage them to seeking advice or refer where appropriate. The project programme design foresaw three groups of 15-18 women, instead of the current 25 CMs in each group. The evaluation team believe this, to a certain degree, has limited CMs ability to get hands-on training during the workshops. After the training, all C Ms were given midwifery kits. The evaluation team found that due to limited technical knowledge and general level of education, some CMs were unable to fully utilise the blood pressure measuring instruments. CMs themselves expressed that they found it difficult to use the blood pressuring measuring equipment in discussions. During the study, the evaluation team found these 50 CMs were providing services like pre-natal and post-natal check-ups, advice on healthy diet, identifying danger signs and referral to the hospitals, examination of the new born baby, importance of vaccination of the new born baby, teaching mothers how to breastfeed their children amongst others subjects. Some of these services included advice and support to communities (women, community leaders, Men’s Support Groups). Advice to MSGs were mostly linked to different actions required for the wellbeing of mothers and children regarding health, healthy diet, importance of consulting doctors in time, taking women to hospital in case of danger signs during pregnancy amongst others. The CMs were encouraging pregnant women to access available pre- and post-natal services. Even though it was to a limited extent as compared to the number total deliveries taking place


in the village, CMs provided safe delivery services to low risk patients in the villages. In total, according to NRDP data, in the 50 villages, CMs conducted 1,009 deliveries (against a target of 1,180 deliveries). They also helped in providing a counter-narrative to mothers and other women in the communities regarding some of the harmful local myths on pregnancies, childbirth and infant health with the aim to abandon certain practices. An example is to give an elder’s saliva (elders are mostly men) to a new born so that s/he will grow in his image, or not giving certain foods to pregnant or new mothers under the pretext it has “cold” or hot” effect on their health. They were also available to provide advice on contraception. They were not directly addressing the issue of family planning but working through the committee. This was due to the cultural consideration where proving such information was not always welcomed. Rather, other tactics like linking the issue of child spacing with mothers’ health or with the financial resources required to educate a child were found to be more effective ways to address family planning issues. In the case a woman would ask for a direct opinion related to family planning, CMs will provide a qualified opinion and also refer to women to district hospital for further advice and support. The CMs reported in conversations with the evaluation team that initially villagers were hesitant to use their services and were relayed more on traditional birth attendants. Over time, however, community perceptions have changed and more families trust them with their deliveries. The CMs were creating patient records both for pregnant women and children and were even helping mothers to keep vaccination records for their children. There was close collaboration with the District Health Department. For this purpose, the CMs were using smart phones to seek advice on specific cases and make emergency referrals. According to the CMs, two negative issues affecting their wider usage of the referral system was the language of the software since they were unable to write complete messages in English and the lack of available credit in the phones. Based on NRDP data, in total, CMs sent a total of 327 messages of which they received 266 responses from the doctors. The GLOW evaluation team has not been able find an explanation for the gap between messages sent and responses received within the scope of this evaluation. According to the CMs they normally receive a response within one to two hours after sending a text message. During key informant interviews, the respondents mentioned there is no incentive for the doctors to treat/prioritise NRDP beneficiaries over any other patients. As a result, when NRDP patients go to the hospitals, they have to wait – often for hours - just like other patients. With this said, as the focus of any efforts should be system-wide improvements, the evaluation team believe there should be no special treatment or provisions for NRDP beneficiaries, rather the focus should be improving services for all mothers, children and other patients. CMs also mentioned once they refer a patient, they do not actively follow up on further medical treatment. However, as the patients are from the same village, the CMs usually find out what happened to them. During the FGDs when beneficiaries were asked how they would like to improve the situation, they mentioned that a focal point doctor at the district headquarter hospital will help them avail of better health services.

3.2.2 Feedback from Communities:

During the FGDs and Key Information Interviews (KIIs) interviews, participants confirmed that CMs are providing them with advice and support on reproductive health, guidance on the


course of normal pregnancies and ante- and post–natal care. They described how CMs are helping them identify danger signs during pregnancies, providing advice on treatment and more importantly referring them to health facilities if necessary. In addition, community members described how CMs are examining new born babies, assisting with breast feeding and advice on nutrition. One of the participants explained how her baby had not been feeding well initially and the CM taught her how to properly lay the baby and feed her. On two occasions did the FGD participants describe how they used to put saliva from an elder of the family on the mouths of babies at birth as a sign of good luck but that this stopped after CMs giving advice. One woman mentioned that CMs are also advising on family planning (when she asked for it and showed an active interest), first aid, and family nutrition, all of which they are finding very useful. These women mentioned they are passing on these messages to their husbands and now can discuss family planning issues with them. Such discussions include why it is important to space child births, that temporary birth control pills does not mean women will be unable to have children in the future, the balance between family finances and the number of children amongst others. FGD participants mentioned the importance to proactively provide medical and health information to women as otherwise they do not seek any medical opinions. An example mentioned was that women were not given certain food which has “hot” or “cold” properties (like hindering mothers from consuming milk and yogurt). They also mentioned that for mothers whose infants were colic, the elderly women in the village would advise them to stop consuming certain food such as eggs, mangos, turnip, and milk. With the CMs now around, women in the village now can seek a qualified opinion from them. The CMs do not charge a formal fee, however in cases where CMs have supported deliveries they were paid an average amount of around PKR 1,500 per delivery. This amount increased up to PKR 2,500 when a boy was born as compared to a girl child reflecting deep rooted gender discrimination still persistent in communities. Similarly, in cases of first born children, families were paying more. Those having better financial status were also paying more. Families with lower income were paying as low as PKR 500 or nothing at all and were supported by the WHC through the community fund. This payment system pre-existed CMs, and Traditional Birth Attendants providing similar services were receiving these compensations. CMs under this system inherited it. The evaluation team found the CMs helped the majority of women of childbearing age in the village who were in need of medical help. As these CMs are part of the community, are they now a permanent resource for health related services to mother and children in their area.

3.3 Health Literacy Facilitators The WHC members selected 50 Health Literacy Facilitators (HLF), one in each village, against a total target for three years of 150 HLF. Each of the HLFs had sound learning and reading abilities – which was demonstrated during interactions with the evaluation team. They were motivated to bring about a change to their communities. This can be reflected from the fact that no HLF dropped out during the training or during the duration of their classes. The 50 HLFs were divided into two batches of 25 for training purposes. They received 27 days of training in total. An initial training of 15 days was provided before the start of the classes. (One batch was


trained in September 2014 and the other in October 2014). This was followed by a seven day training, carried out in November and December 2014, and a five day training carried out in March 2015. The HLFs were provided with four books with health education materials to use in their classes in the local language of Punjabi. HLFs found these books and the teaching materials relevant to the needs of the communities. HLFs expressed that the training they received covered their needs. As a teaching methodology, education materials used images and simplified text. HLFs were introduced to participatory teaching methodologies where the instructor encouraged discussions about the topic in the classes. HLFs were also trained to ask participants to share knowledge and ask questions. As far as teaching methodologies are concerned, evaluators found that even if content relied on images and simplified text, was it a new concept for the participants – including instructors – to ask questions, discuss a topic and share experiences. The FGD participants mentioned they felt shy when HLFs was were asking about their experiences. In cohort 2 and 3, the evaluation team believes more efforts will be needed to encourage participants to share their questions or situations. This will require HLFs putting in practice their new acquired skills in participatory teaching techniques.

3.4 Health Literacy Classes

3.4.1 Learners and attendance

A health literacy class was established in each of the 50 villages. There were a total of 32 women in each health literacy class. Of these, seven were Women’s Health Committee members and the remaining 25 were other community members. During first cohort 1,600 women were enrolled in the 50 villages. This was as per target. The overall programme target is 4,800 women students during the three years. Students in Year 1 classes included women from farming communities with no land holding or land holdings below four hectares, Christian community members, daily wage labourers who were working in the farms and other marginalised groups. These literacy classes had two main components; the participants were given basic knowledge about mother and child health related issues as well as about general health issues of relevance to them like malaria, nutrition, dengue, vaccination; whereas in the second aspect, they were taught how to read and write. The health and literacy classes ran for approximately eight hours per week and continued for six months. These classes mostly took place when men were away from homes for work, children were in schools and the women had completed domestic work, usually around 10 – 11. Based on the feedback from HLFs and WHCs, it is estimated that 80% of the students successfully completed the six months course. The remaining 20% did not formally drop out, rather attended fewer classes – i.e. they attended classes whenever they were free but would prioritise their domestic work. During the FGDs, participants mentioned domestic engagements and lack of interest by the participants (e.g. what will be the benefit even if I learn these new skills) as the main reasons for those who did not join the majority of the classes. Participants in the FGDs also mentioned that during the sowing and harvesting season the dropout increased. This is the time when women were either very busy or too tired to attend the


classes. To address this issue, the project adopted more flexible timing in order to be responsive to the constraints for women. It is also suggested to have individual follow-up with students who continue to miss classes.

3.4.2 Coverage of Health Literacy Classes

Punjab is the province with the largest population in Pakistan and on average are village sizes in Narowal much larger than compared to other parts of the country, for instance Sindh. During the evaluation, the team found that on average, at least 100 or more women in each village could have joined the classes, but are not doing so because of class size restrictions. During the focus group discussions, participants estimated that on average three out of every four male children in their area go to school (either public or private) compared with about two out of every girls. This places enrolment for boys at around 75% and for girls around 50%. These estimations are slightly different than the 2015 Economic Survey of Pakistan11 which places boy’s enrolment at 66% and girl’s at 63%. The evaluation team believes the Economic Survey of Pakistan data is more accurate as it adopts a more rigorous statistical methodology as compared to community estimates. During the FGDs participants identified poverty, age of the girl students (girls are more likely to leave school around 12-13 years of age when are considered reaching maturity age), early marriages, support needed with domestic help, distance from the school and availability of facilities at schools as some of the reasons leading to low enrolment and high dropout of girls. During the FGDs, the participants said the NRDP/FTM intervention to improve health and literacy amongst young girls and women was very useful since it gave them health related knowledge which they otherwise would not have received. When FDGs were asked why girls would not have been able to gain such health knowledge, the participants mentioned the taboo to discuss health related issues, especially related to family planning, child spacing or sex education.

3.4.3 Learning from Health Literacy Classes

Participants in the Health Literacy Classes found hygiene related classes very useful. During the FGDs, they demonstrated how to use hands in a hygienic way. When the participants in the FGDs were asked to narrate some of the key messages they learnt during the hygiene classes, they mentioned washing of hands before and after eating, washing of hands before feeding children, use of soap after toilet visits, teaching children not to pick noses as some of the examples. Interestingly, they mentioned what they learnt now was not something new for them but they had heard these messages before. Rather what they were not preciously aware of was why it was important to practices these measures and how it could help reducing disease in their family. Once aware, the FGD participants mentioned they are now practicing it and also

11 http://www.finance.gov.pk/survey/chapters_15/10_Education.pdf


mentioning these messages to their young ones and other family members, encouraging them to adhere to a hygienic life. Within the health literacy component, women were told about the benefits of family planning, vaccination and medication. They were also informed about care during and following pregnancy. Besides, they also learnt about infant and child health. They were told about anaemia and why it is a danger for mother and child health, especially in case of expecting mothers. The health literacy students were also made aware of septicaemia and how it can originate from urinary tract infections, pneumonia or infections in the abdominal area. The class participants were also told that due to unhygienic conditions in the hospital during processes, such as a surgery or child birth, may a person also get exposed to septicaemia. During the classes, participants were also informed about malaria and dehydration. As dengue has become a serious health issue in Punjab, health literacy classes included information on different aspects of dengue and how a dengue mosquito looks different to the normal malaria mosquito. An example is how one participant in the FGDs mentioned she was unaware that the dengue mosquito is a clean water mosquito and they have to cover their drinking water sources whereas malaria mosquito uses unclean water for laying its eggs.

3.4.4 Health Literacy Classes Impact on Vaccination and Diarrhoea

During the KII, the government district health officer mentioned that dropout amongst the children was a major issue for vaccination programmes. Similarly pregnant women were also not coming for completing required vaccination prior to delivery, including Tetanus Toxoid vaccination. With the help of intervention of health literacy classes, they believe women in these villages are more aware of the importance of vaccination. Local health practitioners confirmed this analysis and mentioned they are now receiving more women and children for vaccination as compared to one year ago (at least 15-20% increase compared to a year earlier – for both women and children). In light of the findings from FGDs and interactions with the BHU team, immunization coverage for the cohort 1 villages (50 villages selected in year 1) was almost 70%. This is more than the general 61%12 for the coverage in rural areas of Punjab. Based on the data from the programme business case, BHU team and FGDs, the evaluation team believe in year 1, the programme assisted some 7,000 children with accessing vaccination. Further, a noteworthy change is diarrhoea situation in the programme area. In the six villages the team visited in December, there were only an aggregate of 12 cases (8 girls, 4 boys) amongst children reported with diarrhoea related issues in the last 30 days. According to the 2014 Punjab Multi Cluster Indicator Survey, 17% of children under the age of 5 who were assessed had diarrhoea in the last 2 weeks13.

12 http://www.chip-pk.org/wp-content/uploads/2015/02/Status-of-Immunization-Coverage-Maternal-

Child-Health-Care-Punjab-2014.pdf 13

http://www.pndpunjab.gov.pk/system/files/Key_Findings_Report_MICS_Punjab_2014%20-%20Final.pdf


Narowal has better literacy and awareness amongst women as compared to other NRDP/FTM implementation areas. The evaluation team found that as a result these participants were in a better position to benefit from the classes. The evaluation team believe learners did comparatively better than other areas in terms of learning new skills to have basic ability to read. However, over 50% of the participants were still unable to fill out the forms specified as one of the learning objectives. (These are forms requiring individuals to document essential personal information about themselves and their families when accessing health services.) With that said, over 85% of the participants were able to write their names in Urdu. The evaluation team believes those participants who had some previous skills in reading or writing (from formal or informal learning) were comparatively better off as compared to those who had no educational background and were starting from scratch. The Health Literacy Classes helped in improving health related understanding amongst the communities including familiarising participants with common medicines, vaccination and health services. The evaluation team believe health literacy classes helped to increase survival rate and well-being of pregnant women and their infants in the targeted villages.

3.5 Community Advocacy and Awareness

3.5.1 Advocacy Activities

Women’s Health Committees who were trained on advocacy skills used these skills to reach government officials and other decision makers to find solutions to their problems. According to NRDP, 50 WHCs conducted 155 advocacy events on medical camps14, mosquitoes spraying, measles vaccination, Anti TB Camps, special services for displaced persons and improvement of basic health facilities. As a result, five medical camps, four Anti TB Camps, 50 measles vaccination and 50 polio vaccination campaigns, and three general medical camps were arranged during the July - October 2015 period. Some of these activities (e.g. medical camps) might not have taken place without active advocacy efforts by the communities. During conversations with communities, they confirmed these events and mentioned they are useful in terms of raising awareness and mobilising support at the community level. Their advocacy efforts were aimed at bringing services to villages which can help promoting mother and child health. The NRDP team mentioned the establishment of a sewing centre at one of the village with 15 machines as one successful example of community advocacy efforts. They also mentioned the repair of a clean drinking water plant at Munday Kay Goray with the support of

14 A medical camp is established in/close to a village for a short duration of time, staffed by a

doctor and other medical personnel. In the medical camp, patients receive free of cost medical check-up alongside medicine and referral services to other hospitals.


PKR 700,000 by the Tehsil Municipal Administration and medical camps by District Health Department with the help of Mercy Corps as other examples of successful interventions. A key achievement of the advocacy components of the programme is the expansion in immunization amongst women and children. In light of the findings from FGDs and interactions with the BHU team, as previously mentioned, was immunisation coverage for cohort 1 villages almost 70%, compared to the usual 61%15 coverage in rural areas of Punjab.

3.5.2 Awareness Raising Activities

Women’s Health Committees were also trained on awareness raising activities to reach out to the wider community in the villages. During discussions with WHCs, they mentioned different community workshops and awareness raising events in which they used information, education and communication materials. Students from Health Literacy classes and Men’s Support Groups participated in these awareness raising events. One such event was the celebration of Global Measles Day in all 50 Villages; WHCs sensitized the communities by conducting an Awareness Raising Walk, educational sessions, corner meetings, distribution of a brochure on Measles provided by the District Health Department and display of messages developed by WHCs. Another awareness raising activity was related to the vaccination of children. This was carried out in response to a Government of Punjab campaign on measles from 6th February to 12th February 2015. An awareness walk regarding family planning was organised together with the Family Planning Department. Further, health department officials, project beneficiaries (WHCs, CMs, HLFs etc.) participated in this walk. During discussions with health officials, they confirmed the active participation in these campaigns. These health officials also mentioned these advocacy efforts are helpful as it helps them to bring issues that affect pregnant women and young children into the public arena. The health officials also lauded NRDP and WHCs contributions to the polio campaigns and mentioned they helped them to provide such vaccination to more than 2,300 children in the 50 targeted villages.

3.6 Men’s Support Groups Beside Women’s Health Committees, in the first year of the programme, 50 Men’s Support Groups were formed. Each of the Groups had 20 members, thus totalling 1,000 members for cohort 1. In year 2 and 3, another 100 MSGs will be engaged. These MSGs were formed to solicit support for mother and child health related interventions. Members of the MSGs were selected based on their active engagements in the communities. They were mainly selected by the villagers themselves with active support from NRDP team members. Each of the MSGs had a president, a secretary and a treasurer. These three positions were mostly filled by the MSGs through consensus, and at least in two occasions through election. Almost in all groups, MSG

15 http://www.chip-pk.org/wp-content/uploads/2015/02/Status-of-Immunization-Coverage-Maternal-

Child-Health-Care-Punjab-2014.pdf


members had a close relative who is also a member of WHCs. This helped coordination between MSGs and WHCs. Once MSG members were selected, they were provided with project orientation, roles and responsibilities of the MSGs and how to effectively coordinate with WHCs. These MSGs were provided with two additional trainings on Mother and Child Health Care in which some 860 members (out of 1,000) participated whereas in training on Diet of Pregnant Women and Infants, some 780 members participated. These trainings were delivered by the NRDP team. Each of the sessions lasted for 2-3 hours and were held with all the fifty MSGs. Topics covered in these sessions included mother and child health care, dehydration, importance of mother feed, Hepatitis, Malaria, Jaundice, Cholera amongst others. The MSGs were also contributing financially to the work of WHCs. During discussions with MSGs, the members revealed they found these trainings very useful. An example mentioned was that they were not aware of the importance of rest for pregnant women. Rather they were of the opinion they the more a woman is active, the more likely is it she will deliver a healthy baby. They also mentioned now they are joining their women at the hospital in the city whereas previously women were going on their own. MSGs were collecting monthly donations for pregnant and lactating women in the communities. They were keeping a proper record of these donations. Once collected, they were handing it over to WHCs to distribute amongst the women most in need. During conversations with the evaluators, MSGs members confirmed they are meeting every month and keeping a proper record of their meetings in the form of minutes of the meeting in the registers provided by NRDP staff. They showed these registers to the evaluation team, which clearly reflected proceedings of the meeting. During interactions with MSGs, the members also mentioned they are from poor households and could not afford health services for their family members. However as now they all are putting resources together, it is making it easier for both men and women to solicit health services for pregnant and lactating women and new born children. During discussions with MSG members did they also mention active participation in both awareness raising and lobbying activities. For the awareness raising component, they mentioned active participation in awareness walks and workshops. MSGs members also mentioned participation in sensitization events on mother and child health care, dietary requirements for pregnant women, importance of hygiene, negative effects of dehydration amongst others. According to NRDP data, more than 5,000 men benefited from these sensitisation events. For the lobbying component, MSG members mentioned they joined women while meeting with district officials and other decision makers. MSG members also mentioned they provided volunteer services during the vaccination campaigns and free medical camps, which were made possible as a result of joint advocacy efforts by MSGs and WHCs. Coordination between WHCs and MSGs was not a major issue as both men and women were from the same village and were often related to each other. The presence of elderly women and men in WHCs and MSGs among the members also eased discussions between the two communities. With this said, men from MSGs mentioned that discussions on sensitive health topics was difficult in group situations and preferred to discuss such issues with their relatives (mostly their wives).


Moving forward, one key area of improvement they mentioned is more trainings for MSG members. The MSG members were of the opinion that if possible, the same set of trainings should be provided to them which WHCs are currently receiving.

Figure 8: Discussion with MSG in Katho Wali


Figure 9: Discussion with MSG and Beneficiaries in Sarjal

3.7 Other Programmatic Aspects:

Targeting of Beneficiaries: The programme mainstreams gender equality on relevant issues both in its design and delivery of activities. Overall 63% of the direct beneficiaries were women though men were also drawn in through Men's Support Groups.

Value for Money: As far as the overall productivity of the programme, the project activities provide good value for money. From a managerial or staffing perspective, the ratio of staff to beneficiaries/villages spread activities is appropriate. Similarly, staff salaries are in accordance with the market price for national level organizations for similar positions.

From a timeliness viewpoint, despite the fact that the activities took as much time as necessary to be fully functional and it took several rounds of discussions before the logical framework for this programme had been concluded, generally speaking, by now the project is fully functioning and relevant. Village selection in year 1 along with other activities like selection of WHCs, CMs etc. were closely coordinated with the government and required their consent and authorisation. This led to some delays for year 1 start-up of activities in Narowal.

Working with Others:

From the KIIs, it was clearly evident that NRDP enjoys very strong networks with government institutions. This was best reflected from the very good knowledge of NRDP activities held by the District Executive Officers covering health and social welfare. Narowal has other government run health programmes like the National Maternal New-born and Child Health Programme and the Prime Minister’s National Health Insurance programme amongst others. NRDP is coordinating its activities with these programmes through the health department.


When a District Health Officer was asked to comment on the health literacy classes during the key informant interviews, his response reflected complete awareness of the intervention. He was of the opinion a major cause of the mother and child health related issues is lack of awareness and that this intervention is addressing this issue. He further mentioned that his intervention is helping to reduce the prevalence of preventable diseases. At the same time it is addressing the issue of clean water and better hygiene through improved sanitation. He was also of the opinion that a key problem facing mother and children health is nutrition, and is an issue that has to be addressed to achieve any significant progress.

As Narowal is a not a priority area for many NGOs due to its comparatively better situation as compared to other districts in Punjab and Pakistan (the worst indicators are for Punjab districts are for Southern districts like Rajanpur, Dera Ghazi Khan, Muzaffargarh etc.), are only a few NGOs operating in the area. Mercy Corps operating in Narowal and the organisation has participated in NRDP knowledge sharing workshops. Help & Development Organization and an NGO called Community Development and Anti TB Programme are also operating in Narowal. However these NGOs are currently not actively participating in NRDP programme activities. Besides these, the evaluation team is not aware of any other major NGOs that are functional in Narowal. However, other philanthropy associations like Edhi Foundation exist who have a specific free of cost ambulance service. Key informant interviews with the NRDP team, MSGs and WHCs amongst others, showed that the NRDP team has limited interactions with the Edhi Foundation. This is mainly due to the nature of Edhi Foundation’s approach as it does not generally coordinate its activities with other NGOs.

Given the limited role by NGOs in Narowal, no KIIs took place with them. The team has also not received any evidence of formal coordination with between NRDP and other NGOs (e.g. minutes of meetings). With this said, the team believe there is a need to for increased coordination and collaboration between NRDP and other non-profit actors present in Narowal for delivering better results to the project beneficiaries. An example of a potential outcome of such collaboration could be availability of free of cost ambulance services for NRDP beneficiaries through Edhi Foundation.


SECTION 4: CONCLUSIONS AND RECOMMENDATIONS

4.1 Conclusions

Overall the Health Education and Literacy Project for Women programme is positively contributing to improve the health situation for women and children in Narowal. The programme is doing this by building up community level health response capacity through: trained health service providers (Community Midwives), increased health literacy capacity amongst community members, developing local female leadership within Women’s Health Committees supported by Men’s Support Groups and advocacy and awareness raising with key stakeholders. The programme has a total target of 150 villages – equally divided into three cohorts. It is following a phased implementation approach. In year 1, it selected first cohort of 50 villages and trained WHCs, HLFs, MSGs and others. Currently it is expanding the programme to the second cohort of 50 villages and in year 3 the programme will be implemented in the remaining 50 villages. This evaluation focused on villages in cohort 1. These villages were selected from rural areas of Narowal and selection involved active engagement of district level government officials. Once selection of the villages took place, 50 Women’s Health Committees (one in each village) were established. The selection of 350 committee members (7 members for each committee) involved active participation from the communities. These women were from farming families, minorities, people with low literacy and labourers working the farms. The evaluation team did not come across persons with disabilities in the committees. All committee members were active in the community where they lived. In the 50 programme villages, WHCs were regularly meeting and were making an active contribution to improving the health situation of women and children in their communities. They also successfully fundraised to support women in accessing health services who otherwise would not be able to do so due to lack of financial resources. The 50 WHCs collected around PKR 80,000 over the last nine months, distributed PKR 45,000 to some 20 poor pregnant women and are holding the remaining balance for future needs. The WHCs and MSGs were engaging with local community elders, government officials and other stakeholders to lobby for additional health services for their communities. Further, WHCs and MSGs who were trained in awareness raising used these skills to reach to the wider community in the villages to promote mother and children health related issues. Several other awareness raising activities were initiated and led by WHCs and supported by MSGs such as sensitization events on dehydration, water borne diseases, health & hygiene, malaria, hepatitis, dengue fever, hand washing amongst others. The WHC members with the support of NRDP selected 50 Community Midwives. These CMs were given 15 days training in two groups (25 CMs each) in September and October 2014, followed by additional training in the beginning of 2015 thus giving a total of 30 days training over a year for cohort 1. These trainings were conducted by Integrated Community Development Initiative. A key aspect of the training was making CMs aware of their limitations and encouraging them to seek expert advice or refer where appropriate. The programme


design foresaw three groups to consist of 15-18 women, instead of 25 CMs. The evaluation team believe this, to a certain degree, limited the CMs ability to get hands-on training during the workshops. After the training, all Community Midwives were given a midwifery kit. The evaluation team believe (based on the feedback from CMs and community members) that due to limited technical and education knowledge, CMs were unable to fully utilise the blood pressure measuring instruments. During the study, the evaluation team found the 50 CMs were providing a number of different services to pregnant women and infants in their villages. Some of these services included advice and support to communities (women, community leaders, Men’s Support Groups). The CMs were also encouraging pregnant women to access pre and post-natal services. In total, according to NRDP data, in the 50 villages CMs were present at 1,009 deliveries. CMs were also using smart phones to seek advice and make emergency referrals to medical professionals. During the FGDs, CMs mentioned the absence of a delivery station within their village, like the one they saw during training, affected their work. They were of the opinion that the absence of these stations made it difficult for them to conduct regular check-ups and assist at deliveries. According to the CMs, two issues affected their usage of the referral system; firstly the language of the software since they were unable to write complex messages in English and secondly the lack of availability of phone credit for text messages. Based on NRDP data, CMs sent a total of 327 messages to which they received 266 responses from the doctors (the evaluation team were unable to explicitly determine the reasons for the lower response). According to the CMs, they normally receive a response within thirty minutes to two hours after sending a text message. The average income per delivery for the CMs was PKR 2,000, even though it varied from case to case. This payment structure was already in existence before the start of this project for Traditional Birth Attendants. The evaluation team believe Community Midwives helped the majority of women in the village who were in need of medical help and given they are already part of these communities, will continue to do so in the days to come as well. The WHC members also selected 50 Health Literacy Facilitators (one in each village). Even though there were no formal education criteria for the HLF, each of the HLFs had sound learning and reading abilities. These facilitators were provided with 27 days of training in total; 15 days training before the start of the classes, followed by seven days follow-up training and five days of on-going support. They were also provided with four books of health education materials to use in their classes. In each of the 50 villages, health literacy classes were established. There were a total of 32 women in each class. Of these 32 women, seven were Women’s Health Committee members while the remaining 25 were other community members. These classes focused on teaching basic literacy related to health knowledge and skills. As Narowal generally has better literacy level and awareness amongst the women compared with other NRDP/FTM implementation areas, the evaluation team found the participants in these classes to be in a good position to benefit. 1,600 women were enrolled in the first cohort of 50 villages. Participatory teaching methodologies were used. During the FGDs, the participants mentioned that the NRDP/FTM intervention to improve health and literacy amongst young girls and women was very useful since it gave them health related knowledge which otherwise they would not have received. With regard to the participants’ literacy skills, the evaluation


team believe learners did comparatively better in terms of improving their ability to read than to write. The evaluation team believe, all these measures related to health literacy classes will help in increasing the survival rate and well-being of pregnant women and their infants in the targeted villages.

Beside Women’s Health Committees, in the first year of the programme, 50 Men’s Support Groups were formed. Each of these groups had 20 members, thus totalling 1,000 members for cohort 1. Each MSG has three office bearers, who were mostly selected through consensus. As MSG members were from the same village, most of them were directly or indirectly related with female members in the WHCs. This helped communication and coordination between the committees. Around 860 members from MSGs were provided training in mother and child health care and 780 members from MSGs were provided with training in dietary requirements for pregnant and lactating women. The MSGs contributed financially to the work of WHCs and supported their day to day activities. They also took women to the hospital when they needed it, something men were not usually doing before. During discussions with MSGs, the members admitted they had not been aware of the importance of rest for pregnant women. Rather they had been of the opinion that the more active a woman is, the more likely is it that she will deliver a healthy baby. Some of the members also mentioned they would like to receive additional training just like the WHCs are provided with. Registration of both WHC and MSG with the government did not appear as an issue during the current activities, however the evaluation team believes registration is an essential tool in transforming these village level community based organisations into a more permanent structure in the medium to long term. This is also important because the evaluators believe different stakeholders (government, NGOs etc.) will take a formal community body more seriously as compared to an informal group.

4.2 Recommendations

The following are some of the key recommendations from the mid-term evaluation of this NRDP/FTM project:

4.2.1 Women’s Health Committees Inclusion of Women with Disabilities in the Women’s Health Committees: Even though Women’s Health Committees were generally representative, the evaluation team

did not come across any women with disabilities who were members of a committee. As

women with disabilities are marginalised, it is recommended to include them in committees,

especially in connection with cohort 3 selection.

Written criteria for selection of vulnerable persons for financial assistance:


WHCs distributed financial assistance amongst poor pregnant women. The evaluation team

found written records of the donations collected and spent in each visited village; however it

was not clear how a beneficiary is selected for assistance. Even though there was a general

understanding within the community how to select a beneficiary based on her needs, to make

the system transparent it is recommended that NRDP team together with the community

develop written criteria for selection of the beneficiary eligible for financial assistance from the

community fund.

4.2.2 Community Midwives Training of Community Midwives: The current training of Community Midwives for cohort 1 took place in two batches of 25 participants each, whereas the business plan for this programme foresaw three batches of 15-18 participants each. The evaluation team believe it will be better to revert back to the original business plan with fewer participants each time since it provides more opportunities for the participants to get hands-on experience during the training. As NRDP now has its own master trainers, the evaluation team believes it will not lead to any significant increase in costs. Composition of Medical Kit for CMs The evaluation team recommends replacement of manual medical instruments with automated medical instruments (e.g. blood pressure measuring instruments) to reduce the possibility of mistakes by CMs. Even though the issue of medical supplies was not raised by beneficiaries as part of the evaluation, the evaluation team believe it would also be good to consider the provision of pyodine and medical gloves as part of the medical kit for CMs. Perishable supplies once finished should also be provided on regular basis to the CMs. Refresher training for trained CMs: Even though some refresher training for cohort 1 CMs was still to be conducted at the time field work was undertaken, the evaluation team is also recommending additional refresher courses for the NRDP trained CMs that would run throughout the programme cycle. Such courses will be helpful in recalling and upgrading skills as well as discussing obstacles they have faced during the process. It is also recommended that CMs should be involved in the planning phase for the refresher training. Establishment of Delivery Station / Birth Station: It is also recommended to establish a delivery station/birth station within villages for longer sustainability of the project. These stations will be used for regular check-ups and for delivery. It will also be convenient for both CMs and community to interact easily in times of emergencies. Active Engagement of District Headquarter Doctors in Referral Mechanism: Currently there is limited engagement with doctors in District Headquarter Hospitals in terms of providing treatment to NRDP referral cases. The programme should explore the possibility of nominating a focal point at the district headquarter hospital for NRDP beneficiaries for better follow-up. Other possibilities like research collaboration should be explored for the active


engagement of doctors with NRDP (e.g. looking at the impact on health for NRDP beneficiaries as compared to control groups). Further, currently there is limited or no formal follow-up by the CMs of the women referred to government hospitals. It is suggested that case profiles for each of these referrals (especially for serious cases) should be developed, and the programme team should follow up on these referrals. This will help to address health issues in the community through public facilities more effectively. Issue regarding Charging of Smart Phones There was long load shedding and/or no electricity in all the villages this evaluation visited. As a result, the smart mobile phones cannot always be recharged and this hinders the ability of the CMs to communicate with doctors. It is recommended to find ways enabling women to recharge these phones. Language of the Smart Phone Software: As most of the community members are unable read and write English confidently, it is recommended to change the language of the software in the smart phone from English to Urdu. Availability of Credit for Text Messages: The target communities for this programme are extremely poor and cannot afford to buy credit in their phones. The responsibility for getting credit usually lies with the CMs who seldom have any other income sources. It is recommended that for the first six months, the programme should buy text messaging packages which can cost as little as PKR 200 per phone per month. The evaluators believe this activity may also be linked up with the Corporate Social Responsibility aspect of the telephone service provider. In the medium and long term, community funds can be used to support the CMs with text messaging costs.

4.2.3 Health and Literacy Classes Extending Literacy Classes to Excluded Women: Currently literacy classes consist of 32 women learners and are unable to accommodate all women in a village. As villages in Narowal are larger than in many other areas, it is advisable to extend the programme of classes by another six months in order to include women in the targeted villages who could not join these trainings earlier. This will enable all women in a village to have basic literacy skills. Flexible Timing for Literacy Classes to Reduce Dropouts: The evaluation team was told that during the sowing and harvesting seasons, attendance went down in the health literacy classes. It is recommended to adjust the timings of classes in line with community engagements. This will reduce the number of absentees. The evaluation team believe NRDP could also use participatory planning methods, like preparing a sessional calendar that could address this issue. If a student misses her class more than three times, it is suggested to individually follow up with the aim to bring her back to class.


4.2.4 Registration of MSGs and WHCs: It is recommended to register MSGs and WHCs as CSOs with clear and specific mandates with government departments in the future. These groups will be then be enabled to raise voices for their rights on different forums. The evaluation team believes that government officials take a formal community based structure more seriously as compared to an unregistered community entity.

4.2.5 Men’s Support Groups Additional Training for MSG: Currently MSG members are provided with a limited number of trainings on issues related to mother and child health care. MSG members were of the opinion there should be additional training for them on these issues just like there is for WHC members. MSG members believe this would further their understanding of mother and child health related issues and also equip them to better respond to these challenges. MSG members also expressed an interest in receiving training on other topics such as financial management and organisation, hygiene, knowledge of common diseases, effective advocacy, how to do awareness raising. Most of these topics are currently taught to WHC members as well.


ANNEX 1 EVALUATION MATRIX

Evaluation Criteria Evaluation Questions

Data Collection Instruments

NRDP/FTM Staff Interviews

Beneficiaries FGDs

Key Informant Interviews

Desk Review of Project Records

Observation through

Field Visits

Relevance: To what extent does the

project respond to the priorities and

needs of the target population?

To what extent did the grantee support achievement towards the MDGs, specifically off-track MDGs?

To what extent did the project target, reach and benefit the poor and marginalised?

To what extent did the project mainstream gender equality in the design and delivery of activities (and/or other relevant excluded groups)?

To what extent do beneficiaries perceive that the activities provided by the project are relevant to their needs and priorities?

Efficiency: To what extent has the

project achieved its objectives?

To what extent are projects delivering activities that are PROVIDING value for money with regards to their economy and efficiency?

To what extent has project delivery provided value for money?

To what extent has project used evidence to improve programming?





Beneficiaries FGDs



Observation through

Field Visits

What are the key drivers and barriers affecting the delivery of results?

Effectiveness: What are the key drivers and barriers (factors) affecting the ability of WHC & MSG to achieve the overarching objective of poverty alleviation?

In what ways has the project worked with others to enable them to effectively deliver their activities?

To what extent do the activities, services and results being delivered by the project align with the intended objectives of Big Lottery Fund?

Sustainability: How sustainable are the activities funded by the project and has the project been successful in leveraging additional interest and investment?

To what extent has the project leveraged additional resources (financial and in-kind) from other sources? What extent has this had on the capacity of the project to deliver its activities?

To what extent is the project engaging with other actors to ensure their interventions complement existing activities?

To what extent are the benefits realised by civil society groups sustainable?





Beneficiaries FGDs



Observation through

Field Visits

Impact: What has the project achieved that would not have been achieved without external funding?

What difference has the project made that would not otherwise have been achieved without project funding?

To what extent has the project put in place M&E systems that are fit for purpose and are being used to cover current health practices delivered by project?

To what extent has the project changed the capacity of civil society and the lives of the poor and marginalised in ways that would not otherwise have been achieved?

How many people are receiving support from grantees that otherwise would not have received support? And how many of these people were unintended beneficiaries?

Case Studies

Can you Identify some best case studies related to the projects implemented by NRDP? (Yes or No)

If yes, please document it.

Best practices

Can you Identify some best practices related to the projects implemented by NRDP? (Yes or No)

If yes, why do you think these were the best practices?

If yes, can this be replicated in other places? (Yes or No)





Beneficiaries FGDs



Observation through

Field Visits

If yes or no, why or why not it can be replicated

Key Lessons Learnt

Can you Identify some key lessons learnt related to these projects implemented by NRDP 2015? (Yes or No)? (Yes or No)

If yes, why do you think these were the key lessons learnt?

Is there is a need to make these lessons learnt part of future design?


ANNEX 2 LIST OF VILLAGES FOR COHORT 1

National Rural Development Programme

Targeted Villages HELP for Women Project – Narowal

Sr# Village Name Cluster Union Council Tehsil District

1 Bhattian Kalan Hakeem Pura Langah Shakargarh Narowal

2 Hakeem Pura Hakeem Pura Wali Pur Bora Shakargarh Narowal

3 Khosa Hakeem Pura Wali Pur Bora Shakargarh Narowal

4 Bhattian Khurd Hakeem Pura Langah Shakargarh Narowal

5 Balaki Chack Hakeem Pura Wali Pur Bora Shakargarh Narowal

6 Jermian Hakeem Pura Wali Pur Bora Shakargarh Narowal

7 Budhy Langah Hakeem Pura Langah Shakargarh Narowal

8 Goal Basti Hakeem Pura Wali Pur Bora Shakargarh Narowal

9 Goal Hakeem Pura Wali Pur Bora Shakargarh Narowal

10 Faredky Langah Hakeem Pura Langah Shakargarh Narowal

11 Mirzy Wali Goralla Goralla Shakargarh Narowal

12 Zahid Pur Goralla Bua Shakargarh Narowal

13 Nangal Goralla Kanjroor Shakargarh Narowal

14 Malik Bagh Goralla Langah Shakargarh Narowal

15 Olakh Goralla Malah Shakargarh Narowal

16 Achal Wali Goralla Goralla Shakargarh Narowal

17 Kathowali Goralla Goralla Shakargarh Narowal

18 Melo Selo Goralla Bua Shakargarh Narowal

19 Fazal Pur Goralla Bua Shakargarh Narowal


20 Bowani Pur Goralla Bua Shakargarh Narowal

21 Cheema Kaleer Mehlowala Ghona Shakargarh Narowal

22 Natt Mehlowala Ghona Shakargarh Narowal

23 Einowali Mehlowala Mehlowala Zafarwal Narowal

24 Sarjal Mehlowala Mehlowala Zafarwal Narowal

25 Fatowal Mehlowala Dhodu Chak Shakargarh Narowal

26 Fatah Toor Mehlowala Mehlowala Zafarwal Narowal

27 Sangyal Mehlowala Mehlowala Zafarwal Narowal

28 Budha Pind Mehlowala Mehlowala Zafarwal Narowal

29 Dongian Murali Dongian Narowal Narowal

30 Sathyala Mehlowala Jabal Zafarwal Narowal

31 Khara Megah Jasar Nedoky Narowal Narowal

32 Dhani Dev kalar wala Jasar Chandarky Narowal Narowal

33 Fasih Pur Jasar Jasar Narowal Narowal

34 Dehlozi Jasar Jasar Narowal Narowal

35 Malok Pur Jasar Jasar Narowal Narowal

36 Thethar wali Jasar Talwandi Narowal Narowal

37 Rasool Pur Jasar Talwandi Narowal Narowal

38 Dhanvady Jasar Jasar Narowal Narowal

39 Daty wal Jasar Talwandi Narowal Narowal

40 Purana wala Jasar Jasar Narowal Narowal

41 Sahdoky (Mandihali) Murali Dongia Narowal Narowal

42 Virk Murali Bobak Murali Narowal Narowal

43 Bobak Murali Bobak Murali Narowal Narowal

44 Mundiky Murali Bobak Murali Narowal Narowal


45 Saddowala Newa Murali Saddowala Narowal Narowal

46 Awan Murali Bobak Murali Narowal Narowal

47 Data Goraya Murali Bobak Murali Narowal Narowal

48 Maan Murali Sadowala Narowal Narowal

49 Dairh Murali Bobak Murali Narowal Narowal

50 Akal Garh Murali Dongia Narowal Narowal


ANNEX 3 FOCUS GROUP DISCUSSIONS GUIDE

Mid-Term Evaluation of “Health Education and Literacy Project for Women” (HELP for Women project) in Narowal, Pakistan

Dated: / / 2015

Name of Facilitator:

Instructions for facilitator:

The interview guide covers only the participants who are beneficiaries of Feed the Minds and will be selected for FGD.

Please introduce clearly the objective and purpose of this interview before starting it.

Please use this questionnaire when interviewing the targeted group.

Please make sure that all questions are covered in the discussion

Part A

1. General Information about FGD

Interviewer Name

1. 2.

Province District

Tehsil / Taluka Union Council

Village # of FGD participant

FGD Start Time: End Time: _________

# Name of Participants

Name of Father

Gender (M/F)

Age Occupation Disability Literacy

Level

1

2

3

4

5

6

7


8

9

10

11

12

Part B

1. Have you heard about Feed the Minds and / or NRDP? 2. Have you participated in Women’s Health Committee or any other committees

(e.g. Men’s Group) under this project? When are they established? What are the functions of these committees? How frequently these committees were meeting? What can be done to make these committees more functional? Do they have any impact on women and children?

3. Were there any traders, farmers, labourers, and different religious group people in these committees? How were they selected? Were there any women in these committees? How were they selected? Are any of these women are widows, disabled, non-literate women? What can be done more to increase role of women in these committees?

4. Were you accessing Community Midwife before? Are you accessing Community Midwife now? What has change since project start regarding access to community midwives

5. Have you seen any improvement in terms of additional health services in your village in last six months? If yes, has it been because of NRDP? Were there any engagements with local government leaders and NGOs to lobby for additional health services by NRDP?

6. Do you know about Men’s Support Groups? Have you participated in these groups or interacted with them? What about you think about the role of men and boys in disseminating messages around the survival and welfare of pregnant women and infants? What can be done to improve the role of Men’s Support Group?

7. Do you have any fund for marginalised women to access health services? Have you availed this fund? Have you contributed to this fund? How can this fund be better improved? How can fundraise and lobbying be improved.

8. Do you know about any of NRDP run community workshops and awareness-raising events using information, education and communication materials, theatre and song? Have you participated in it? What was the impact of these activities? How can they be further improved?

9. Have you ever participated in NRDP bi-annual knowledge sharing workshops? Do you know who participated in it?


10. Have they recruited a Health Literacy Facilitator from the local communities? Are they taking health literacy classes? If yes, please explain the impact on the communities? If no, please let us know why not. What can be improved about this health literacy facilitator’s function?

11. Do you know about the overall project activities and mechanisms available for addressing issues related to reproductive systems, normal pregnancy and ante-natal care, danger signs during pregnancy, labour and delivery, treatment and referral, post-natal care of mothers, new-born baby examination and care (i e vaccinations and danger signs), breastfeeding and nutrition, family planning, first aid, preventable diseases, sanitation and family nutrition? If not, why?

12. What type of interventions / support you have received from FTM / NRDP program?

13. Did interventions have satisfied the greatest needs where needs were greatest? 14. In your opinion which intervention was most appropriate to cater for your

needs? 15. Was provided support appropriate to cater for the needs related to maternity and

child health. If yes, how? And If not, why? 16. Did FTM / NRDP keep close liaison with Men’s Support Groups (MSGs) about

deliverance of trainings to Women’s Health Committee (WHCs)? What sort of activities MSGs have carried out so far?

17. Do you know how FTM / NRDP selected HL facilitators from the community? Are you satisfied with the selection criteria or not? If not, why? Are women in your village get health related knowledge from HLC’s?

18. Are you aware of any training activity and its selection criteria? If yes, have you or any other member of committee received the training? If you were not selected for the training, what was the main reason behind it? Have you filled any complaint against FTM / NRDP? Were your issue resolved or still pending?

19. If you have received the training, are you satisfied with its quality or not? If not, why? Please also state about the location and language in which the training was conducted by HL Facilitator?

20. Are you satisfied with the performance of Community Midwives as well as equitable access to them? If yes, how you can say so? If no, why? In latter case, have you launched any complaint against community midwives? Have you ever paid to any CM? How much?

21. Are you aware of any complaint mechanism installed by FTM / NRDP? Are you satisfied with its performance? Please explain it in detail.

22. Have your knowledge increased on maternity and child health? If yes, how you can say so.

23. In case complicated pregnancy or any other issue, do you have any knowledge about FTM / NRDP referral system? If yes, how its work? If not, why not?

24. Have you got Information, Education and Communication (IEC) material? In which language, training materials were printed and sessions were delivered?


25. Do you believe FTM / NRDP activities contributed towards improving food intake by Pregnant and Lactating Women - PLWs? Do you believe FTM / NRDP activities have sustainable impacts on PLW lives? If yes, how?

Thank you very much for your time and valuable discussion


ANNEX 4 KEY INFORMANT INTERVIEWS

Key Informant Interviews Tools Mid-Term Evaluation of “Health Education and Literacy Project for

Women” (HELP for Women project) in Narowal, Pakistan

Introduction:

Assalamoalekum, my name is ….. We are conducting a survey in (name of the district) on communities views about NRDP Project. We will be interviewing different key informants to help us improve the program we are implementing in your community. I thank you for participating in this survey. I assure that your answers will remain confidential and we would not keep record of your name or other personal information. I also want to assure you that you have the right to stop this interview at any time and feel free to skip questions that you don’t want to answer. Let me also share with you that there are no right or wrong answers. The more candid your answers are, the better we shall understand your views on the issue of women and economy.

Interview will take about 30-40 minutes to complete. Your participation is voluntary and your opinion will be very helpful.

Do you have any questions? Do I have your consent for this interview? (TICK ACCORDINGLY)

[ ] DOES NOT AGREE TO BE INTERVIEWED THANK & WIND UP

[ ] AGREES TO BE INTERVIEWED

Q. 1

Are you familiar with NRDP Project? Do you think the project proposal was

addressing major needs of the targeted communities? If yes, how it was computed

and/or identified? If not, why not?

Q. 2 To what extent pre-assessment findings of the targeted area built the initial framework

of the project? Please elaborate it in detail? Please also describe the NRDP systems to

improve monitoring in order to control project / program quality and the delivery of

support services?


Q. 3 Have you taken into account the specific needs of maternal and child health (MCH) in

project designing phase? How gaps, keeping in mind the local context, were identified

and how their needs were addressed? Please also tell us about their involvement in

project implementation and in activities on ground?

Q. 4 What sort of transparent and accountability mechanisms were adopted in project

implementation phase? What gaps so for you have identified in NRDP accountability

mechanism and any suggestion / recommendation that how to improve it?

Q. 5 Can you scale up that project activities were aligned with specific needs of the targeted

communities in the targeted areas? Do you think these procedures to target

communities were fair and transparent? why not? If not, what is the reason behind it?

Please explain what measures were taken to avoid wastage of resources?

Q. 7 To what degree NRDP project activities were aligned with donor strategic plans and

guidelines? (for NRDP staff)

Q. 8 Do you think FTM/NRDP activities in different components of improving maternal

health and infant survivals are sustainable?

Q. 9 Please let us know about involvement of local stakeholders in the formation and

executing phases of the project of the targeted geographical area? To what degree is

this project parallel to FTM / NRDP national strategic plan and with FTM agenda? Is

this project provides any help to other organizations in identification of need or

complement with their activities? If yes, please give any instances? (for NRDP staff)

Q. 10 What local, provincial, national and international (if applicable) forums you have

attended and efficiently corresponded NRDP/FTM activities in the targeted area /

targeted country with different stakeholders?

Q. 11 Can you integrate FTM / NRDP project with other sectors like Nutrition and WASH?

If yes, how is it possible? If not, what are the main obstacles behind it? Please elaborate

it in detail.

Q. 12 How villages and beneficiaries were selected and formation of committees for FTM /

NRDP intervention and were balance maintained with respect to women and other

vulnerable groups including girls? Have you used any gender disaggregated data?

Have you involved community members or any local committee in the selection

process? If not, why not? If yes, what was there role? Please also elaborate roles and

responsibilities of new committees formed by FTM / NRDP in detail? (for NRDP staff)

Q. 13 How FTM / NRDP selected villages for intervention? Do you think the village

selection criteria used was appropriate and transparent? If yes, how you can claim it?


If no, what’s the main cause of it?

Q. 14

Q.15

Have any deviation is recorded and how it was addressed? Please explain by giving

recorded incident as an example with supporting means of verifications.

Please also state about the FTM / NRDP referral System in detail?

Q. 16 Would you like to identify any gaps needs to be addressed in the project? Would you

like to recommend continuation of the FTM / NRDP project in same spirit without any

changes? If not, why not?

Q. 17 Do you really think the project was economical in terms of resources and inputs?

Q. 18 Did the project achieve proposed and targeted indicators within proposed time so for?

What were the main challenges to not achieve within time?

Q. 19 How effective was Women’s Health Committees (WHCs) and Community Midwives

(CM) model in achieving MDG goals? Did WHCs and CM approach really works?

Q. 20 To which extent FTM / NRDP senior management team and available project team

were capable and supportive in the achievement of the project? Please explain in

detail.

Q. 21 Do you believe sustainability of this project? What measures were taken to sustain

project outcome for longer period of time till date?

Q. 22 To which extent FTM / NRDP activities will contributes significant impact on existing

mortality and morbidity rate? Have you monitored and evaluated the changes in

health in the targeted communities? If not, why not.

Thank you very much for your time and valuable discussion


ANNEX 5 EVALUATION QUESTIONS AND RELATED KII/FGD QUESTIONS

Evaluation Criteria

Evaluation Questions Relevant Questions and Associated Questions in the Tools

General

Have you heard about Feed the Minds and / or NRDP? (Question 1 of FGD),

Are you familiar with NRDP Project? (Question 1 of KII)

This will help us establish whether or not the participants are beneficiaries of the programme and/or if they have received any kind of support from the programme.

Relevance: To what extent does

the project respond to the priorities and needs of the

target population?

To what extent did the grantee support achievement towards the MDGs, specifically off-track MDGs?

How effective was Women’s Health Committees (WHCs) and Community Midwives (CM) model in achieving MDG goals? Did WHCs and CM approach really works? (Q 19 of KII)

Does the Women’s Health Committee or any other committees (e.g. Men’s Group) under this project have any impact on women and children? (Q2: in FGD) Do you have any fund for marginalised women to access health services? Have you availed this fund? (Q7 in FGD) (Once we establish there is a positive impact of the activities on women and children health, then we will be able to link it up with MDGs based on literature review)

To what extent did the project target, reach and benefit the poor and marginalised?

Have you participated in Women’s Health Committee or any other committees (e.g. Men’s Group) under this project? (Q2: in FGD)Do you know about Men’s Support Groups? Have you participated in these groups or interacted with them? (Q6: in FGD)Do you have any fund for marginalised women to access health services? (Q7 in FGD).

Were there any traders, farmers, labourers, and different religious group people in these committees? How were they selected? Were there any women in these committees? How were they selected? Are any of these women are widows, disabled, non-literate women? What can be done more to increase role of women in these committees? (Q3 in FGD)


Evaluation Criteria


Have you ever participated in NRDP bi-annual knowledge sharing workshops? Do you know who participated in it? (Q9 in FGD) Do you know how FTM / NRDP selected HL facilitators from the community? Are you satisfied with the selection criteria or not? If not, why? Are women in your village get health related knowledge from HLC’s? (Q17 in FGD) (This will let us know engagement of poor people in the project. Attendance sheet will help us find out demographic of the participants like age, gender, disability etc. and link it up with their participation in the programme. Part of the evaluation question related to benefit will be answered by remaining part of Q2 which relates with the impact of the project).

To what extent did the project mainstream gender equality in the design and delivery of activities (and or other relevant excluded groups)?

How villages and beneficiaries were selected and formation of committees for FTM / NRDP intervention and were balance maintained with respect to women and other vulnerable groups including girls? Have you used any gender disaggregated data? Have you involved community members or any local committee in the selection process? If not, why not? If yes, what was there role? Please also elaborate roles and responsibilities of new committees formed by FTM / NRDP in detail? (For NRDP staff). How FTM / NRDP selected villages for intervention? Do you think the village selection criteria used was appropriate and transparent? If yes, how you can claim it? If no, what’s the main cause of it? (Q 12 and 13 of KII)

Were there any women in these committees? How were they selected? Are any of these women are widows, disabled, non-literate women? What can be done more to increase role of women in these committees? (Q3 in FGD) Do you know about Men’s Support Groups? Have you participated in these groups or interacted with them? (Q6: in FGD)Have you ever participated in NRDP bi-


Evaluation Criteria


annual knowledge sharing workshops? Do you know who participated in it? (Q9 in FGD)

(This question along with beneficiary data for the project will help us understand mainstreaming of gender and other excluded groups in the project design and activities).

To what extent do beneficiaries perceive that the activities provided by the project are relevant to their needs and priorities?

Have you taken into account the specific needs of maternal and child health (MCH) in project designing phase? How gaps, keeping in mind the local context, were identified and how their needs were addressed? Please also tell us about their involvement in project implementation and in activities on ground? (Q3 of KII)

To what extent pre-assessment findings of the targeted area built the initial framework of the project? Please elaborate it in detail? (Q 2of KII)

Were you accessing Community Midwife before? Are you accessing Community Midwife now? What has change since project start regarding access to community midwives (Q4 in FGD) Do you have any fund for marginalised women to access health services? Have you utilised it? (Q7 in FGD)

Do you think the project proposal was addressing major needs of the targeted communities? If yes, how it was computed and/or identified? If not, why not? (Q1 of KII)

(These questions will help us understand whether or not people were accessing relevant services before and if yes, whether or not they needed similar services. In case they were not receiving these services, whether or not they were in need of such services. This way we will establish whether or not such services were


Evaluation Criteria


needed for them). Do you know about the overall project activities and mechanisms available for addressing issues related to reproductive systems, normal pregnancy and ante-natal care, danger signs during pregnancy, labour and delivery, treatment and referral, post-natal care of mothers, new-born baby examination and care (i e vaccinations and danger signs), breastfeeding and nutrition, family planning, first aid, preventable diseases, sanitation and family nutrition? If not, why? What type of interventions / support you have received from FTM / NRDP program? Did interventions have satisfied the greatest needs where needs were greatest? In your opinion which intervention was most appropriate to cater for your needs? Did interventions have satisfied the greatest needs where needs were greatest? (Q12 – 15 of FGD) (These questions will not only help understand the extent to which communities are aware of the activities but also the extent to which they are relevant to their needs)

Efficiency: To what extent has

the project achieved its objectives?

To what extent are projects delivering activities that are providing value for money with regards to their economy and efficiency?

Do you really think the project was economical in terms of resources and inputs? (Q 17 of KII) Did the project achieve proposed and targeted indicators within proposed time so for? What were the main challenges to not achieve within time? (Q 18 of KII) (This will help us understand value for money within the project)

To what extent has project delivery provided value for money?

Can you scale up that project activities were aligned with specific needs of the targeted communities in the targeted areas? Please explain what measures were taken to avoid wastage of resources? (Q6 of KII) (This will help us understand usage of resources within the project)


Evaluation Criteria


To what extent has project used evidence to improve programming?

(this question will be linked up with the changes in the design since start of the activities and will be based on review of different documents received from the communities)

What are the key drivers and barriers affecting the delivery of results?

Can you integrate FTM / NRDP project with other sectors like Nutrition and WASH? If yes, how is it possible? If not, what are the main obstacles behind it? Please elaborate it in detail.(Q11 of KII) (This will help us understand key obstacles and drivers affecting delivery of results)

Effectiveness: What are the key drivers and barriers (factors) affecting the ability of WHC & MSG to achieve the overarching objective of poverty alleviation?

In what ways has the project worked with others to enable them to effectively deliver their activities?

Have you seen any improvement in terms of additional health services in your village in last six months? If yes, has it been because of NRDP? Were there any engagements with local government leaders and NGOs to lobby for additional health services by NRDP? (Q5 in FGD) Did FTM / NRDP keep close liaison with Men’s Support Groups (MSGs) about deliverance of trainings to Women’s Health Committee (WHCs)? What sort of activities MSGs have carried out so far? (Q16 in FGD)

To what degree NRDP project activities were aligned with donor strategic plans and guidelines? (for NRDP staff) (Q 7 of KII)

(This question will help us understand whether or not the programme worked with other actors to deliver activities more efficiently). Do you know about Men’s Support Groups? Have you participated in these groups or interacted with them? What about you think about the role of men and boys in disseminating messages around the survival and welfare of pregnant women and infants? (Q6 in FGD)

(This question will help us understand engagement of men and boys in the


Evaluation Criteria


programme activities).

To what extent do the activities, services and results being delivered by the project align with the intended objectives of GPAF?

(This question will be answered based on literature review and review of documents and overall activity framework)

Sustainability: How sustainable are the activities funded by the project and has the project been successful in leveraging additional interest and investment?

To what extent has the project leveraged additional resources (financial and in-kind) from other sources? What extent has this had on the capacity of the project to deliver its activities?

Do you have any fund for marginalised women to access health services? Have you availed this fund? (Q7 in FGD)

(This question will help us understand available funding from other sources).

To what extent is the project engaging with other actors to ensure their interventions complement existing activities?

Please let us know about involvement of local stakeholders in the formation and executing phases of the project of the targeted geographical area? To what degree is this project parallel to FTM / NRDP national strategic plan and with FTM agenda? Is this project provides any help to other organizations in identification of need or complement with their activities? If yes, please give any instances? (for NRDP staff) (Q 9 of KII) Were there any engagements with local government leaders and NGOs to lobby for additional health services by NRDP.(Q5 of the FGD) What local, provincial, national and international (if applicable) forums you have attended and efficiently corresponded NRDP/FTM activities in the targeted area / targeted country with different stakeholders? (Q10 of the FGD) (This question will help us understand engagements from/with others).

To what extent are the benefits realised by civil society groups sustainable?

Do you think FTM activities in different components of improving maternal health and infant survivals are sustainable? (Q8 of KII) Do you believe sustainability of this project? What measures were taken to


Evaluation Criteria


sustain project outcome for longer period of time till date? (Q21 of KII) (This question will help us understand sustainability of the program for different groups including civil society).

Impact: What has the project achieved that would not have been achieved without external funding?

What difference has the project made that would not otherwise have been achieved without project funding?

To which extent FTM / NRDP activities will contributes significant impact on existing mortality and morbidity rate? Have you monitored and evaluated the changes in health in the targeted communities? If not, why not.(Q 22 of KII) Access to Community Midwives – before and after their intervention (Q4 of FGD) (This question will help us understand impact of NRDP in terms of access to mid wives in the community). Improvement in terms of additional health services in the village in last six months? If yes, has it been because of NRDP? Were there any engagements with local government leaders and NGOs to lobby for additional health services by NRDP? (Q5 of FGD) Have your knowledge increased on maternity and child health? If yes, how you can say so. (Q22 of FGD) (These questions will help us understand impact of NRDP in terms of additional health services in the communities and knowledge on maternity and child health). NRDP run community workshops and awareness-raising events using information, education and communication materials, theatre and song? What was the impact of these activities? (Q8 of FGD) (This question will help us understand impact of NRDP community workshops and related activities impact on the communities).


Evaluation Criteria


Have they recruited a Health Literacy Facilitator from the local communities? Are they taking health literacy classes? If yes, please explain the impact on the communities? If no, please let us know why not. (Q10 of FGD) (This question will help us understand NRDP Health Literacy Facilitator and related activities impact on the communities). In case complicated pregnancy or any other issue, do you have any knowledge about FTM / NRDP referral system? If yes, how its work? Do you believe FTM / NRDP activities contributed towards improving food intake by Pregnant and Lactating Women - PLWs? Do you believe FTM / NRDP activities have sustainable impacts on PLW lives? If yes, how? (Q25 of FGD) (These two questions will help us understand impact of NRDP activities on expecting mothers).

To what extent has the project put in place M&E systems that are fit for purpose and are being used to cover current health practices delivered by project.

What sort of transparent and accountability mechanisms were adopted in project implementation phase? What gaps so for you have identified in NRDP accountability mechanism and any suggestion / recommendation that how to improve it? (Q4 of KII) Please also describe the NRDP systems to improve monitoring in order to control project / program quality and the delivery of support services? (Q 2 of KII) Are you aware of any complaint mechanism installed by FTM / NRDP? Are you satisfied with its performance? Please explain it in detail. (Q21 of FGD). Have you filled any complaint against FTM / NRDP? Were your issue resolved or still pending? (Q18 of FGD). Are you satisfied with the performance of Community Midwives as well as equitable access to them? If yes, how you can say so? If no, why? In latter case, have you launched any complaint against community midwives? Have any deviation is recorded and how it was addressed? Please explain by giving recorded incident as an example with supporting means of verifications.


Evaluation Criteria


Please also state about the FTM / NRDP referral System in detail? (Q 14 and 15 of KII)

(These questions will help us understand beneficiary complaint and feedback mechanism in the programme).

To what extent has the project changed the capacity of civil society and the lives of the poor and marginalised in ways that would not otherwise have been achieved?

Are you aware of any training activity and its selection criteria? If yes, have you or any other member of committee received the training? If you were not selected for the training, what was the main reason behind it? (Q18 of FGD) If you have received the training, are you satisfied with its quality or not? If not, why? Please also state about the location and language in which the training was conducted by HL Facilitator? (Q19 of FGD) (This question will help us capacity development interventions in the programme).

How many people are receiving support from grantees that otherwise would not have received support? And how many of these people were unintended beneficiaries?

(This question will be answered by cross comparison of before and after intervention status/situation of people).

Case Studies

Can you Identify some best case studies related to the projects implement NRDP? (Yes or No)

Do you have any fund for marginalised women to access health services? Have you availed this fund? Have you contributed to this fund? (Q7 in FGD)

(This question will help us develop a case study around fund for women to access health services). Do you know about any of NRDP run community workshops and awareness-raising events using information, education and communication materials,

If yes, please document it.


Evaluation Criteria


theatre and song? Have you participated in it? (Q8 in FGD)

(This question will help us develop a case study around NRDP community workshops and related activities). Have they recruited a Health Literacy Facilitator from the local communities? Are they taking health literacy classes? If yes, please explain the impact on the communities? (Q10 of FGD) Have you got IEC material? In which language, training materials were printed and sessions were delivered? (Q24 of FGD) (This question will help us develop a case study around NRDP Health Literacy Facilitator and related activities). Was provided support appropriate to cater for the needs related to maternity and child health. (Q15 of FGD) (This question will help us develop a case study around NRDP interventions related to maternity and child health).

Best practices

Can you Identify some best practices related to the projects implemented by NRDP? (Yes or No)? (Yes or No)

Do you have any fund for marginalised women to access health services? Have you availed this fund? Have you contributed to this fund? How can this fund be better improved? How can fundraise and lobbying be improved. (Q7 in FGD)

(This question will help us identify best practices related to funds for women to access health services). Do you know about any of NRDP run community workshops and awareness-raising events using information, education and communication materials, theatre and song? Have you participated in it? What was the impact of these activities? How can they be further improved? (Q8 in FGD) (This question will help us develop a case study around community workshops and related activities) Health Literacy Facilitator from the local communities? Are they taking health

If yes, why do you think these were the best practices

If yes, can this be replicated in other places (Yes or No)

If yes or no, why or why not it can be replicated


Evaluation Criteria


literacy classes? If yes, please explain the impact on the communities? (Q10 of FGD) (This question will help us document best practices around NRDP Health Literacy Facilitator and related activities). Was provided support appropriate to cater for the needs related to maternity and child health. If yes, how? And If not, why? (Q15 of FGD) (This question will help us develop best practices around NRDP interventions related to maternity and child health).

Key Lesson Learnt

Can you Identify some key lesson learnt from related to these projects implemented by NRDP 2015? (Yes or No)? (Yes or No)

What can be done to improve the role of Men’s Support Group? (Q6 in FGD)

Fund for marginalised women to access health services: How can this fund are better improved? How can fundraise and lobbying be improved. (Q7 in FGD) (These questions will help us capture some of the lesson learnt and also help us understand engagement of men and boys in the programme activities). NRDP run community workshops and awareness-raising events using information, education and communication materials, theatre and song? How can they be further improved? (Q8 in FGD) (This question will help us capture some of the lesson learnt around community workshops and related activities). Was provided support appropriate to cater for the needs related to maternity and child health. If yes, how? And If not, why? (Q15 of FGD) (This question will help us develop lesson learnt around NRDP interventions related to maternity and child health). Health Literacy Facilitator from the local communities: What can be improved about this health literacy facilitator’s function? (Q10 of FGD) (This question will help us document key lesson learnt around NRDP Health Literacy; Facilitator and related activities). In case complicated pregnancy or any other issue, do you have any knowledge

If yes, why do you think these were the key lesson learnt?

Is there is a need to make these lesson learnt part of future design?


Evaluation Criteria


about FTM / NRDP referral system? If yes, how its work? If not, why not? (23 of FGD) (This question will help us capture lesson learnt around access to expecting mothers, especially those experiencing complications). Would you like to identify any gaps needs to be addressed in the project? Would you like to recommend continuation of the FTM / NRDP project in same spirit without any changes? If not, why not? (Q 16 of KII)

independent mid-term evaluation report...may 2016 – final version independent mid term evaluation...

Documents