microfinance-impact-assessment-designs (august 2013).pdf

8/10/2019 Microfinance-Impact-Assessment-Designs (August 2013).pdf

1/36

Microfinance Impact Assessment Designs

Page - 1


Joan Hall, Consultant

August 2013


2/36


Page - 2

Acknowledgements

Te consultant thanks LIF staff or their kind attention in providing inormation and arranging the logis-tics or this mission. As well, she is very appreciative o the willingness o MFI and other managers to meet

with her during the field work.

Copyright notice

Tis report may be reproduced in whole or in part and in any orm or educational or non-profit purposeswithout special permission rom LIF Fund, provided the reproduction includes this Copyright noticeand the Disclaimer below. Te LIF Fund would appreciate receiving a copy o any publication that usesinormation rom LIF as a source. No use o this data may be made or resale or or any other commer-cial purpose whatsoever without prior permission in writing rom the LIF Fund, email: [email protected]

Disclaimer

LIF Fund donors and UNOPS shall have no responsibility or liability whatsoever in respect o any inor-mation in any external website or in any document using this report.

LIVELIHOODS AND FOOD SECURIY RUS FUND

Livelihoods and Food Security rust FundUNOPS Fund Manager Office

No. 12 (O), Pyi Tu Lane, 7 Mile, Mayangong ownship, Yangon, MyanmarPhone: +95 1 65 7703~06, 65 7280~87, Fax: +95 1 65 7702, +95 1 65 7279Email: [email protected]

Website: www.lif-und.org


3/36


Page - 3

Table of Contents

ACRONYMS .............................................................................................................................. 4

1. General Background of Project/Assignment .................................................................... 5

2. Purpose and Scope of Assignment.................................................................................... 6

3. Methodology ...................................................................................................................... 6

4. Findings of the Desk Review .............................................................................................. 6

5. Alternatives to Impact Assessment ................................................................................... 8

6. Myanmar MFIs and Impact Assessment ........................................................................... 9

7. Proposal RCT with Qualitative Tools (Mixed Methods) ............................................... 11

8. Preparatory Steps ............................................................................................................. 12

9. Hypotheses ....................................................................................................................... 15

10. Challenges ...................................................................................................................... 15

11. Budget/Cost ................................................................................................................... 17

12. Key Indicators for Inclusion in MFI MIS ......................................................................... 18

13. Survey template ............................................................................................................. 19

14. Stakeholder Advisory Committee ................................................................................ 19

15. Next Steps ....................................................................................................................... 20

Appendix 1 ........................................................................................................................... 22

Appendix 2 ............................................................................................................................ 24

Annex A ................................................................................................................................. 26

Annex B ................................................................................................................................. 30

Annex C ................................................................................................................................. 31


4/36


Page - 4

ACRONYMS

3IE International Initiative or Impact evaluation

EA Evaluation AssessmentFIW Financial Inclusion WindowFSP Financial Services ProviderIA Impact AssessmentIP Implementing PartnerLIF Livelihoods and Food Security rust Fund (LIF) MyanmarM&E Monitoring And EvaluationMADB Myanmar Agricultural Development BankMF MicrofinanceMFI Microfinance InstitutionsMIS Management Inormation SystemMMSE Myanmar Microfinance Supervisory EnterpriseNGO Nongovernmental OrganizationsPG Preparation GrantRC Randomised Control rialsRP Request For ProposalSAC Stakeholder Advisory CommitteeoR erms O ReerenceUNDP United Nations Development ProgrammeUSAID United States Agency For International DevelopmentVSLA Village Savings And Loan Associations


5/36


Page - 5

1. General Background of Project/Assignment

UNOPS is the Fund Manager or the Livelihoods and Food Security rust Fund (LIF) in Myanmar. LIF isa multi-donor und or seven years (2010 2016) to address ood insecurity and income poverty in Myan-mar. Te Donor Consortium o LIF comprises Australia, Denmark, the European Community, France, the

Netherlands, New Zealand, Sweden, Switzerland, the United Kingdom and the United States.

Te overall objective o LIF is to contribute resources to a livelihoods and ood security programme withthe aim o making progress towards the achievement o Millennium Development Goal 11(the eradicationo extreme poverty and hunger) in Myanmar. Working through a trust und modality, LIFs purpose is tosustainably increase ood availability and incomes o 2 million target beneficiaries.

Tis is to be achieved through delivering the ollowing programme outputs:1. Increased agricultural production and incomes supported through improved production and posthar-

vest technologies, improved access to inputs and markets.

2. argeted households supported in nonagricultural livelihood activities and/or trained in livelihoodskills or employment.3. Effective social protection measures supported that increase the incomes, enhance the livelihood oppor-

tunities or protect the livelihoods assets o chronically poor households.4. Sustainable natural resource management and environmental rehabilitation supported to protect local

livelihoods.5. Capacity o civil society strengthened to support and promote ood and livelihoods security or the poor.6. Monitoring and evaluation evidence and commissioned studies used to inorm programme and policy

development.

And the ollowing management outputs:7. Funds allocated in line with Fund Board policies and are accounted or in a transparent manner.8. Fund flow and partner perormance monitored and evaluated.

LIF is implemented through a variety o local implementing partners (IPs) who were successul in submit-ting proposals that supported the LIF purpose in the areas targeted.

LIF has long recognized the importance o increasing access to financial services as a means to achieveits outputs or both agriculture and non-agriculture livelihoods. Recently, LIFs baseline study ound that,whilst 83% o households in the our agro-ecological zones took a loan in the prior 12 months, only 16%were able to do so rom a low interest financial services provider (FSP).

LIF has unded a number o microfinance institutions (MFIs) in Myanmar. However, only with the pass-ing o the new Microfinance Law in November 2011 did it become possible or LIF to significantly scale upits assistance to the provision o microfinance services as part o a broader strategy o financial inclusion. InMarch 2012, the LIF Fund Board resolved to open a Microfinance (renamed Financial Inclusion) Win-dow (FIW), now constituted with a total amount o US$30m.

At the same time, there is a growing need to demonstrate that financial inclusion brings real benefits to thepoor in terms o increased income, better housing, clothing, ood and education or their children. LIFwished to take the lead in this area in Myanmar, or example though upcoming projects or working with

some o its IPs.

However, there are several ways to conduct impact assessments, e.g. through experimental research, ran-1 Reduce by hal the proportion o people living on less than a dollar a day; achieve ull and productive employment anddecent work or all, including women and young people; reduce by hal the proportion o people who suffer rom hunger.


6/36


Page - 6

domised control trials (RCs), quasi-experiments or qualitative research. Generally, RCs (ideally accompa-nied by qualitative research) are considered to be the most rigorous approach to impact evaluation, but theycome with issues regarding lead times and cost.

2. Purpose and Scope of Assignment

Te overall purpose o the consultancy is to develop a microfinance impact assessment system that is basedon the best practices available but customized to the particular needs and current standard operating prac-tices o Myanmar. Te oR or this consultancy can be ound in Annex A.

3. Methodology

Te consultancy period was April 29 to July 31, 2013. Te field work was carried out between April 29 andMay 22, 2013. Tis work consisted o a desk review, meetings with stakeholders (LIF, UNDP, and MFIs),

and phone/Skype interviews with evaluation experts. A list o interviewed stakeholders can be ound inAnnex B. A stakeholder debriefing was held on May 17th, at which 8 organizations were present (Annex C:Stakeholder Debriefing).2Te presentation (in PowerPoint ormat) is an annex to this report (Annex D).Subsequently, interviews were done with several local survey firms, to get an idea o the potential cost othe impact assessment. A sample budget is included in Annex D. Tis is only an estimate; the final budgetwill depend on the final design o the impact assessment. In order to hire a research firm with expertise inbest practice impact assessment or microfinance sectors, a sample terms o reerence (oR) is included inAnnex E.

4. Findings of the Desk Review

a. Alternative Types of Impact Assessments

Currently there is significant debate going on between donors and academics about the validity o claimsabout microfinance impact. Some academics claim that microfinance impacts have been exaggerated dueto weak impact assessment methodologies.3Other academics assert that the most rigorous methodologiesare not suitable or microfinance or any development intervention. Stakeholders seem to understand theneed to keep tools practical and cost-effective. Tey also agree that it is important to use the evidence ordecision-making purposes.

CGAP has been mediating this debate, and in January 2013, brought together a group o academics, re-searchers and donor representatives or a discussion o the various methodologies that have been and can beused to determine the impact o MF. Te presentations covered the state o the art or MF impact assessment.

A table o the types o methodologies or MF IA, and their advantages and disadvantages, is below:

2 PAC/UNDP, Myanmar Finance, Save the Children, BRAC, CARD, ACLEDA Myanmar, LIF, and UNDP were repre-sented.3 For example, the DFID publication authored by Duvendack et al, entitled What is the evidence o the impact o mi-crofinance on the well-being o poor people?, 2011, which examines 58 recent MF IAs.


7/36


Page - 7

Table 1: Methodologies for MF IA

Methodology Description Advantage Disadvantage

RandomizedControl rials(RCs)

Long term. Population isgrouped randomly intotreatment* and control

groups. Te same peopleare surveyed at the begin-ning and end o the study.

It is the only way to elim-inated selection bias, sothat any changes to the

treatment group must bedue to the treatment itsel,and not other actors.

Expensive. Requires a long study. Useo control groups may interere withMF expansion. Needs very ocused

hypotheses and can only make con-clusions about a ew hypotheses at atime. op down (i.e. not participa-tory in design). Ethics o not treatingclients with MF during the evaluationperiod. Results cant be generalized toother contexts. Maintaining controlgroups/households/villages is nec-essary and yet may be difficult toenorce. Maintaining the same type otreatment (i.e. the MF methodology)

is necessary and yet may intererewith responding to client demands.

Quasi-Experi-mental rials

Long term. Tere arecontrol groups (that maybe chosen afer the inter-

vention) and treatmentgroups. No randomizationoccurs.**

Has the advantages oRCs, but does not requirerandomization, thereorethere are no ethical issues.

Expensive. Long study. op down.Results cant be generalized to othercontexts. Not as rigorous as RCs dueto potential biases.

ParticipatoryResearch***

Uses methods that involvethe beneficiaries in thedevelopment o hypothe-

ses, data analysis, definingindicators, and the de-

velopment o solutions.Provides a eedback loopbetween researchers andbeneficiaries that canimprove the depth o find-ings. ools such as wealthmapping

Flexible methodology, ad-aptable to local conditions.Can be lower cost. Can

be useul to set the stageor other types o assess-ment, or refining researchquestions, or deepeningunderstanding o findings.Are not top down; cancontribute to learning andempowerment.

Are subjective. Do not establish cau-sality. Require a certain level o skill,and the trust o the participants.

QualitativeResearch****

Investigates the practiceand process rather than

outcomes. Te ocus is onparticipants perceptionsand experiences and theway they make sense otheir lives.

Involves observing andrecording behavior andevents in their naturalsetting. ools includeocus groups, lie histories,

participant observation,community mapping andinstitutional analysis.

Allows or smaller butocused samples. Provides

inormation and under-standing o processes andperceptions. Gives insightsinto intra-householdrelations, causes or causalprocesses. Can access dataon issues that quantitativesurveys cannot.Can help understand whychange does not occur.

Does not give a statistical representa-tion o population; does not establish

causality; Requires high level o skill,may be expensive.


8/36


Page - 8

Methodology Description Advantage Disadvantage

Mixed methodsresearch

Combines quantitativewith qualitative methods

Provides inormation onboth what happened andwhy it happened; deep-ens the analysis.

Expensive, requires skill, takes time.Limited number o hypotheses can beinvestigated.

* reatment means that these people will be getting the MF services, while the control group will not, during the peri-od o the evaluation.** Quasi-experimental design has control groups. But since there is no randomization, the control and treatment groupsmay not be comparable, due to sel-selection bias. See: http://en.wikipedia.org/wiki/Quasi-experiment*** Catley, Andrew, John Burns, Dawit Abebe, and Omeno Suji. Participatory Impact Assessment: A Guide or Practitioners.Feinstein International Center, ufs University. 2008.**** An Overview o Qualitative Research Methods. About.com: Sociology.

Te orum participants concluded that the mixed method research, combining both an RC and qualitativetools, is perhaps the most rigorous as well as useul type o research, i expense is not a constraint. LIF alsowants a rigorous evaluation method that will clearly attribute impact (i any) to the microfinance activities.

Te consultant agrees with that this would be the best approach, given the desires o the stakeholders.

5. Alternatives to Impact Assessment

Tis section discusses tools that can be used as alternatives to doing animpact assessment as well as to supplement an impact assessment.

MonitoringImpact assessment experts in the field o international development, in-cluding in microfinance, place emphasis first on monitoring. Monitoring

is critical to development programs and institutions because it providesinormation in a timely manner that can be used to correct or improveimplementation. Monitoring requires having indicators and targets. Indi-cators can be financial, social, and/or environmental. Tey can be at di-erent levels: individual, household, business, and/or village/community.Monitoring can take place at different stages, such as entry into the pro-gram, during the program, exiting the program, and post-exit. It can take place at periodic times: monthly,annually, etc. Monitoring can be done thematically, or example, to know client satisaction, changes inclients lives, etc. A good monitoring system has a variety o these elements, and most importantly, a goodmonitoring system is designed to supply the inormation that the institution needs.

Monitoring is used or decision-making. Tereore, inormation that is captured by monitoring tools shouldbe analyzed and presented to the appropriate decision-making body(ies) in the institution or program. 4Inormation is costly to collect or the MFI and is an imposition on clients. I inormation is not being used,then there is no point in collecting it.

In Myanmar, analysis and use o monitoring inormation or decision-making seems rudimentary, or threereasons: 1) MFIs or the most part lack the sofware to assist in this; 2) MFI staff are too busy; and 3) MFIstaff may not know how to translate the inormation into appropriate decisions.

Te sofware is a critical element in analyzing data. World Vision has recently installed its new MIS sofwareand reports that the efficiency o its data analysis and the use to which it puts the data have improved greatly.5

4 Different departments have different inormation needs. Different departments have different inormation needs.5 Interview with Neal Youngquist, General Manager, World Vision Myanmar Microfinance Program, 20 May 2013.

While evaluation can bedone some o the time, mon-itoring should be done all thetime.

Source: Highlights: ImpactEvaluation for FinancialInclusion: A Retreat forCGAP Members with LeadingResearchers. January 10-11,2013, London


9/36


Page - 9

Monitoring tools can include, among many others:

Purpose Tool

Financial Monitoring SEEP Frame ool*, MFI Factsheet**

Non-rigorous Client Impact Monitoring SEEP/AIMS ools***

Social Perormance Monitoring@

Social Perormance Management (SPM) Frame-work@@

Client Satisaction MicroSave PRA and FGD tools@@@

argeting Clients Poverty Assessment ools#, Progress out o PovertyIndex##, Poverty Scorecard###

* SEEP Network Frame ool Resources** MicroFact. MFI Factsheet*** Non-rigorous meaning not randomized and without a control group.@ SEEP Network. Learning rom Clients: Assessment ools or Microfinance Practitioners (SEEP AIMS ools)@@ Social Perormance Management Network@@@ MicroSave.

# USAIDs Poverty Assessment ools website.## Grameen Foundations Progress out o Poverty website.### Schreiner, Mark. A Simple Poverty Scorecard or Myanmar. 2012

LearningMFIs ofen collect mountains o inormation and then do not have the capacity to analyze it and use it. Anexample is the loan application or a client, which may contain demographic inormation about the individ-ual and household, financial inormation on the business, and even social data (example: school attendanceo children). Tis data is collected at each new loan, and yet is ofen never compiled and analyzed, not evenby random sampling. So no decisions are made on the basis o this inormation.

In addition to monitoring, there are a significant number o impact studies that have been done on microfi-nance globally. Yet most MFIs, and even their donors, are not aware o the findings o these studies, and havenot incorporated the relevant lessons learned rom these studies in their own operations.6At the same time,MFIs operating in the same environments may not be sharing learning with each other, and MFIs and otherprojects with credit components (revolving unds or armers, or example) may also not be sharing learn-ing between themselves. All o this translates to lost opportunities or improving impact on poor clients, toincreased costs due to monitoring, and to inefficient duplication o efforts.

In Myanmar, many MFI staff are new. Tey are still learning the basics o microfinance, and have not had thetime or opportunity to study microfinance practices in other institutions or countries. Tere is an inormal

MF working group. I this group evolves into an association, this might provide a venue or learning amongpractitioners and rom existing impact studies.

6. Myanmar MFIs and Impact Assessment

Interviews with MFIs in Myanmar show that they are mostly still in the start-up stage. Some are transorm-ing to licensed MFIs, thanks to the new Myanmar MF Law passed in November 2011. Although PAC/UNDP has nearly 600,000 clients, the rest have less than 20,000. Tese new MFIs have many challenges:

Hiring qualied sta; training sta

Product development/renement Monitoring for decision-making MIS purchase and installation

6 For example, see the Innovations in Poverty Actionwebsite, or 3IE.
http://www.poverty-action.org/http://www.3ieimpact.org/media/filer/2013/06/05/3ie_newsletter_june_2013_3.htmlhttp://www.3ieimpact.org/media/filer/2013/06/05/3ie_newsletter_june_2013_3.htmlhttp://www.poverty-action.org/


10/36


Page - 10

Rural outreach Preparing for deposit-taking (for those with a deposit-taking license) Risk management Business and strategic planning Expansion

In the near future, competition

Tese large challenges interere with MFIs ability to measure their impact. Impact occurs over the longterm, whereas these challenges are very immediate. MFIs are dealing with these challenges on a daily basis.

Te greatest difficulty related to monitoring impact or MFIs in Myanmar is the lack o appropriate sofware.For measuring impact rigorously (long term, control group and or randomization), the challenges are theexpertise needed and the expense o doing it.

Yet MFIs in Myanmar seem interested in and committed to measuring the impact o their activities. For

example, PAC, Proximity, and GRE have all attempted to measure the impact o their MF activities.

PACs study (not yet released at the time o this report) on its Sustainable Microfinance to Improve theLivelihoods o the Poor Project analyzed impact in the ollowing way:

A qualitative study, using ocus groups and key inormant interviews, in 12 villages served by the MFPand 12 non-MFP villages. Tis study occurred one time only (2011) and so was not comparative.

Comparisons o clients to non-clients (at the same time) using a survey (this is called a cross-sectionalanalysis)

A panel survey was also conducted (2007, 2011) to show impact o the project over time, but there weretwo actors that limited the validity o the data: i) the limitation o respondents to those living in a MFP

project village did not account or the spillover effects o microfinance within a village, and ii) there wasvariance in how some questions were asked between the 2007 and 2011 surveys.

Te surveys in this study looked at health, quality o housing, number o livestock, amount o armmechanization, availability o wage labor, amount o wages paid, health care expenditures, householdsavings, ood security (amount o ood), rights o women, and other variables.

Te PAC study does not meet the definition o rigorous since it did not randomize client selection.

Te Proximity study ocused on the ollowing: Profile o borrowers

Loan uses Sel-reported results o loans Client satisaction with the methodology

Te Proximity study does not meet the definition o rigorous since it did not randomize client selectionnor did it study changes over time.

Te GRE study (2007) was conducted rom an anthropologic perspective. It was conducted only in thestate o Chin. It had the ollowing characteristics:

In-depth interviews and participatory appraisal techniques over a three-month period Both an anthropologist and an economist conducted in-depth interviews Interviews with clients, non-clients, and ormer clients o the MFI Tirty villages, among them a variety o profiles


11/36


Page - 11

Te GRE study does not meet the definition o rigorous since it did not randomize client selection nordid it study changes over time.

Despite the lack o rigor, all three studies give valuable inormation to MF practitioners, and can be used tomake important changes to MF programming. All three studies should be reviewed by the MF practitioner

community and used to make changes to programs.

7. Proposal RCT with Qualitative Tools (Mixed

Methods)

Tis paper recommends that LIF use a mixed methods approach, including a randomized controlled trial(RC) with qualitative tools (ocus groups and in-depth interviews). Te rationale or using this approach is:

It is the only approach that makes it possible to claim that MF is the reason or the impact (causality)

It seems to be the only approach that will convince the donor community that MF has impact, and Te addition o qualitative methods to an RC allows or a greater depth o understanding o the reasonsor impact

Tere are challenges to doing a mixed method RC. Tese difficulties, and their mitigating strategies, areound in the table below:

Table 2: Challenges of a mixed method RCT and mitigating strategies

Challenge Mitigating Strategy

Finding a sufficient number o matched control

villages*

Since MF is new to Myanmar, this should not be a

problemMaintaining (not contaminating) the control vil-lages or the length o the study

Tis will need to be agreed to by the MF communi-ty in Myanmar, and monitored by the StakeholderAdvisory Committee; participating MFIs shouldsign a memo o understanding with a clause to thiseffect

Use o control villages may interere with MF ex-pansion

MFIs that participate in this study must understandthat the control villages are off limits or expansion;their expansion should be planned into non-con-trol villages; ; they should sign a memo o under-

standing with a clause to this effectEthics o not treating clients with MF during theevaluation period

All stakeholders, especially the government, shouldbe aware o this issue beore the study begins; also,the randomization o clusters (villages) rather thano individuals reduces the ethical concerns

Maintaining the same type o treatment (i.e. theMF methodology) is necessary and yet may inter-ere with responding to client demands.

MFIs participating in the study need to be madeaware that they may not change the methodologythat is being evaluated while the study is underway;they should sign a memo o understanding with aclause to this effect

* Te RC should use separate villages (clusters) as controls, rather than individual households within a village wheresome people are receiving the microfinance services. Tis reduces the possibility o spillovers and contamination.

A good design and a good research firm should be able to manage the challenges, with LIFs assistance toget buy-in rom the MFIs. o start the buy-in process, there was a stakeholder debriefing with MFIs on 17


12/36


Page - 12

May 2013, which covered the definition o impact, o impact assessment, the types o impact assessmenttools and the level o rigor o each, the challenges to doing a random control trial (the most rigorous meth-odology), and the conditions that would need to be met in order to do one.

Te MFI representatives present at the debriefing generally indicated their agreement with a mixed methods

approach.7Tey agreed to send their causal models and expansion plans to the consultant within the 2 weeksollowing the debriefing workshop. Tis did not occur.

LIF asked or advice on the choice o MFIs to include in the IA. Te research firm hired to implement thestudy should chose those MFIs that are using a common methodology, so that the findings are relevant tothe greatest number o MFIs. Te areas o expansion, the presence o existing microfinance providers (com-petitors) and the need or control villages will also determine which MFIs are used. Te MFIs used should bestable institutions, able to control their expansion to the designated villages. Te research firm should limitthe number o MFIs participating to one or two.

Te consultant suggested the idea o using an IA to test new products. Tis has the advantage o helping thesector innovate to develop products with greater impact. It also reduces ethical concerns, since all clientswill receive a product, but only the treatment clients will receive the new product. None o the MFIs sug-gested an innovation to be tested.

Regarding the timing o the IA, LIF has unded five MF projects to date, and their status is the ollowing: 3 projects are ending mid-2014, reaching 50,000 clients. 1 project is ending in 2014, and its targeted clientele is 45,000. Another MFI has just started and plans to reach up 20,000 clients by 2015.

Because o the growth o LIFs own MFI partners, the prolieration o other private MF providers, as well

as the time that it takes to organize an IA, it is advisable or LIF to do the IA as soon as is easible.

8. Preparatory Steps

David Hulme, in his article Impact Assessment Methodologies or Micronance: Teory, Experience andBetter Practice, notes that there are three elements o a conceptual ramework: 1) the causal model, 2) thelevels o units which will be evaluated, and 3) the types o impact that are to be assessed.8LIF and its MFIpartners in the assessment should think about these elements prior to the hiring and arrival o the researchfirm.

Causal Models (Theory of Change of Microfinance Interventions)MFIs were asked to submit their causal models to the consultant or LIF or review. Te purpose o thisexercise is twoold: 1) to encourage them to think about how their microfinance interventions will achievethe impact that they expect, and 2) to ormulate the hypotheses (research questions). Researchers who haveevaluated the impact o MF programs ofen comment that the causal models are not well thought out. Bythinking through the causal models, microfinance practitioners may find that there are missing elementsthat need to be incorporated in their activities in order to achieve the impact they want. Tese missing ele-ments might include financial literacy programs, womens empowerment activities, legal counseling, healtheducation and other themes. Qualitative tools, such as ocus group discussions with clients and/or loanofficers, can be useul to clariy the causal model. I MFIs are not able to provide their causal models or this7 Te idea o having the MFIs own staff assist in the implementation o the IA was discussed. However, given the level origor that LIF would like to see, and the credibility that LIF would like to establish with this evaluation, it is best that an inde-pendent research firm be hired, and that MFI staff only be involved to the extent needed to plan and coordinate activities.8 Hulme, David. Impact Assessment Methodologies or Micronance: Teory, Experience and Better Practice in WorldDevelopment. Vol. 28, No. 1, pp. 7998, 2000.


13/36


Page - 13

exercise, or i their causal models are incomplete, LIF might want to consider training in qualitative toolsor MFIs or this purpose. By understanding their causal models better, MFIs can create better impact ontheir clients.9

The Levels of Units to be Evaluated

Te second task is to determine the levels o unit to be evaluated. Tis will also depend on the causal model.Do MFIs expect their impact to be on the individual, on the household, on the business run by the loan re-cipient, on the children, on the group (i a group model is used), or on the community as a whole? I MFIsare not sure about this, using qualitative tools beore the impact assessment can help clariy this. More thanone unit level can o course be evaluated, but knowing beorehand where the impact is likely to occur willimprove the likelihood that impact can be measured. Te consultant recommends that the units be individ-uals, households, and businesses, so as to determine the impact on each o these.

The Types of Impact to be EvaluatedTe type o impact to be evaluated also depends on the causal model. Tis theme was discussed in the

stakeholder debriefing. MFIs mentioned the ollowing possibilities, although not all MFIs chose the sameoutcomes: Poverty alleviation Financial inclusion HH Wealth building Business Growth, Diversication Womens Empowerment Health outcomes Education outcomes

Democracy outcomes

While there are many impact assessments that evaluate all o these issues, research experts recommend thatassessments ocus on specific impacts that are most important or MFIs to know about. It is also importantthat the impact assessment be used to validate LIFs own lograme.

Te assessment will give more useul results i this it is ocused. Also, it is important to understand the time-line o impacts. I a study lasts two years, and the impact takes three years to occur, it will not be measurablein the study time period.10

It is also important to define exactly what is meant by impact and which specific outcomes are being meas-

ured. For example, i an MFI expects an impact on childrens education, it needs to speciy which children(boys/girls, ages, in-school/out-o-school), what impact/outcome (better attendance, better grades, etc.),and so on. Tis is necessary so that the survey tool can be correctly designed.

Example of a Causal Model

PAC/UNDP has described its causal model this way:

Te theory o change or the program suggests [that] loans to women will allow them to establish and buildup their own small businesses. Te women may then use the profits to reinvest in their business or set up

and operate additional businesses. Te businesses provide an income stream separate rom the womens hus-9 Tere are online tutorials or this purpose, or example, Te Community ool Box.10 A recent impact assessment in Mali lasted or three years. See: Final Impact Evaluation o the Saving or Change ProgramIn Mali, 2009 2012, by the University o Arizona Bureau o Applied Research in Anthropology and Innovations or Poverty Ac-tion.
http://ctb.ku.edu/en/tablecontents/sub_section_main_1877.aspxhttp://www.oxfamamerica.org/issues/community-finance/files/final-impact-evaluation-saving-for-changehttp://www.oxfamamerica.org/issues/community-finance/files/final-impact-evaluation-saving-for-changehttp://ctb.ku.edu/en/tablecontents/sub_section_main_1877.aspx


14/36


Page - 14

bands, resulting in higher household income. Larger incomes and the availability o loan products to meet avariety o needs allow client households to take advantage o opportunities as they arise and to better meetchallenges to the amily budget such as health emergencies.

Using this causal model as an example, it is clear that:

1. PAC/UNDP could confirm (and may already have confirmed) the validity o its causal model by usingocus groups. During these ocus groups, PAC staff might ask: i) What are you doing with the loan(consumption vs business investment)? What do you need to be able to invest more in the business andless in consumption? ii) I you invested in a business, is it your business alone or is it a joint business withyour husband? iii) Is your business generating more profit due to the investment? What else do you needto improve your business profits? iv) How do you spend your profits? And so on.

2. Impact could be measured at the level o the individual (women), at the business level, and at the house-hold level. Since PAC/UNDPs model does not include group or community level impacts, there is no

point in measuring at that level.

3. PAC/UNDP has defined the type o impacts that it expects:

Business Establishment, Growth, Protability, Diversication Health outcomes Other (unspecied) well-being outcomes at the household level

Having this inormation will thereore help PAC/UNDP elaborate its hypotheses to be tested, know whichlevels to evaluate, and design a survey with the indicators to measure the expected outcomes/impact.

For the purposes o choosing the indicators or the proposed IA, a causal model or the IA is suggestedbelow. Te consultant recommends that this causal model be reviewed with the MFIs or their input. Tiscausal model assumes that:

the impact of loans (rather than savings or other services) is of interest to LIFT, the majority of LIFTs partners provide loans for small income-generating activities (rather than for

agricultural investment), the majority of borrowers are female.

Under these assumptions, the causal model is:

Loans are provided to clients in groups. e group provides a structure that increases the likelihood of good loan repayment. Clients invest the majority of the loan in their existing income-generating activity to increase inven-

tory is increase in inventory leads to greater sales e greater sales lead to greater prots e majority of the prots are not reinvested in the business; rather, they are used to increase the

households standard o living (increase in ood consumption, in education and health expenditures,and in assets)

e contribution of additional income to the household raises womens status in the household


15/36


Page - 15

9. Hypotheses

LIF and the MFIs that wish to participate in this IA should think about the possible hypotheses that couldbe tested. Ideally, these hypotheses are based on their causal models. It is possible that most o the MFIs willuse more or less the same causal models, but this will be verified once the causal models have been reviewed

by the research firm hired to do the study.

Some hypotheses that are currently being tested or have been tested in microfinance are ound below:

Does the group joint liability reduce impact without improving repayment rates? Who determines the use of the loan, based on what criteria? What percentage of a rst loan is used for consumption vs investment? Subsequent loans? Does the provision of small business management training (the GRET Chin model) improve busi-

ness income? Do grants to VSLAs increase household income for group members?

Is there a greater impact on household income when women borrow? Does having a loan increase or reduce stress on the household? On the borrower? What changes to the loan terms would reduce farmers vulnerability to market prices? Does access to microloans increase employment, and if so, what kind and for whom?

For LIF and its MFI partners (current and uture), some o the ollowing hypotheses could be tested:

MFIs are reaching the poor and vulnerable (a LIF objective). Access to microfinance has a positive effect on economic activity and household poverty level, which

increases with the length o participation in a microfinance program. Use o microcredit is reducing household vulnerability to economic shocks (a LIF objective). Providing microcredit to women increases their status in the household. Te group structure increases repayment without decreasing impact on economic activity and

household poverty level. Te timing o the loans (date o disbursement) has no effect on impact.

10. Challenges11

Tere are a number o challenges that will need to be considered and possibly addressed by the research firmdoing the IA. Tese challenges were mentioned during the stakeholder meeting in May.

a. Survey FatigueTere are many surveys going on in Myanmar at this time or in the recent past, as donors (and the privatesector) attempt to gather inormation or their programs and products. Tis may result in people beingsurveyed more than once, and developing a set o responses to questions.12People who have been surveyedseveral times may not respond accurately or may decline to be interviewed. Te research firm hired by LIFor IA will need to be aware o what surveys have been done, and where, so as to avoid survey atigue byrespondents.

11 In Effect o Microfinance Operations on Poor Rural Households and the Status o Women (SS: REG 2007-19. Spe-cial Evaluation Study. ADB publication. September 2007), Appendix 21 has some interesting lessons learned about conductingan impact evaluation.12 Mayoux, Linda. Impact Assessment o Microfinance: owards Sustainable Learning Process. 2001. She notes: Teseproblems are obviously not confined to microfinance but are particularly acute in large-scale quantitative surveys.


16/36


Page - 16

b. Well-Matched Control VillagesAs noted above, many surveys have occurred, are occurring, or are planned. Many NGOs and MFIs areworking in the same zones, and this overlap will be exacerbated as they expand. A sufficient number omatching control villages will be needed or the control group. Tese villages must be careully matchedto the treatment villages so that biases are reduced or eliminated. I a sufficient number o well-matched

control villages cannot be identified, the study may need to change to a quasi-experimental design (no ran-domization).13

c. ContaminationTese control villages will need to be maintained as such during the time period o the study (perhaps 2years). During that time, it will be important that MFIs, whether participating in the study or not, rerainrom working in these villages. While the effects o other financial actors (MADB and inormal providers)can be statistically eliminated or reduced in the data analysis, this will be more difficult i actors with similarmethodologies to the study MFIs start operating in the control villages.

d. Lack of compliance with the methodologyAt the same time, the loan methodology must be rigidly applied during study period. Any variations in themethodology being studied may invalidate the research findings. MFIs whose programs are being evaluatedmay want to appoint a compliance monitor or this purpose.

e. SpilloverTis is an effect where the impact (good or bad) o a program is measured in a control village even whenthe program is not implemented there. Tis can be mitigated by locating control villages ar enough romtreatment villages.

f. Lack of clarity in the research question(s)I the research questions are too broad, i the hypotheses to be tested are not clear, i the hypotheses to betested are too numerous, then the impact will not be measured correctly. Experienced researchers recom-mend no more than 3 hypotheses to be tested.14In addition, a researcher always starts with the null hypoth-esis, that is, assuming that the impact is zero, and then designing a tool to prove or disprove this. 15

g. Quality of tools (surveys and qualitative tools)I the survey questions do not accurately reflect the causal model (level o impact expected, type o impactexpected, who it is impacting) and the hypotheses to be tested, then it will be difficult to accurately measurethe impact o the microfinance activities.

h. Political interferencePolitical intererence can affect the quality o results rom an IA. Te Microfinance Market Outlook (2013)notes that Te greatest threat to microfinance on all continents is intererence by politicians. 16In the caseo an impact assessment, political intererence may occur when government representatives pressure MFIsto provide services in control villages, or to change the MF model that is being tested. o avoid this, gov-ernment representatives should be inormed about the IA rom the beginning o the process, in order to gettheir buy-in.

13 Comparison villages would be established post-treatment.14 Interview with Dr. Geetha Nagarajan, Senior Evaluation Specialist, Development & raining Services, Inc. 9 May 2013.15 Developing a Research Question. PP prepared by IFMIR Center or Microfinance. Partially adapted rom rochim,William. Te Research Methods Knowledge Base. 2nd Edition. 2006.16 Microfinance Market Outlook 2013. responsAbility publication.
http://www.centre-for-microfinance.org/about-us/organization-profile/http://www.socialresearchmethods.net/kb/http://www.responsability.com/domains/responsability_ch/data/free_docs/rA_Microfinance_Market_Outlook_2013_EN.pdfhttp://www.responsability.com/domains/responsability_ch/data/free_docs/rA_Microfinance_Market_Outlook_2013_EN.pdfhttp://www.socialresearchmethods.net/kb/http://www.centre-for-microfinance.org/about-us/organization-profile/


17/36


Page - 17

11. Budget/Cost

LIF asked or a budget or the impact assessment. Ultimately, the cost will depend on many actors thatcannot be determined at this time. However, by making some assumptions, and rom interviewing surveyfirms, it is possible to give a range within which the IA might be done.

Te assumptions are the ollowing: e sample size will be determined by the research team, but will not be larger than the LIFT base-

line survey report o around 4,000 people; e geographic area will be limited; not all departments will be surveyed; e IA will be done in the dry season, in other words, access to sites will be easier than in the wet

season; A competent, experienced rm will be hired, and it may be international or national; e rm will do all the analysis of data; e IA will be mixed methods, including quantitative (survey) tools and qualitative (focus groups)

tools; ere will be a control group and a treatment group, randomized; ere will be 2 data collection rounds, a baseline and a nal.17

o ensure the best evaluation or value possible, it would be advisable to ollow the process used by the In-ternational Initiative or Impact evaluation (3IE) described here18:

A ToR for a preparation grant (PG) is disseminated. e PG will cover costs associated with activities such as travel to meet with LIFT, preliminary

checking o administrative /secondary data, site visits, researcher time, etc. necessary to develop animpact evaluation design (around $20,000)

As part of the collaboration with LIFT, the research team is also expected to provide one or moreworkshops to LIF officials and other stakeholders in order to finalize evaluation questions.

At the end of the preparation grant period, the PG grantee will submit a proposed design including aproposed budget. Te proposed design (s) is due within three months o signing the PG grant agree-ment.

e proposed design for the evaluation is reviewed and scored by a team of reviewers. LIFT may provide comments and request a resubmission of the proposal if the proposed design does

not meet quality criteria. If the proposed design is approved, LIFT will award the research team a grant to conduct the impact

evaluation under its standard terms and conditions.

Tis process should help avoid the ollowing scenario:

Another pragmatic step would be to invest more in the recruitment and management o evaluation con-sultants. Spending time and effort in developing clear, realistic terms o reerence (oRs) is critical. Ofen,the questions posed to evaluators, e.g. the extent the project reduced poverty or empowered women, are notonly too many and too vague but also typically impossible to credibly answer, particularly given prevailingbudget, time, and/or data constraints. Furthermore, a lack o technical oversight and management ofenleads to a poor application o methodological rigour. Nevertheless, the evaluators final findings are usuallyaccepted, by both ourselves and our donors, with minimal scrutiny (Nelson 2008). Tese issues call or the

technical involvement o capable evaluation staff rom our organisations in overseeing external evaluations,

17 Note that there is debate among experts on whether a baseline study is needed i the assessment is to be randomized.Tis issue should be explored with the research firm. Eliminating the baseline will reduce the cost.18 For document examples, please see Te International Initiative or Impact Assessments website, here.
http://www.3ieimpact.org/en/engage/3ie-professional-services/request-qualifications-cabi-kenya/http://www.3ieimpact.org/en/engage/3ie-professional-services/request-qualifications-cabi-kenya/


18/36


Page - 18

and we should not be naive as to how much time and effort this entails. 19

Other costs will include, but not be limited to, the ollowing: Initial stakeholder meetings to explain the process, get input into the design, and obtain buy-in; Review of existing surveys in Myanmar, for the purpose of data mining (if possible);

Design of the IA protocol and tools; production of a study protocol document; Training of data collectors and data entry personnel; Conducting the survey in the designated areas; Conducting the focus groups in the designated areas; Data entry Data analysis Report writing Debrieng of stakeholders Conducting second round of data collection in 2 years

Given these assumptions and costs, the total cost o the evaluation (including the preparation grant) over the2 or so years o treatment might be as little as $200,000 to up to $500,000. It is also important to note thatwage inflation is happening now in Myanmar and local costs will be going up.

12. Key Indicators for Inclusion in MFI MIS

Te LIF oR or this assignment included a request or key indicators or impact assessment that mightbe included in the MIS o LIF MFIs. MFIs will not be able to capture sophisticated data such as householdincome or poverty levels. It is better to use simpler indicators or MFI.

Also, since an MFI would not be able to measure impact (as defined above, in other words, randomizedand with a control group), these indicators would instead be monitoring indicators. Tey would monitor thechanges o the indicators, without being able to clearly attribute these changes to the microfinance activities.Monitoring o these indicators can be done through the loan application, the savings account application,by random sampling and surveying o clients, by ocus groups, and so on. Tese indicators should be moni-tored and analyzed yearly. More requent monitoring increases costs without showing changes.

In any case, a list o practical and cost-effective monitoring indicators that MFIs could choose rom includes:

19 Hughes, Karl and Claire Hutchings. Can we obtain the required rigour without randomisation? Oxam GBs non-ex-perimental Global Perormance Framework. International Initiative or Impact Evaluation (3IE) Working Paper 13. August2011


19/36


Page - 19

Expected Impact MIS Monitoring Indicator (changes in)

Poverty alleviation/Standard o living amount and value of land procured by type of land house conditions including type of materials used for the

roo, walls, flooring, and pillars, source o lighting sources of potable water type of latrines amount and types of productive assets such as bullock

carts, pumps, boat, trees, sewing machines, generators, etc. number and type of livestock personal assets such as radios, televisions, bicycles, motor

cycles, mosquito nets nancial assets such as gold, cash, and savings amount of debt*

HH Wealth building

Food Security Perception of changes in household food supply from the

previous year. Perception of change in quantity and quality of food eaten Perception of change in diversity of food eaten Perception of number of hungry days

Business Growth, Diversification Number of new businesses opened size of business (inventory, clients, physical space) business equipment/assets number of paid employees number of businesses operated by clients

Womens Empowerment who decides the use of the loan

who decides the use of the prots perception of ability to participate in decision-making in

the household

Health outcomes Expenditures on health Feeling of health

Education outcomes Expenditures on education Number of school-age children in school full time

* aken rom PACs indicator table.

13. Survey templateA number o survey templates were reviewed or this assignment, including but not limited to:

UNDP Household Survey LIFT Baseline Survey PACT Baseline Survey Millennium Villages Project Survey Compartamos Mexico Survey (IPA study)

Te LIF survey instrument is comprehensive, and should be used or the base o the IA survey. wo mod-ules could be added rom the UNDP/PAC survey or the IA: Module H: Loan Purpose, Utilization andCompetitor Analysis or Most Recent Loan and Module K: Women Empowerment.20Te Compartamos

20 Appendices 1 and 2.
http://www.poverty-action.org/node/5121http://www.poverty-action.org/node/5121


20/36


Page - 20

survey21covers health, asset acquisition, control o assets, ood consumption, migration, changes in busi-ness, stress levels, response to economic shocks, and financial access, among other categories. LIF and itsMFIs should review the Compartamos survey to see i any o these indicators are relevant to their programs,and incorporate those in the IA survey.

14. Stakeholder Advisory Committee

LIF should consider orming a stakeholder advisory committee (SAC) to monitor and oversee the design,implementation, and results o the IA. Tis is an excellent learning opportunity or the sector. A SAC couldconsist o:

LIFT Programme Ocer Markets and Micronance BRAC representative 22

A government representative, perhaps from the Central Bank or the Myanmar Micronance Super-visory Enterprise (MMSE)23

UNDP MF Programme Ocer LIFT M&E Ocer An MF technical service provider (for example, EdM, CARD) A reputable national university (social research department) An MFI whose clients are not being surveyed

Te committee would be advisory, not decision-making. Te committee would be ormed afer the hiring othe research team. Te SAC should meet intensively during the design phase, with the research firm hired,and then semi-annually during the implementation stage. LIF will want to include some unds or manag-ing this committee in the IA budget.

Te tasks o the SAC would be: Participate in the design of the study; Ensure that the research team understands the complexities of the Myanmar context; Disseminate information about the study to the broader sector (for example, to the MF Working

Group); Help ensure compliance with the design of the study of MFIs directly involved; Help ensure that no contamination of control villages occurs; Other tasks as recommended by LIFT and/or the research team;

15. Next StepsTe operational manual or the impact assessment is ound in Annex E. It includes a step-by-step plan orpreparing or and implementing the IA. Te workplan

a. Identify causal modelsTe next step or LIF is to work with MFIs to identiy their causal models.24Te more clarity that MFIshave in their causal models, the better will be the results o the evaluation. As noted in an USAID publica-tion on evaluations:

21 Innovations or Poverty Action website, Posting Hypotheses or an Impact Study o Compartamos.22 BRAC has significant experience in impact evaluations.23 In any case, this person should be sufficiently senior to be able to help resolve issues with local authorities during theevaluation.24 An attempt was made to do this during this assignment, but was not successul.
http://www.poverty-action.org/node/5121http://www.poverty-action.org/node/5121


21/36


Page - 21

Te central task in any evaluation assessment is to locate and veriy the causal model o the program to beevaluated. Tere are a range o possible situations that may ace the EA team. In the ideal case, a detailedcausal model already exists, having been developed during the program design phase. In this case, the evalu-ation team can proceed directly to the task o veriying the comprehensiveness and plausibility o the causalmodel:

Comprehensiveness: Te causal model should include all o the programs major activities, outputs,outcomes, and impacts presented in a way that indicates their relative importance.

Plausibility:Te causal chain depicted in the causal model should be logical in the sense that it isreasonable to anticipate that program activities and outputs could lead to the outcomes and impactsthat are indicated in the model.25

Te results o this task will also be validated by the research team.

b. Expansion plansTe second task is to obtain expansion plans rom the MFIs that are interested in the results o the evalua-

tion. Tis is necessary in order choose both the treatment and the control areas. MFIs should provide LIFwith their expansion plans. Tis inormation should be confidential, i.e. shared only between LIF and theresearch firm.

c. Design the studyPreparation Grant: A decision should be made whether to ollow the procedure outlined above or biddinga preparation grant (see Section 11). I this is seen as viable, then a oR would be developed or this phase.26Te result would be a project design.

Peer Review: Since there is so much debate and criticism around the subject o impact evaluations in mi-

crofinance, and there is so much riding on the results, and since the evaluations tend to be expensive, it maybe advisable to request some sort o peer review o the design by other experts in the field. CGAP is alreadytrying to lead the field in establishing parameters or IAs or MF, and it may be willing to use the LIF IA asa test case. Tis would likely lengthen the time needed or the IA, but would hopeully improve the designwhile serving as a model o peer review or the sector.27

d. ImplementationTe implementation o the study requires a series o steps, which are explained in the LIF Impact Assess-ment Operational Manual (Annex E).

e. Dissemination of ResultsDisseminate: Afer each phase o data collection and analysis, the results should be disseminated to theMyanmar MF sector, in a written publication and through a workshop/seminar. Tis ensures that all stake-holders benefit rom the research.

25 Planning or Cost Effective Evaluation with Evaluability Assessment. Impact Assessment Primer Series Publication # 6.USAID publication. 2008.26 See a template or a PG here. International Initiative or Impact evaluation. Request or Qualifications: Impact evalua-tion o CABIs Plantwise programme in Kenya.27 Innovations in Poverty Action has a blog that it uses to solicit outsider inormation on its hypotheses, so the idea is notnew.
http://pdf.usaid.gov/pdf_docs/PNADN200.pdfhttp://www.3ieimpact.org/media/filer/2013/05/14/3ie_rfq_cabi_.pdfhttp://www.3ieimpact.org/media/filer/2013/05/14/3ie_rfq_cabi_.pdfhttp://pdf.usaid.gov/pdf_docs/PNADN200.pdf


22/36


Page - 22

Appendix 1

Outcome/ Impact Assessment for Microfinance

Project 2011UNITED NATIONS DEVELOPMENT PROGRAMME/PACT MYANMAR

(HOUSEHOLD QUESTIONNAIRE)

Module H: Loan Purpose, Utilization and Competitor Analysis for Most Recent Loan

Compet-

itor

Loan

term or

duration

(months)

Amount

(Ks, 000)

Loan Purpose Loan utilization Interest

rate

%

Type o

collateral

Moneylender1

|_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_|

Agri.....1Livestock.2Fisheries/Agri.3Non-armers...4Health.5Other what.6______________________________

Medical care1Buying inputs. 2Emergencies issues..3Consumption..4Funeral5rading....6Pig breeding....7Other whtat8.

Labor..1Gold...2Crop...3No collateral..4Farm Land.5Other what6

Input-Provid-er2

|_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_|

|_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_|

Agri.....1Livestock.2Fisheries/Agri.3Non-armers...4

Health.5Other what.6______________________________

Medical care1Buying inputs. 2Emergencies issues..3Consumption..4

Funeral5rading....6Pig breeding....7Other whtat8.

Labor..1Gold...2Crop...3No collateral..4

Farm Land.5Other what6

Pawnshop.3

|_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_|

|_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_|

Agri.....1Livestock.2Fisheries/Agri.3Non-armers...4Health.5

Other what.6______________________________

Medical care1Buying inputs. 2Emergencies issues..3Consumption..4Funeral5

rading....6Pig breeding....7Other whtat8.

Labor..1Gold...2Crop...3No collateral..4Farm Land.5

Other what6

FriendRela-tives4

|_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_|

|_|,| __|__|_||_|,| __|__|_||_|,| __|__|_|

Agri.....1Livestock.2Fisheries/Agri.3Non-armers...4Health.5Other what.6

______________________________

Medical care1Buying inputs. 2Emergencies issues..3Consumption..4Funeral5rading....6

Pig breeding....7Other whtat8.



23/36


Page - 23

Compet-

itor

Loan

term or

duration

(months)

Amount

(Ks, 000)

Loan Purpose Loan utilization Interest

rate

%

Type o

collateral

Other(What)..5(FirstResponse)

|_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_|




Other(What)..6(SecondResponse)

|_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_||_|,| __|__|_|





24/36


Page - 24

Appendix 2Module K:Women Empowerment

Question Response categories(Now) Code

Women Empowerment on household decision making

k1 Decision on marriage o children Follow always.... 1Consult with husband and lose... 2Decide both.. 3Consult with husband and win... 4Initiate new idea, win her idea. 5DK/NR.. 9

|__||__||__||__||__||__|

k2 Decision on sending children to school Follow always.... 1Consult with husband and lose... 2Decide both.. 3Consult with husband and win... 4

Initiate new idea, win her idea. 5DK/NR.. 9

|__||__||__||__|

|__||__|

k3 Decision on Asset purchase/input purchase Follow always.... 1Consult with husband and lose... 2Decide both.. 3Consult with husband and win... 4Initiate new idea, win her idea. 5DK/NR.. 9

|__||__||__||__||__||__|

k4 Decision on selling o crops/selling price ingrocery shop/

Follow always.... 1Consult with husband and lose... 2

Decide both.. 3Consult with husband and win... 4Initiate new idea, win her idea. 5DK/NR.. 9

|__||__|

|__||__||__||__|

k5 Decision on Donation Follow always.... 1Consult with husband and lose... 2Decide both.. 3Consult with husband and win... 4Initiate new idea, win her idea. 5DK/NR.. 9

|__||__||__||__||__||__|

k6 Decision on Daily Expanse Follow always.... 1Consult with husband and lose... 2Decide both.. 3Consult with husband and win... 4Initiate new idea, win her idea. 5DK/NR.. 9

|__||__||__||__||__||__|

Participation in community on how ofen you participate

k7 Are you a MEMBER o any CBO (otherthan the microfinance)?

N0..0Yes..1

|__| k10|__|


25/36


Page - 25

Question Response categories(Now) Code

k8 Please NAME o all CBOs you are a mem-ber o and your participation in them.

Name # o participation in a month

|__|

|__|

|__||__|

k9 Are YOU consulted in Village level deci-sion-making process?

Never1Rarely2Sometime...........................3Ofen4DK/NR9

|__||__||__||__||__|

k10 Is the opinion o WOMEN GENERALLYtaken into consideration in overall decisionmaking in the Village?

Never1Rarely2Sometime...........................3

Ofen4DK/NR9

|__||__||__|

|__||__|


26/36


Page - 26

Annex ATERMS OF REFERENCE

(Individual Contract Agreement)

itle: Microfinance Impact Assessment SpecialistProject: Livelihoods and Food Security rust Fund (LIF)Duty station: Home BasedSection/Unit: LIFContract/Level: International Individual Contractor Agreement IICA 3Duration: 30 working days within six (6) weeks starting rom 1 May 2013Supervisor: Programme Officer - Markets and Microfinance

1. General Background of Project/AssignmentUNOPS is the Fund Manager or the Livelihoods and Food Security rust Fund (LIF) in Myanmar. LIF isa multi-donor und or seven years (2010 2016) to address ood insecurity and income poverty in Myan-

mar. Te Donor Consortium o LIF comprises Australia, Denmark, the European Community, France, theNetherlands, New Zealand, Sweden, Switzerland, the United Kingdom and the United States.

Te overall objective o LIF is to contribute resources to a livelihoods and ood security programme withthe aim o making progress towards the achievement o Millennium Development Goal 128(the eradicationo extreme poverty and hunger) in Myanmar. Working through a trust und modality, LIFs purpose is tosustainably increase ood availability and incomes o 2 million target beneficiaries.

Tis is to be achieved through delivering the ollowing programme outputs:1. Increased agricultural production and incomes supported through improved production and post-

harvest technologies, improved access to inputs and markets.2. argeted households supported in nonagricultural livelihood activities and/or trained in livelihood

skills or employment.3. Effective social protection measures supported that increase the incomes, enhance the livelihood

opportunities or protect the livelihoods assets o chronically poor households.4. Sustainable natural resource management and environmental rehabilitation supported to protect

local livelihoods.5. Capacity o civil society strengthened to support and promote ood and livelihoods security or the

poor.6. Monitoring and evaluation evidence and commissioned studies used to inorm programme and

policy development.

And the ollowing management outputs:7. Funds allocated in line with Fund Board policies and are accounted or in a transparent manner.8. Fund flow and partner perormance monitored and evaluated.

LIF is implemented through a variety o local implementing partners (IPs) who were successul in submit-ting proposals that supported the LIF purpose in the areas targeted.

LIF has long recognized the importance o increasing access to financial services as a means to achieveits outputs or both agriculture and non-agriculture livelihoods. Recently, LIFs baseline study ound that,

whilst 83% o households in the our agro-ecological zones took a loan in the prior 12 months, only 16%were able to do so rom a low interest financial services provider (FSP).

28 Reduce by hal the proportion o people living on less than a dollar a day; achieve ull and productive employment anddecent work or all, including women and young people; reduce by hal the proportion o people who suffer rom hunger.


27/36


Page - 27

LIF has unded a number o Microfinance Institutions (MFIs) in Myanmar. However, only with the pass-ing o the new Microfinance Law in November 2011 did it become possible or LIF to significantly scale upits assistance to the provision o microfinance services as part o a broader strategy o financial inclusion. InMarch 2012, the LIF Fund Board resolved to open a Microfinance (renamed Financial Inclusion) Win-dow (FIW), now constituted with a total amount o US$30m.

At the same time, there is a growing need to demonstrate that financial inclusion brings real benefits to thepoor in terms o increased income, better housing, clothing, ood and education or their children. LIFwished to take the lead in this area in Myanmar, or example though upcoming projects or working withsome o its IPs.

However, there are several ways to conduct impact assessments, e.g. through experimental research, ran-domised control trials (RCs), quasi-experiments or qualitative research. Generally, RCs (ideally accom-panied by qualitative research) are considered to be the most rigorous approach to impact evaluation, butthey come with issues regarding lead times and cost.

2. Purpose and Scope of AssignmentTe overall purpose o the consultancy is to develop a microfinance impact assessment system that is basedon the best practices available but customized to the particular needs and current standard operating prac-tices o Myanmar.

Under the direct supervision o the LIF Programme Officer - Markets and Microfinance, the Impact As-sessment Specialist will perorm the ollowing duties:

a) Consult with the LIF Fund Managers Office (FMO) and LIF unded MFIs/IPs about their objec-tives and priorities with regard to improved impact assessment or microfinance projects.

b) Conduct a desk-based review and synthesis o existing microfinance evaluation and impact assess-ment approaches in Myanmar, including those conducted by LIF unded MFIs/IPs.

c) Identification o impact indicators complying with microfinance industry impact assessmentbest-practice and o specific relevance to Myanmar.

d) Analyse the advantages and disadvantages o different impact assessment systems, including issueso cost, time to achieve results and robust outputs, and advising on the most appropriate.

e) Propose those LIF unded MFIs/IPs/projects, and any others o relevance, where the impact assess-ment system could be implemented.

) Facilitate workshop amongst LIF IPs and others to discuss preliminary findings and seek interestin participating.

g) Develop a potential impact assessment system, containing, but not limited to, the ollowing ele-ments: Impact assessment framework, including key indicators for inclusion in MFI management

inormation system database as appropriate and required; Baseline survey template, if any; Operational manual for impact assessment system including data collection and manage-

ment strategies; Outline workplan for piloting and roll-out of LIFTs impact assessment system; A committee of stakeholders for overseeing assessments and results; and A budget for the potential impact assessment system, assuming that LIFT will fund the entire

system (even though cost sharing with other stakeholders may be possible).

Te consultant is responsible to abide by security policies, administrative instructions, plans and procedureso the UN Security Management System and that o UNOPS.


28/36


Page - 28

3. Methodology and DeliverablesTe ollowing are the key activities to be carried out by the consultant (and the impact assessment systemdevelopment will be conducted in a consultative, participatory approach with all relevant stakeholders): Desk study of relevant documentation including materials provided by FMO and publicly available

existing microfinance evaluation and impact assessment approaches in Myanmar and countries with

best practice microfinance models; Brieng and debrieng meetings with the FMO oce in Yangon as agreed throughout the project; Consultations with those LIFT donors in Yangon (Australia, France, Switzerland, UK, US) and in

Bangkok (Denmark, EU, Netherlands, New Zealand, Sweden) who indicate interest to meet; Interviews (either face to face or via telephone/Skype etc) with LIFT funded MFIs/IPs and other

projects relevant to the consultancy as agreed with FMO; A dra ndings paper, for discussion with FMO, along with proposed outline of the nal report, to

be agreed with FMO; and A nal report, in both electronic form and in hard copy with any databases/templates from analyti-

cal and developmental work

4. Monitoring and Progress ControlsTe consultant: is expected to submit to the FMO a detailed work plan before traveling to Yangon; is expected to submit to the FMO the dra ndings paper and a nal report that complies with the

agreed ormat; and will be monitored against the agreed work plan and accomplishments will be assessed for quality and

timeliness by the FMO.

5. Timing

e total billable days for this assignment are 30 days. 15 days are allocated for eld work in Myanmar and, where applicable, Bangkok (including travel

days). e eld work will take place between 1 May and 30 June 2013. 3 days of preparation and research and 12 days of report writing in the consultants home country are

included.

6. Qualifications and Experiencea. Education

A Masters degree in social sciences, business management, development economics, rural

development, agricultural economics or related field. A Bachelor degree in social sciences, business management, development economics, ruraldevelopment, agricultural economics or related field in combination with 10 years qualiyingexperience may be accepted in lieu o a Masters degree.

b. Work Experience A minimum of 8 years professional experience in a eld related to rural development, liveli-

hoods, planning or project management. At least 4 years of this eld experience should be at a professional-level in micronance. Proven track record of high quality project evaluation/baseline surveys work. High degree of up-to-date experience in designing impact assessment systems.

Excellent communication skills, both orally and in writing, in English. Signicant professional experience in South-east Asia with professional experience in Myan-

mar an asset.


29/36


Page - 29

c. Key Competencies Professionalism Ability to conduct independent research and analysis, identify issues, ana-

lyze options and recommend solutions. Ability to work systematically, accurately and underpressure.

Planning and organizing - Ability to establish priorities and to plan, coordinate and monitor

own work plan to meet the deadlines and those under his/her supervision. Result-oriented - Ability to focus on the result for the clients and respond positively to feed-

back. Client orientation - Ability to identify clients needs and appropriate solutions; ability to es-

tablish and maintain productive partnerships with clients. Communication - Proven ability to write in a clear and concise manner and to communicate

effectively orally. Demonstrated ability to develop and maintain effective work relationshipwith procurement counterparts and substantive offices. Ability to communicate technicalprocurement matters in a simple and clear manner to individuals not well versed in the intri-cacies o procurement.

Teamwork - Strong interpersonal skills and; ability to establish and maintain eective work-ing relations with people in a multi-cultural, multi-ethnic environment with sensitivity andrespect or diversity, and with high level stakeholders.

Self-reliance: Ability to act independently with a minimum of supervision. Technological awareness - Excellent computer skills and ability to use soware tools to pres-

ent data clearly and concisely.

Project Authority (Name/itle)Andrew Kirkwood LIF Fund Director

Contract Holder (Name/itle)

Signature Date Signature Date


30/36


Page - 30

Annex B

LIST OF PEOPLE INTERVIEWED

Name Position/Organization

Mr. Barclay OBrien LIF Programme ManagerMr. Neal Youngquist General Manager

World Vision Myanmar Microfinance Program

Ms. Janis Sabetta Microfinance Advisor, Save Te Children

Ms. Yin Yin Myint, Dr. Pyae Phyo Lwin Directors, Myanmar Finance

Mr. Kyaw Tu M&E Officer, PAC

Mr. Kazi Faisal bin Seraj BRAC

Ms. Juvy V. Gacutan Myanmar Liaison Office ManagerCredit & Savings Specialist/ Area Manager

Center or Agriculture & Rural Development Inc.Mr. Alexandre Goutchkoff Entrepreneurs du mondeMyanmar Program Manager

Ms. Kerry Bruce, Mr. Jason Meikle (by phone) Director, Results and Measurement and CountryDirector, PAC

Mr. Kim Bunsocheat and Mr. Sambath Kheang General Manager and Company Secretary, ACLE-DA MFI Myanmar Co., Ltd.

Ms. Murielle Morisson Project ManagerGRE Chin State

Ms. Gill Pattison and Ms. Maria Fulwiler Acting GM and Associate, Proximity FinanceDr. Stuart Rutherord SaeSave, Financial Diaries

Mr. Heinz Willems UNDP Microfinance echnical Advisor

Mr. Kris Hendrickx LIF M&E Officer

Dr. Geetha Nagarajan Senior Evaluation Specialist | Development &raining Services, Inc.

Dr. Nathaniel Goldberg Innovations in Poverty Action

Ms. Nina Holle CGAP Clients and Product Manager

Mr. Htun Htun Oo and Mr. Lutha Kyaw Managing Director and Director, Decision Support

Service Co., Ltd. (survey firm)Ms. Suman Joseph Myanmar Survey Research


31/36


Page - 31

Annex COperational Manual

Tis manual describes the steps that LIF should ollow in order to obtain a high-quality impact evaluationo its MF program.

Step 1: Prepare RFP/TORs for the Preparation GrantAs discussed in the main document, it would be useul to use a mechanism like a preparation grant to beginthe IA. An RFP/OR should be prepared. Te PG will cover costs associated with activities such as travelto meet with LIF, preliminary checking o administrative /secondary data, site visits, researcher time, etc.necessary to develop an impact evaluation design (around $20,000). At the same time, LIF should preparethe OR or the actual implementation o the study, so that once the design is approved, the firm can goorward immediately with the study.

imeline: 1 month to write the OR

Responsible Party: LIF MF Program

Step 2: Disseminate the RFP/TOR for preparation grantTe RFP/OR should be disseminated to a wide range o qualified firms. Some firms that should be contact-ed or bidding include, but are not limited to29:

Development & Training Services, Inc. Innovations for Poverty Action/Abdul Latif Jameel Poverty Action Lab BRAC Research Myanmar Survey Research

CARD Decision Support Service Co, Myanmar Myanmar Development Partners Co University of Arizona, Bureau of Applied Research in Anthropology

imeline: 1 month or receiving bids rom interested partiesResponsible Party: LIF MF Program

Step 3: Hire research firmTe submissions are reviewed and a decision to hire is made.

imeline: 1 month or reviewing bids and hiring chosen firmResponsible Party: LIF MF Program

Step 4: Stakeholder advisory committee formed and meetsLIF organizes and convenes the stakeholder advisory committee (SAC). Once the research firm is hired, itwill meet periodically during the design preparation phase with the SAC to discuss hypotheses to be tested,sampling strategy, which MFIs will be involved in the study, proposed indicators and data collection toolsto be used, and so on.

imeline: Concurrent with the design preparation phase (Step 5)

Responsible Party: LIF MF Program (convenes); Research firm and SAC (meetings)

29 Te author o this document has no financial interest in any o these firms and makes no warranties about the qualityo their work.


32/36


Page - 32

Step 5: Research firm designs the IATe firm designs the IA in consultation with LIF and MFIs. Te design should include:- Hypotheses to be tested- Which MFIs will be involved in the study, and what their responsibilities will be- A sampling strategy (based on cluster randomization to reduce ethical concerns)

- ools or collecting data (quantitative and qualitative)- Define impact indicators- Decision on whether prior surveys can be mined or data- Evaluation team makeup, with roles and responsibilities- imeline or implementation- Budget- List o resource needs or carrying out the study (logistics)

Regarding the impact indicators, the LIF survey instrument is comprehensive, and should be used tomeasure the indicators. wo modules could be added rom the UNDP survey or the IA: Module H: Loan

Purpose, Utilization and Competitor Analysis or Most Recent Loan and Module K: Women Empower-ment. Te Compartamos survey covers health, asset acquisition, control o assets, ood consumption, mi-gration, changes in business, stress levels, response to economic shocks, and financial access, among othercategories. LIF and its MFIs should review the Compartamos survey to see i any o these indicators arerelevant to their programs, and incorporate those in the IA survey.

imeline: 2 monthsResponsible Party: Research firm

Step 6: Peer Review and Design ApprovalLIF should establish a small, inormal committee o peer reviewers, and send the design to them or com-

ments and suggestions. CGAP can help assemble this committee. Once the suggestions are in and the designmodified accordingly, LIF approves the design. Tis will ensure that the design o the study meets bestpractice standards.

imeline: 1.5 monthsResponsible Party: LIF MF Program, Peer reviewers, research firm

Step 7: LIFT hires the firm to implement the studyLIF hires the research firm that did the design.30

imeline: 1 monthResponsible Party: LIF MF Program

Step 8: Prepare to implement the studyTe research firm prepares to implement the study, ollowing these steps (which may be concurrent in somecases): Develop the IA protocol and tools; produce a study protocol document and distribute to MFIs and

LIF or comments and approval/buy-in Design survey questionnaire (see the recommendations in Step 5 for which indicators to include) Develop the qualitative methodology (focus groups and semi-structured interviews with key in-

ormants are recommended at a minimum); develop the interview questionnaires and ocus groupsguides Pre-test and pilot test the survey questionnaire and the qualitative tools, revise as necessary. Design the database for data entry of quantitative data30 I the firm ails to deliver an adequate design, then LIF may re-start the process and hire another firm.


33/36


Page - 33

Hire and train a data collection supervisor, data collectors and data entry personnel Hire and train focus group facilitators and interviewers Prepare eld work schedule; arrange the logistics (transportation, workspaces and meeting spaces,

data recording tools) Select participants and non-participants and villages using appropriate sampling methodology

Update the SAC periodically with progress reports Update the participating MFI(s) with progress reports

imeline: 2 monthsResponsible Party: Research Firm

Step 11: Implement the study Conduct baseline survey among the selected individuals/households (2 months) Conduct the focus groups and semi-structured interviews with the sample (concurrent) Data entry and cleaning (1 month)

Data analysis (1 month) Update the SAC periodically with progress reports Update the participating MFI(s) with progress reports


Step 11: Dissemination of ResultsTe research firm writes a report describing findings and recommendations rom the baseline survey. Tefirm then presents a summary to stakeholders in a seminar.


Step 12: Repeat the study (quantitative and qualitative elements)It is advisable to use the same research firm or the final study, which should occur 2 years later.

Conduct a follow-up survey among individuals/households surveyed in the baseline (2 months) Data entry (1 month)

imeline: 3 months

Responsible Party: Research Firm

Step 13: Dissemination of ResultsTe research firm writes a report describing findings and recommendations rom the baseline survey. Tefirm then presents a summary to stakeholders in a seminar.

First dra of impact evaluation report (2 months) Final dra of impact evaluation report (within 3 weeks of receiving comments on the rst dra) Stakeholder debrieng (1 day)


8/10/2019

microfinance-impact-assessment-designs (august 2013).pdf

Documents