financial market development (fmd) activity …
TRANSCRIPT
FMD Activity MEL Plan, December 2017 1
FINANCIAL MARKET DEVELOPMENT (FMD)
ACTIVITY MONITORING, EVALUATION,
AND LEARNING (MEL) PLAN - REVISED
DECEMBER 2017 This publication was produced for review by the United States Agency for
International Development. It was prepared by Management Systems International, A
Tetra Tech Company, and the Trust for Democratic Education and Accountability.
FMD ACTIVITY MONITORING,
EVALUATION, AND LEARNING PLAN
DRAFT REVISION – DECEMBER 2017
DISCLAIMER
The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency
for International Development or the United States Government.
FMD Activity MEL Plan, December 2017 1
ACRONYMNS
AR Activity Result (FMD level result)
AOR Agreement Officer’s Representative
CDNS Central Directorate of National Savings
COP Chief of Party
COR Contracting Officer's Representative
DCOP Deputy Chief of Party
DCA Development Credit Authority
DCM Debt Capital Management
DO Development Objective (USAID/Pakistan result)
DQA Data Quality Assessment
EBND Electronic Bond Trading Platform
EGA Economic Growth and Agriculture
EFW External Finance Wing
eWRS Electronic Warehouse Receipts
FMD Financial Market Development
GDP Gross Domestic Product
GOP Government of Pakistan
IMF International Monetary Fund
IP Implementing Partner
IR Intermediate Result (USAID/Pakistan result)
IST Indicator Summary Table
IT Information Technology
LOA Life of Activity
MIS Management Information System
MoF Ministry of Finance
M&E Monitoring and Evaluation
ME&L Monitoring, Evaluation & Learning
MEP Monitoring and Evaluation Plan
NBFI Non-Banking Financial Institution
NOC No Objection Certificate
NSS National Savings Scheme
FMD Activity MEL Plan, December 2017 2
OCAT Organizational Capacity Assessment Tool
PIRS Performance Indicator Reference Sheets
PMEX Pakistan Mercantile Exchange
PMP Performance Management Plan
PSX Pakistan Stock Exchange
PKR Pakistani Rupees
RF Results Framework
SBP State Bank of Pakistan
SECP Securities and Exchange Commission of Pakistan
SAR Sub Activity Result (FMD level result)
SIR Sub Intermediate Result (USAID/Pakistan result)
SME Small and Medium Enterprise
SOP Standard Operating Procedure
SPV Special Purpose Vehicle
TBD To Be Determined
USAID United States Agency for International Development
USD United States Dollars
USG United States Government
FMD Activity MEL Plan, December 2017 3
CONTENTS
ACRONYMNS ..................................................................................................................................................... 1
I. INTRODUCTION TO FINANCIAL MARKET DEVELOPMENT ACTIVITY (FMD)
MONITORING, EVALUATION, AND LEARNING (MEL) PLAN ............................................ 5
PURPOSE .............................................................................................................................................. 5
ACTIVITY INFORMATION AND CONTEXT ................................................................................ 6
THEORY OF CHANGE ....................................................................................................................... 8
II. ACTIVITY MEL PLAN ................................................................................................................................... 9
PERFORMANCE INDICATORS ....................................................................................................... 9
DATA QUALITY ASSURANCE AND DATA QUALITY ASSESSMENT
PROCEDURES ..................................................................................................................... 10
REPORTING OF DATA .................................................................................................................... 10
ROLES AND RESPONSIBILITIES AND DATA COLLECTION
METHODOLOGIES ............................................................................................................ 11
DATA COLLECTION METHODOLOGIES .................................................................................. 12
GENDER M&E ................................................................................................................................... 13
CALENDAR OF MEL EVENTS ....................................................................................................... 13
LEARNING .......................................................................................................................................... 13
ANNEX A: PERFORMANCE INDICATOR REFERENCE SHEETS (PIRS) .......................................... 14
ANNEX B: ASSURING DATA QUALITY ................................................................................................... 25
DATA QUALITY CHECKLIST AND RECOMMENDED PROCEDURES .............................. 25
List of Tables
Table 1: Project Summary ............................................................................................................................. 5
Table 2: Performance Indicator TrackingTable ..................................................................................... 9
Table 3: Performance Reports ................................................................................................................... 11
Table 4: Summary of FMD's indicator data colllection methodologies .................................... 12
Table 5: Calendar of MEL events .............................................................................................................. 13
List of Figures
FIGURE 1: FMD ACTIVITY REVISED RESULTS FRAMEWORK ........................................................... 7
FMD Activity MEL Plan, December 2017 4
FMD Activity MEL Plan, December 2017 5
ACTIVITY SUMMARY
Table 1 Summarizes basic information about the Financial Market Development Activity (FMDA)
TABLE 1: PROJECT SUMMARY
Title/Field Project/Activity Information
Contract/agreement numbers AID-391-C-15-00013
Contracting/agreement officer’s
representative (COR/AOR) Nadia Dawood
Start date March 01, 2016
Completion date February 28, 2019
Location Islamabad, Pakistan
Implementing partner(s) The Pragma Corporation
USAID/Pakistan Mission Strategic
Framework objectives addressed DO 2, IR2.2: Improved Business Enabling Environment
I. INTRODUCTION TO FINANCIAL MARKET
DEVELOPMENT ACTIVITY (FMD) MONITORING,
EVALUATION, AND LEARNING (MEL) PLAN
The goal of the three-year Financial Market Development Activity (hereinafter “FMD Activity,” “FMD,”
or the “Activity”) is to strengthen Pakistan’s debt financial markets, and contribute to the expansion of
financing for private sector investment and economic growth, through targeted policy reforms that helps
create a more open and free business enabling environment.
The FMD Activity seeks to target and effectively address critical factors that limit the deepening and
diversification of overall capital market financing, and that restrain the demand for innovative capital
market instruments in these areas. FMD will work with key public and private sector institutional actors,
under USAID guidance, to both prioritize and develop a consensus on how to effectively address these
key constraints, and support implementation of technical, training and outreach activities designed to
move critical capital market reforms forward.
Purpose
The purpose of this revised MEL plan is to describe how FMDA will monitor its activities:
• Presents the linkages between indicators and related USAID/Pakistan’s Mission Strategic
Framework (MSF) sub-intermediate results (IRs), IRs, and development objective (DO);
• Includes indicators to measure progress in achieving the expected activity results;
• Describes the processes that FMDA will use to conduct performance monitoring and collect,
manage, analyze, and report data;
• Defines the M&E unit’s staff structure and performance monitoring roles;
• Provides a timetable for performance reporting; and
• Describes the data quality control mechanisms.
FMD Activity MEL Plan, December 2017 6
Annexes A-C include Performance Indicator Reference Sheets (PIRS) and tools for conducting data
quality assessments (DQAs).
Activity Information and Context
Project Context
FMD is part of the Mission’s Economic Growth portfolio that is helping the Government of Pakistan
(GOP) implement key policy/regulatory and institutional reforms affecting trade, the financial sector, and
small- and medium- sized enterprise development. In particular, FMD directly contributes to
USAID/Pakistan Mission Development Objective Two (DO2): Improved Economic Status of Focus
Populations and Sectors. The activity will directly support DO2 Intermediate Result 2.2 Improved
Business Enabling Environment, and DO2 Sub-Intermediate Results 2.2.1 Improved Ability to
Develop and Implement Reform and 2.2.2 Strengthened Private Sector and Civil Society
Engagement in Policy-Making. For a graphic depiction of the DO 2 Results Framework, please see
Figure 1 below. The end goal of the FMD activity is a strengthened debt finance and capital market.
FMD Activity Description
The FMDA project started on March 2016, and the project will now continue through the revised end
date of February 28, 2019. FMDA contributes to USAID’s DO #2, IR2.2, “Improved Business Enabling
Environment.” The goal of the FMDA activities is to support policy/regulatory and institutional
interventions in the debt and market will help promote rationalization and diversification of debt product
offerings. Main components include:
• Strengthened Debt Finance and Capital Market
• Increased adoption of sound financial management practices by key financial institutions
The FMD project was de-scoped in August 2017. De-scoped components included SME financing enabling
environment reforms and support for the issuance and secondary market trading of domestic sovereign
debt. The original performance indicators and results statements for these de-scoped components are no
longer relevant and were removed. This revised MEL Plan serves to update the previously approved MEL
plan.
Activity Location and Institutional Context
Under this activity, FMD is working in the Islamabad Capital Territory, Khyber Pakhtunkhwa province,
Punjab province, and Sindh province.
The linkages between FMDA activities and the Project Goal, Purpose and Sub-Purposes are illustrated in
Figure 1.
FMD Activity MEL Plan, December 2017 7
FIGURE 1: FMD ACTIVITY REVISED RESULTS FRAMEWORK
MSF Goal: Increased stability, democracy and prosperity for the men and women of
Pakistan
Project Purpose - Intermediate Result 2.2: Improved Business Enabling Environment
Indicator 2.2a: Number of enabling environment policies analyzed, consulted on, drafted or revised,
approved, and implemented with USG assistance
Development Objective 2: Improved Economic Status of Focus Population
Project sub-purpose – IR 2.2.1: Improved
ability to develop and implement Reforms
of Policies, Laws and Regulations
Indicator 2.2.1a: Person hours of USG-supported
training completed in trade and investment
(Standard EG.2-1)
Indicator 2.2.1b: Number of persons receiving
USG supported training on policy development,
analysis, and implementation
Project sub-purpose – IR 2.2.2:
Strengthened Private Sector and Civil
Society Engagement in Policy-Making
Indicator 2.2.2a: Number of USG-assisted
organizations that participate in legislative
proceedings and/or engage in advocacy at all
levels
Indicator 2.2.2b: Number of events for
public/stakeholders consultation to enhance
sector governance or facilitate private sector
engagement in policy making as a result of USG
assistance
FMD Activity MEL Plan, December 2017 8
Theory of Change
The revised MELP incorporates guidance provided in a “De-Scoping Order” delivered to Pragma
Corporation in August 2017. In accordance with that De-Scoping Order, FMD indicators and scope of
MELP now cover the revised timeframe from September 2017 through February 2019.
As per its revised scope of work, FMD’s theory of change is aimed at effectively addressing critical
factors that are limiting the deepening and diversification, as well as the introduction of innovative capital
market financing solutions. FMD’s current approach is based on assisting key counterparts in their quest
for policy, regulatory, operational and technical reforms to (1) support the robust expansion of the
range of tradeable debt capital instruments; (2) improve the efficiency and cost-effectiveness with which
those instruments are traded; and (3) significantly enhance confidence in the transparency and integrity
of capital market trading and clearing/settlement operations.
At the same time, FMD’s impact through its scope of work is focused to enable mobilization of capital
for marginalized agricultural communities by introduction of international state of the art electronic
warehousing receipt system in Pakistan. FMD’s work with provincial governments for second tier debt
product issuance shall initiate the first ever provincial debt financing in Pakistan.
FMD Activity MEL Plan, December 2017 9
II. ACTIVITY MEL PLAN
Performance Indicators
The expected results of FMD’s activities are measured through performance indicators. The performance indicator tracking table below
illustrates how FMD will monitor its performance data. The table includes indicator baselines, annual targets, and target justifications.
TABLE 2: PERFORMANCE INDICATOR TRACKING TABLE
#
Indicator
Baseline
FY 2017
Target
FY 2018
Target
FY 2019, February
Target
LOA Target Justification
AR
1
2.2a: Number of enabling environment
policies analyzed, consulted on, drafted or
revised, approved, and implemented with
USG assistance (EGA, Modified F)
0 9 9 9 27 Assuming these policies include all policy level interventions
within the FMD approved workplan.
SAR
1.1
2.2.1a: Person-hours of USG-supported
training completed in trade and investment
(EGA, Standard F)
0
320
M: 256
F: 64
3840
M: 3072
F: 768
1920
M: 1536
F: 384
6080
Institutional and investor level capacity building initiatives of
FMD within the approved workplan Sub-targets for FY19
males/females will be established at the end of FY18.
2.2.1b: Number of persons receiving USG
supported training on policy development,
analysis, and implementation (EGA) 0
80
M: 64
F: 16
80
M: 64
F: 16
40
M: 32
F: 8
200
FMD’s capacity and consensus building outreach events with
key financial institutions. Sub-targets for FY19 males/females
will be established at the end of FY18.
SAR
1.2
2.2.2a: Number of USG-assisted
organizations that participate in legislative
proceedings and/or engage in advocacy at
all levels (EGA)
0 6 5 2
13 FMD’s ongoing technical assistance, as per the FMD workplan,
to SECP, PSX, PMEX, NCCPL & CDC.
2.2.2b: Number of events for public/
stakeholders’ consultation to enhance
sector governance or facilitate private
sector engagement in policy making
because of USG assistance (EGA)
0 19 8 3 30
For each of the legal/regulatory development
proposal/recommendation, FMD shall organize stakeholders’
feedback sessions followed by awareness sessions.
FMD Activity MEL Plan, December 2017 10
Data Quality Assurance and Data Quality Assessment Procedures
High quality data is the cornerstone of performance monitoring systems and evidence-based decision-
making, and FMD recognizes that a robust MEL system is required to provide evidence-based results to
implement the activities of partners and donors. USAID’s Automated Directives System (ADS) provides
the standards and criteria for data quality which include validity, integrity, precision, reliability, and
timeliness. Data quality assessments ensure that USAID Missions and implementing partners are aware
of the strengths and weaknesses of data, as determined by applying the ADS data quality criteria, as well
as the extent to which the data’s integrity can be relied upon to support management decisions.
FMD’s M&E manager develops the protocols and standard procedures to ensure that data is collected in
a consistent and technically sound manner, meets USAID’s data quality standards, and can be compared
over time. The FMD M&E Expert will be responsible for assessing the strengths and weaknesses of the
chosen indicators prior to submission of each annual report, and will use the updated USAID data
quality assessment checklist (including a follow-up action tracker) to conduct this internal assessment
(see Annex B). The FMD M&E Expert is also responsible for keeping a record of all issues related to
data quality, along with actions taken.
To manage performance data, all the required data will be collected on agreed indicators based on the
sources listed in the PIRS. The performance monitoring data will be quantitative as well as qualitative.
The following categories of data sources will be used:
• FMD maintained database on training participants and stakeholder engagement: FMD will be
responsible for designing a system for collecting training participant data and for making sure that
sign-in sheets are used at all FMD-supported trainings. FMD must also design and maintain a system
for documenting characteristics of all Activity stakeholders.
• International data and statistics: The FMD Activity will take into account international research
and statistics regarding Pakistan’s economic rankings, potential, and competitiveness in capital and
debt markets development. These will be used to provide contextual information regarding the
financial market and inform dialogue with other donors and development partners. Contextual data
may help the FMD team understand trends and pattern as they fine-tune strategies and
interventions. Additionally, the Activity’s ME&L Plan will provide insight to the possible contribution
of FMD interventions to the changes in data gathered through these sources.
Reporting of Data
In order to provide regular updates to USAID and key stakeholders of the FMD Activity, various types
of reports will be generated and shared internally and externally, as agreed upon with USAID in
advance. Reports will include:
• Activity Meeting Reports: Including quarterly progress sharing meeting reports (these reports will
be shared with the COP and the senior management to keep them informed of progress on the
Activity’s results and to enable them to determine program priority areas)
• Activity Reports/Plans:
o FMD Work Plan
o Monitoring reports and/or site visit reports of M&E Unit
o Quarterly Performance Monitoring Reports, which will include progress against each ME&L
Plan indicator
FMD Activity MEL Plan, December 2017 11
o Quarterly reporting, on the USAID Pakistan “Pak Info” platform, of the indicators linked to
the EGA results framework
o Annual Performance Monitoring Reports, which will include progress against each ME&L
Plan indicator
o End of Activity Report.
FMD will submit quarterly and annual reports to USAID, as indicated in table 3. FMD will organize and
update the quarterly performance data and review the data with the COR. At the end of the fiscal year,
FMD will submit an annual performance report that will include a compilation of the year’s actual results
versus targeted results for each indicator and an explanatory narrative. FMD will submit all reports to
the COR for approval.
TABLE 3: PERFORMANCE REPORTS
Performance Report Content
Quarterly
Quarterly accomplishments; lessons learned/challenges faced and solutions
proposed; planned activities for the next quarter; performance indicator
data versus targets; analytical narrative for results under each activity;
results achieved and means of verification for each result; and photos
Weekly Weekly updates on implementation of activities.
Annual Annualized information included in the quarterly performance reports.
Roles and Responsibilities and Data Collection Methodologies
The FMD chief of party (COP) has overall responsibility for assuring that the work of the M&E unit meets
the needs of the project and USAID reporting requirements. The M&E expert is responsible for all M&E
activities and reports directly to the COP.
Implementing the MEL Plan
The M&E expert is the primary person responsible for implementing the MEL plan and is also
responsible for:
• Overseeing and providing technical guidance with respect to MEL approaches, practices, and
tools;
• Meeting with the USAID/Pakistan COR on performance management; and
• Submitting performance data and reports to USAID/Pakistan.
Data Review Process
FMD has instituted a comprehensive data review process that assesses the validity and accuracy of the
data at successive levels. Reviewed and verified data is reported on PakInfo USAID.
Reporting
FMD gives data management special consideration to produce error-free and timely reporting. The M&E
expert is responsible for providing the verified data which is then consolidated into high-quality quarterly
performance reports and annual reports by FMD and submitted to USAID/Pakistan for review.
FMD Activity MEL Plan, December 2017 12
TABLE 4: SUMMARY OF FMD'S INDICATOR DATA COLLLECTION
METHODOLOGIES
INDICATOR DATA COLLECTION METHODOLOGY
2.2a: Number of enabling environment policies analyzed,
consulted on, drafted or revised, approved, and implemented
with USG assistance Collected by the FMD technical staff
2.2.1a: Person-hours of USG-supported training completed in
trade and investment FMD training data collection forms
2.2.1b: Number of persons receiving USG supported training
on policy development, analysis, and implementation FMD training data collection forms
2.2.2a: Number of USG-assisted organizations that participate
in legislative proceedings and/or engage in advocacy at all levels FMD data collection forms
2.2.2b: Number of events for public/ stakeholders’ consultation
to enhance sector governance or facilitate private sector
engagement in policy making as a result of USG assistance
FMD data collection forms
Data Collection Methodologies
Performance monitoring plays a central role throughout the project cycle. The M&E expert is responsible
for data collection.
Planning
Specific data collection tools to be used by FMD include: training attendance sheets, training participant
feedback questionnaires, and pre- and post-training assessments; and customized data collection tools for
other performance indicators. Most data collection instruments can be modified from tested tools to
reduce cost and biases, and increase reliability.
Data Analysis
Analysis of indicator performance data is critical for continuous learning and to determine whether targets
are being met. Data is disaggregated in accordance with USAID requirements for each indicator. Common
disaggregates are by geographical region, project theme, age group, and male/female (for all indicators that
measure number or percentage of citizens). The M&E expert reviews the quality of the data received
based on the five standards of data quality. The analysis also provides evidence-based guidance for
monitoring and taking corrective action when necessary.
Various tools will be used for the analysis of data. FMD will maintain an Excel database of stakeholders
and their characteristics, and training participants and their characteristics to facilitate analyses across
organizations and other disaggregation criteria. Every effort will be made to quantify data that can be
quantified. Beyond the standard use of charts, graphs and tables, where possible, the team will use creative
ways to present qualitative data to highlight patterns and relationships, such as mappings and network
diagrams.
FMD Activity MEL Plan, December 2017 13
Gender M&E
FMD MEL plan contains two performance indicators that are gender sensitive, and will be disaggregated
by sex. These indicators specifically target women’s participation in capacity building and skill development
trainings. All gender sensitive indicators will be analyzed to assess the different impact on men and women.
2.2.1a Person hours of USG-supported training completed in trade and investment (EG.2-1)
2.2.1b Number of persons receiving USG supported training on policy development, analysis, and
implementation
Calendar of MEL Events
FMD will submit to USAID/Pakistan quarterly and annual performance reports, and a final closeout plan
and report.
TABLE 5: CALENDAR OF MEL EVENTS
Performance Report Calendar Year Date Comments
Quarterly
July 31, 2016
January 31, 2017
April 30, 2017
July 31, 2017
January 31, 2018
April 30, 2018
July 31, 2018
The Quarterly Report due to
be submitted at the same time
as the Annual Report will be
combined with the Annual
Report.
Annual
October 31, 2016 (2 quarters only)
October 31, 2017
October 31, 2018
The report will aggregate
annual data on indicators and
other results along
with the 4th quarter report.
Final
No later than 90 days, following the
completion date of the agreement.
February 28, 2019
The final report will cover the
entire period of the USAID-FMD
agreement
Learning
FMD prides itself on being a committed learning organization that relies on performance data to make
evidence-based decisions on programming. Its performance monitoring system provides the platform to
facilitate learning and adaption throughout activity implementation. FMD envisages a monitoring,
evaluation and learning system that helps identify promising interventions in advance so that they
can effectively be implemented during the course of the activity, to generate approved anticipated
results. Having data available about how well a particular intervention or policy works will provide
useful information for formulating and justifying resource requests. This will facilitate judicious
allocation of resources to undertake interventions that will provide the strongest impact.
FMD Activity MEL Plan, December 2017 14
ANNEX A: PERFORMANCE INDICATOR REFERENCE
SHEETS (PIRS)
PERFORMANCE INDICATOR REFERENCE SHEET (PIRS)
Indicator: 2.2-a Number of enabling environment policies analyzed, consulted on, drafted or revised, approved, and implemented with USG assistance
Development Objective (DO): 2 Improved Economic Status of Focus Populations and Sectors
Intermediate Result (IR): 2.2 Improved Business Enabling Environment
Sub-Intermediate Result (Sub-IR): N/A
Relationship between the Sub-IR and IR or IR and DO: Enter the explanation of the linkage between the lowest level of result represented by the indicator, and the next level of result up; address the “so what?” question to move from outputs to outcomes, or outcomes to impact; explain in terms of the development hypotheses, do not simply restate the structure of the Results Framework.
An improved business enabling environment will allow enterprises to achieve their potential, thereby increasing incomes and employment, because such an improvement indicates that major legal, policy, regulatory, and administrative barriers that inhibit their performance are no longer in effect. The indicator measures the number of policies/ regulations/ administrative procedures in the various stages of progress towards an enhanced enabling environment for agriculture whose sub-elements are specific policy sectors. This indicator is easily aggregated upward from all operating units.
INDICATOR DESCRIPTION
Precise Definition(s): Enter the precise definition of the indicator so it can be operationalized; define all terms, elements, implied actions and calculations; [for example, “farmers using better production techniques” – define “better production” and “techniques”. Describe how this will be determined – e.g. Index, scale, standards]. For indicators that are percent or proportions explain how it will be calculated and what will serve as the numerator and denominator. If the indicator is cumulative, made up of stages or phases, or is a yes-no, please specify this and explain the stages/phases or how it is cumulative. If it is a Standard Program Structure (“F”) Indicator, use and if necessary, refine the standard definition.
The indicator counts the number of economic growth policies related to the institutional architecture for improved policy formulation, the enabling environment for private sector investment, agricultural trade, agriculture input provision, land and natural resource management, or food and nutrition that have completed one or several of the following 5 steps or processes: 1. Underwent analysis (review of existing policy and/or proposal of new policy); 2. Underwent public debate and/or consultation with stakeholders on the proposed new or revised policy. This can also include proposed repeal of an existing policy; 3. Were newly drafted or revised; 4. Received official approval (legislation/decree) by the relevant authority (legislative or executive body) of a new, revised, or repealed policy; 5. Were fully and effectively implemented by the relevant authority (this includes USG support to implementing the effective repeal of a policy). Policies can include laws, legal frameworks, regulations, administrative procedures, or institutional arrangements. Note that the indicator has been revised to acknowledge that these processes are not always linear: newly drafted laws can be defeated by a legislative body and require redrafting or new analysis; or approved regulations can prove difficult to implement and need to be revised. Because of this non-linear approach, double counting is no longer a concern and is in fact appropriate: Partners should indicate if multiple processes/steps were completed in a given year, as this more accurately represents work under a given activity. The disaggregate “Total policies passing through one or more processes/steps of policy change” will count the total number of policies that completed any process/step, regardless of the number of processes/steps each policy completed during the reporting year.
FMD Activity MEL Plan, December 2017 15
Full and effective implementation must meet the following criteria: (1) The policy must be in force in all intended geographic locations and at all intended administrative levels with all intended regulations/rules in place (“full”); (2) Any ongoing activities or tasks required by the policy (e.g., various kinds of inspection, enforcement, collection of documents/information/fees) are being executed with minimal disruptions (“effective”). For example, a new business registration procedure that has been rolled out to just four of six intended provinces would not meet these criteria (not full), nor would a new customs law that is on the books but is not being regularly enforced at the border (not effective). For regional missions, approval (step 4) counts any regionally agreed policies that have been regionally approved (i.e., reached the minimum number of signatory countries to be passed) during the reporting year. Full and effective implementation (step 5) would count any regionally agreed policy for which all countries falling under the policy’s jurisdiction have fully and effectively implemented the policy. To capture individual countries’ progress toward full and effective implementation of regional policies, use FTFMS-only indicator 4.5.1 (TBD9). This is a modified form of standard indicator “EG.3.1-12 Number of agricultural and nutritional enabling environment policies analyzed, consulted on, drafted or revised, approved, and implemented with USG assistance” where it covers broadly Economic Growth and Agriculture Program of USAID/Pakistan
Unit of Measure: Type of Indicator: Category: Desired Direction:
Enter unit of measure (e.g. “number of___”, “percent of
___” etc.)
Enter “output”, “outcome” or “impact”.
Enter “Standard F” or “Custom”
If “Standard F indicator”, enter
the number
Enter “increasing”, “decreasing” or ‘static” to indicate the direction of
success result.
Number of policies Output (Stages 1 & 2) Outcome (Stages 3-5)
Custom (Modified)
EG.3.1-12 Increasing
Aggregation Process: If indicator will be collected by more than one source, explain how the data will aggregate across these multiple sources (e.g. in the case of # of jobs, demonstrate how data definitions for what is counted as a “job” is consistently interpreted across sources and specify that he data reported by each partner will be added together for a combined total; or in the case of a stage of phase indicator, state how data from different partners will combine into one final data). Also specify the timeline for aggregation (e.g. all sources will be added together each quarter). All data are added together across all data collection/reporting entities at respective stage. Formula of aggregation for each stage will be: Ps1,1 + Ps1,2 + Ps1,3 + . . . + Psi,j = Ps1
Ps2,1 + Ps2,2 + Ps2,3 + . . . + Psi,j = Ps2
Ps3,1 + Ps3,2 + Ps3,3 + . . . + Psi,j = Ps3
Ps4,1 + Ps4,2 + Ps4,3 + . . . + Psi,j = Ps4
Ps5,1 + Ps5,2 + Ps5,3 + . . . + Psi,j = Ps5
Ps is Policy stage, i is subscript for stage and j is subscript for partners/projects.
Disaggregates: Enter all disaggregation titles/ categories and values (e.g. title: Household Head Type; values: Female no Male Adult households, Male no Female Adults households, Male and Female Adult households, Child no Adult households.)
Policy Stages Process/step: analysis; stakeholder consultation/public debate; drafting or revision; approval (legislative or regulatory); full and effective implementation; Total policies passing through one or more processes/steps of policy change
DATA COLLECTION, STORAGE, and ANALYSIS
Name of IP/ Responsible Party for Data Collection:
Frequency of data collection: Enter how often the data will be collected (Weekly, Monthly, etc.)
PACE, PEEP, PREIA, Satpara Development Project, Firms, PSSP, FMD
Annually
Data Source: Enter where IP obtains data (e.g. self-collected, GOP records or private sector).
Data Entry Frequency into PakInfo: Enter the anticipated frequency of regular data entry into PakInfo (e.g. Quarterly, Annually, etc)
Responsible Party for Data Entry into PakInfo: Enter who will be responsible for inputting and submitting data via PakInfo.
FMD Activity MEL Plan, December 2017 16
Partner will collect data as regular M&E activities
Annually PACE, PEEP, PREIA, Satpara Development Project, Firms, PSSP, FMD
Data collection method: Enter the tools and methods to be used for data collection and indicate for each method who (IP, USAID or third party) will collect the data. (e.g. telephone survey of household sample, reading assessment administered by third-party, sign-in sheets of training participants by IP)
Project will recode progress of their policy work and collect data about number of policies on specific stage of development through regular monitoring exercise.
Data Analysis Plan: Enter how the data will be analyzed, including description of methodology (e.g. descriptive, comparative, qualitative or quantitative) as well as who will participate in the data analysis process (e.g. activity manager, chief of party, other stakeholders, GOP representatives, etc.)
Descriptive, comparative (pre/post activity, sector wise, grants wise), quantitative (numbers) Simple count of reports and documents by monitoring &evaluation team.
DATA QUALITY
Data Quality Assessment (DQA): Enter the date the DQA was conducted and the person who conducted the DQA
Date: (MM/YY) DQA completed by:
10/2016 Third Party (Monitoring and Evaluation Project with MSI)
Key Data Quality Limitations (if any) and Actions Planned to Address Those Limitations: Enter data limitations identified in the data quality assessment process related to the five quality standards, namely validity, integrity, precision, reliability and timeliness; discuss the significance of data weakness that may affect the conclusions about the extent to which performance goals have achieved; describe corrective actions planned or taken for addressing data weakness.
CHANGES & UPDATES
Date (MM/YY)
Name Enter who made
updates Change or Update Made:
Reason for Change or Update:
08/2016 Nazim Change in Definition and language of indicator
New F-Framework
12/12/17 McHugh Slight modification to correct partner/mechanism name
Slight correction
FMD Activity MEL Plan, December 2017 17
PERFORMANCE INDICATOR REFERENCE SHEET (PIRS)
Indicator: 2.2.1a Person-hours of USG-supported training completed in trade and investment
Development Objective (DO): 2 Improved Economic Status of Focus Populations and Sectors
Intermediate Result (IR): 2.2 Improved Business Enabling Environment
Sub-Intermediate Result (Sub-IR): 2.2.1 Improved Ability to Develop and Implement Reform of Policies, Laws, and Regulations
Relationship between the Sub-IR and IR or IR and DO: Enter the explanation of the linkage between the lowest level of result represented by the indicator, and the next level of result up; address the “so what?” question to move from outputs to outcomes, or outcomes to impact; explain in terms of the development hypotheses, do not simply restate the structure of the Results Framework.
Improving the ability of government and professional bodies to develop and implement reforms of policies, laws, and regulations will result in an improved business environment, provided that those bodies are willing and able to make productive use of such improvements.
INDICATOR DESCRIPTION
Precise Definition(s): Enter the precise definition of the indicator so it can be operationalized; define all terms, elements, implied actions and calculations; [for example, “farmers using better production techniques” – define “better production” and “techniques”. Describe how this will be determined – e.g. Index, scale, standards]. For indicators that are percent or proportions explain how it will be calculated and what will serve as the numerator and denominator. If the indicator is cumulative, made up of stages or phases, or is a yes-no, please specify this and explain the stages/phases or how it is cumulative. If it is a Standard Program Structure (“F”) Indicator, use and if necessary, refine the standard definition.
This indicator uses the following equation to express the number of USG-supported training hours that were completed by training participants: Hours of USG-supported training course x Number of people completing that training course Support from the USG: This indicator counts training hours that were delivered in full or in part as a result of USG assistance. USG support may include provision of funds to pay teachers, providing hosting facilities, or other key contributions necessary to ensure training was delivered. This indicator does not automatically count any course for which the USG helped develop the curriculum, but rather focuses on delivery of courses that was made possible through full or partial funding or in-kind assistance from the USG. People: Only people who complete the entire training course are counted for this indicator. Training: Training is defined as sessions in which participants are educated according to a defined curriculum and set learning objectives. Sessions that could be informative or educational, such as meetings, but do not have a defined curriculum or learning objectives, are not counted as training. Trade and investment is defined as the policies, laws, regulations, and administrative practices affecting international trade and investment and the public and private sector institutions that support sustained, locally driven improvements in the trade environment that benefit both men and women; and the collection of services, technologies, equipment, and techniques used to enhance private sector response to international trade and investment opportunities.
Unit of Measure: Type of Indicator: Category: Desired Direction:
Enter unit of measure (e.g. “number of___”, “percent of
___” etc.)
Enter “output”, “outcome” or “impact”.
Enter “Standard F” or “Custom”
If “Standard F indicator”, enter
the number
Enter “increasing”, “decreasing” or ‘static” to indicate the direction of
success result.
Number of days Output Standard EG.2-1 Increasing
Aggregation Process: All data are added together across all data collection/reporting entities.
FMD Activity MEL Plan, December 2017 18
Disaggregates: Gender (Male, Female)
DATA COLLECTION, STORAGE, and ANALYSIS
Name of IP/ Responsible Party for Data Collection:
Frequency of data collection: Enter how often the data will be collected (Weekly, Monthly, etc.)
PACE, PREIA, PEEP, FMD Quarterly
Data Source: Enter where IP obtains data (e.g. self-collected, GOP records or private sector).
Data Entry Frequency into PakInfo: Enter the anticipated frequency of regular data entry into PakInfo (e.g. Quarterly, Annually, etc.)
Responsible Party for Data Entry into PakInfo: Enter who will be responsible for inputting and submitting data via PakInfo.
Sign in sheets and M&E forms recording training events
Quarterly PACE, PREIA, PEEP, FMD
Data collection method: Enter the tools and methods to be used for data collection and indicate for each method who (IP, USAID or third party) will collect the data. (e.g. telephone survey of household sample, reading assessment administered by third-party, sign-in sheets of training participants by IP)
Specific M&E tools (trackers/templates, attendance sheets, training review forms and reports on status/progress on activities will be filled by partners.
Data Analysis Plan: Enter how the data will be analyzed, including description of methodology (e.g. descriptive, comparative, qualitative or quantitative) as well as who will participate in the data analysis process (e.g. activity manager, chief of party, other stakeholders, GOP representatives, etc.) Data will be analyzed using comparative and descriptive statistics and presented in the forms of tables, graphs and reports.
DATA QUALITY
Data Quality Assessment (DQA): Enter the date the DQA was conducted and the person who conducted the DQA
Date: (MM/YY) DQA completed by:
TBD
Key Data Quality Limitations (if any) and Actions Planned to Address Those Limitations: Enter data limitations identified in the data quality assessment process related to the five quality standards, namely validity, integrity, precision, reliability and timeliness; discuss the significance of data weakness that may affect the conclusions about the extent to which performance goals have achieved; describe corrective actions planned or taken for addressing data weakness.
OTHER NOTES / NEXT STEPS
If the indicator is pending, explain why and expected date when collection will begin. As appropriate, indicate any other important information about the indicator and/or its data collection as well as actions needing to be taken.
CHANGES & UPDATES
Date (MM/YY)
Name Enter who made
updates Change or Update Made:
Reason for Change or Update:
6/2016 Nazim Ali Drafted as new PIRS
12/12/17 McHugh Slight modification to correct partner/mechanism name
Slight correction
FMD Activity MEL Plan, December 2017 19
PERFORMANCE INDICATOR REFERENCE SHEET (PIRS)
Indicator: 2.2.1b Number of persons receiving USG supported training on policy development, analysis, and implementation
Development Objective (DO): 2 Improved Economic Status of Focus Populations and Sectors
Intermediate Result (IR) - # and Title: 2.2 Improved Business Enabling Environment
Sub-Intermediate Result (Sub-IR): 2.2.1 Improved Ability to Develop and Implement Reform of Policies, Laws, and Regulations
Relationship between the Sub-IR and IR or IR and DO: Enter the explanation of the linkage between the lowest level of result represented by the indicator, and the next level of result up; address the “so what?” question to move from outputs to outcomes, or outcomes to impact; explain in terms of the development hypotheses, do not simply restate the structure of the Results Framework. Improving the ability of government and professional bodies to develop and implement reforms of policies, laws, and regulations will result in an improved business environment, provided that those bodies are willing and able to make productive use of such improvements.
INDICATOR DESCRIPTION
Precise Definition(s): Enter the precise definition of the indicator so it can be operationalized; define all terms, elements, implied actions and calculations; [for example, “farmers using better production techniques” – define “better production” and “techniques”. Describe how this will be determined – e.g. Index, scale, standards]. For indicators that are percent or proportions explain how it will be calculated and what will serve as the numerator and denominator. If the indicator is cumulative, made up of stages or phases, or is a yes-no, please specify this and explain the stages/phases or how it is cumulative. If it is a Standard Program Structure (“F”) Indicator, use and if necessary, refine the standard definition.
Indicator counts persons undergoing training relating to policy development, analysis, and implementation. Beneficiaries may include government officials trained on policy development or implementation, journalists trained on reporting on economic policy issues, and researchers trained on selecting and carrying out applied research on topics related to the policy-making process. This indicator does not automatically count any course for which the USG helped develop the curriculum, but rather focuses on delivery of courses that was made possible through full or partial funding from the USG. People: Only people who complete at least 70% of the entire training course are counted for this indicator. Training: Training is defined as sessions in which participants are educated according to a defined curriculum and set learning objectives. Sessions that could be informative or educational, such as meetings, but do not have a defined curriculum and learning objectives are not counted as training.
Unit of Measure: Type of Indicator: Category: Desired Direction:
Enter unit of measure (e.g. “number of___”, “percent of
___” etc.)
Enter “output”, “outcome” or “impact”.
Enter “Standard F” or “Custom”
If “Standard F indicator”, enter
the number
Enter “increasing”, “decreasing” or ‘static” to indicate the direction of
success result.
Number of persons Output Custom Increasing
Aggregation Process: If indicator will be collected by more than one source, explain how the data will aggregate across these multiple sources (e.g. in the case of # of jobs, demonstrate how data definitions for what is counted as a “job” is consistently interpreted across sources and specify that he data reported by each partner will be added together for a combined total; or in the case of a stage of phase indicator, state how data from different partners will combine into one final data). Also specify the timeline for aggregation (e.g. all sources will be added together each quarter).
All data are added together across all data collection/reporting entities.
Disaggregates: Enter all disaggregation titles/ categories and values (e.g. title: Household Head Type; values: Female no Male Adult households, Male no Female Adults households, Male and Female Adult households, Child no Adult households.)
Gender; male, female
FMD Activity MEL Plan, December 2017 20
DATA COLLECTION, STORAGE, and ANALYSIS
Name of IP/ Responsible Party for Data Collection:
Frequency of data collection: Enter how often the data will be collected (Weekly, Monthly, etc.)
PACE, PREIA, PEEP, PSSP, DOC632b, Trade Project
Quarterly
Data Source: Enter where IP obtains data (e.g. self-collected, GOP records or private sector).
Data Entry Frequency into PakInfo: Enter the anticipated frequency of regular data entry into PakInfo (e.g. Quarterly, Annually, etc.)
Responsible Party for Data Entry into PakInfo: Enter who will be responsible for inputting and submitting data via PakInfo.
Sign in sheets and M&E forms recording training events
Quarterly PACE, PREIA, PEEP, PSSP, DOC632b, Trade Project
Data collection method: Enter the tools and methods to be used for data collection and indicate for each method who (IP, USAID or third party) will collect the data. (e.g. telephone survey of household sample, reading assessment administered by third-party, sign-in sheets of training participants by IP)
Specific M&E tools (trackers/templates, attendance sheets, training review forms and reports on status/progress on activities will be filled by partners.
Data Analysis Plan: Enter how the data will be analyzed, including description of methodology (e.g. descriptive, comparative, qualitative or quantitative) as well as who will participate in the data analysis process (e.g. activity manager, chief of party, other stakeholders, GOP representatives, etc.)
Data will be analyzed using comparative and descriptive statistics and presented in the forms of tables, graphs and reports.
DATA QUALITY
Data Quality Assessment (DQA): Enter the date the DQA was conducted and the person who conducted the DQA
Date: (MM/YY) DQA completed by:
10/2016 Third Party (Monitoring and Evaluation Project with MSI)
Key Data Quality Limitations (if any) and Actions Planned to Address Those Limitations: Enter data limitations identified in the data quality assessment process related to the five quality standards, namely validity, integrity, precision, reliability and timeliness; discuss the significance of data weakness that may affect the conclusions about the extent to which performance goals have achieved; describe corrective actions planned or taken for addressing data weakness.
OTHER NOTES / NEXT STEPS
If the indicator is pending, explain why and expected date when collection will begin. As appropriate, indicate any other important information about the indicator and/or its data collection as well as actions needing to be taken.
CHANGES & UPDATES
Date (MM/YY)
Name Enter who made
updates Change or Update Made:
Reason for Change or Update:
PERFORMANCE INDICATOR REFERENCE SHEET (PIRS)
FMD Activity MEL Plan, December 2017 21
Indicator: 2.2.2-a Number of USG-assisted organizations that participate in legislative proceedings and/or engage in advocacy at all levels
Development Objective (DO): 2 Improved Economic Status of Focus Populations and Sectors
Intermediate Result (IR) - # and Title: 2.2 Improved Business Enabling Environment
Sub-Intermediate Result (Sub-IR): 2.2.2 Strengthened Private Sector and Civil Society Engagement in Policy-Making
Relationship between the Sub-IR and IR or IR and DO: Enter the explanation of the linkage between the lowest level of result represented by the indicator, and the next level of result up; address the “so what?” question to move from outputs to outcomes, or outcomes to impact; explain in terms of the development hypotheses, do not simply restate the structure of the Results Framework.
In order to improve the business enabling environment, it is necessary that the private sector and civil society be aware of policy-related issues and able to engage with policy-makers to make their views on these issues known. Such activity forms the “demand” side of the policy development process that results in good policy outcomes.
INDICATOR DESCRIPTION
Precise Definition(s): Enter the precise definition of the indicator so it can be operationalized; define all terms, elements, implied actions and calculations; [for example, “farmers using better production techniques” – define “better production” and “techniques”. Describe how this will be determined – e.g. Index, scale, standards]. For indicators that are percent or proportions explain how it will be calculated and what will serve as the numerator and denominator. If the indicator is cumulative, made up of stages or phases, or is a yes-no, please specify this and explain the stages/phases or how it is cumulative. If it is a Standard Program Structure (“F”) Indicator, use and if necessary, refine the standard definition.
The number of organizations (NGOs) that participate in sessions of federal or regional parliaments (or their committees) or engage in advocacy activities in the public domain that have received USG assistance on advocacy and civic participation. For an organization to be counted, it must be on record as having participated in such parliamentary sessions or engaged in advocacy activities, such as participating in private or public meetings with policy-makers, publishing opinion pieces in widely circulated newspapers or Web sites, engaging in public (e.g., televised or broadcast) debates, and so on. It must also have received USG assistance (e.g., participated in training events) aimed specifically at building its capacity to engage in such activities.
Unit of Measure: Type of Indicator: Category: Desired Direction:
Enter unit of measure (e.g. “number of___”, “percent of
___” etc.)
Enter “output”, “outcome” or “impact”.
Enter “Standard F” or “Custom”
If “Standard F indicator”, enter
the number
Enter “increasing”, “decreasing” or ‘static” to indicate the direction of
success result.
Number of organizations Output Custom Increasing
Aggregation Process: If indicator will be collected by more than one source, explain how the data will aggregate across these multiple sources (e.g. in the case of # of jobs, demonstrate how data definitions for what is counted as a “job” is consistently interpreted across sources and specify that he data reported by each partner will be added together for a combined total; or in the case of a stage of phase indicator, state how data from different partners will combine into one final data). Also specify the timeline for aggregation (e.g. all sources will be added together each quarter).
All data are added together across all data collection/reporting entities.
Disaggregates: Enter all disaggregation titles/ categories and values (e.g. title: Household Head Type; values: Female no Male Adult households, Male no Female Adults households, Male and Female Adult households, Child no Adult households.)
Non
DATA COLLECTION, STORAGE, and ANALYSIS
Name of IP/ Responsible Party for Data Collection:
Frequency of data collection: Enter how often the data will be collected (Weekly, Monthly, etc.)
PACE, PREIA, PEEP, SDP, PSSP, TAP, DOC, FMD
Quarterly
FMD Activity MEL Plan, December 2017 22
Data Source: Enter where IP obtains data (e.g. self-collected, GOP records or private sector).
Data Entry Frequency into PakInfo: Enter the anticipated frequency of regular data entry into PakInfo (e.g. Quarterly, Annually, etc.)
Responsible Party for Data Entry into PakInfo: Enter who will be responsible for inputting and submitting data via PakInfo.
Partners’ reports. Quarterly PACE, PREIA, PEEP, SDP, PSSP, TAP, DOC, FMD
Data collection method: Enter the tools and methods to be used for data collection and indicate for each method who (IP, USAID or third party) will collect the data. (e.g. telephone survey of household sample, reading assessment administered by third-party, sign-in sheets of training participants by IP)
Qualitative and quantitative analysis; specific M&E tools (trackers/templates); through beneficiary registration, attendance sheets and training evaluation forms.
Data Analysis Plan: Enter how the data will be analyzed, including description of methodology (e.g. descriptive, comparative, qualitative or quantitative) as well as who will participate in the data analysis process (e.g. activity manager, chief of party, other stakeholders, GOP representatives, etc.)
Mixed methods using quantitative and qualitative data. Automated generation of output/ summary tables.
DATA QUALITY
Data Quality Assessment (DQA): Enter the date the DQA was conducted and the person who conducted the DQA
Date: (MM/YY) DQA completed by:
10/2016 Third Party MSI
Key Data Quality Limitations (if any) and Actions Planned to Address Those Limitations: Enter data limitations identified in the data quality assessment process related to the five quality standards, namely validity, integrity, precision, reliability and timeliness; discuss the significance of data weakness that may affect the conclusions about the extent to which performance goals have achieved; describe corrective actions planned or taken for addressing data weakness.
CHANGES & UPDATES
Date (MM/YY)
Name Enter who made
updates Change or Update Made:
Reason for Change or Update:
1/2017 Nazim Ali Updated partners reporting and DQA dates, removed disags
12/12/17 McHugh Slight modification to correct partner/mechanism name
Slight correction
FMD Activity MEL Plan, December 2017 23
PERFORMANCE INDICATOR REFERENCE SHEET (PIRS)
Indicator- # and Title: 2.2.2b. Number of events for public/stakeholders consultation to enhance sector governance or facilitate private sector engagement in policy making as a result of USG assistance
Development Objective (DO): 2 Improved Economic Status of Focus Populations and Sectors
Intermediate Result (IR): 2.2 Improved Business Enabling Environment
Sub-Intermediate Result (Sub-IR): 2.2.2 Strengthened Private Sector and Civil Society Engagement in Policy-Making
Relationship between the Sub-IR and IR or IR and DO: Enter the explanation of the linkage between the lowest level of result represented by the indicator, and the next level of result up; address the “so what?” question to move from outputs to outcomes, or outcomes to impact; explain in terms of the development hypotheses, do not simply restate the structure of the Results Framework. In order to improve the business enabling environment, it is necessary that the private sector and civil society be aware of policy-related issues and able to engage with policy-makers to make their views on these issues known. Such activity forms the “demand” side of the policy development process that results in good policy outcomes. This measures the drafting of critical policy/procedural improvements and presenting them for broad stakeholder consultation, which help to further sector reform efforts in a way to build broad consensus and thus help ensure reforms are successful and sustainable.
INDICATOR DESCRIPTION
Precise Definition(s): Enter the precise definition of the indicator so it can be operationalized; define all terms, elements, implied actions and calculations; [for example, “farmers using better production techniques” – define “better production” and “techniques”. Describe how this will be determined – e.g. Index, scale, standards]. For indicators that are percent or proportions explain how it will be calculated and what will serve as the numerator and denominator. If the indicator is cumulative, made up of stages or phases, or is a yes-no, please specify this and explain the stages/phases or how it is cumulative. If it is a Standard Program Structure (“F”) Indicator, use and if necessary, refine the standard definition.
This indicator counts number of events held to consult/present a broadly representative cross-section of stakeholders (e.g. private sector and civil society) on policy issues as a result of USG assistance. These events include conferences, town hall meetings, debates, advocacy events which are arranged to present current policy status of sector/subsector, policy analysis, and proposed changes in policy.
Unit of Measure: Type of Indicator: Category: Desired Direction:
Enter unit of measure (e.g. “number of___”, “percent of
___” etc.)
Enter “output”, “outcome” or “impact”.
Enter “Standard F” or “Custom”
If “Standard F indicator”, enter
the number
Enter “increasing”, “decreasing” or ‘static” to indicate the direction of
success result.
Number of events Output Custom Increasing
Aggregation Process: If indicator will be collected by more than one source, explain how the data will aggregate across these multiple sources (e.g. in the case of # of jobs, demonstrate how data definitions for what is counted as a “job” is consistently interpreted across sources and specify that he data reported by each partner will be added together for a combined total; or in the case of a stage of phase indicator, state how data from different partners will combine into one final data). Also specify the timeline for aggregation (e.g. all sources will be added together each quarter).
All data are added together across all data collection/reporting entities.
Disaggregates: Enter all disaggregation titles/ categories and values (e.g. title: Household Head Type; values: Female no Male Adult households, Male no Female Adults households, Male and Female Adult households, Child no Adult households.)
District; EGA Type of Event (Conference, Workshop, Seminar)
DATA COLLECTION, STORAGE, and ANALYSIS
Name of IP/ Responsible Party for Data Collection:
Frequency of data collection: Enter how often the data will be collected (Weekly, Monthly, etc.)
PACE, PEEP, PREIA, GAIN, PSSP, DOC632b, USDA632b, FMD
Quarterly
FMD Activity MEL Plan, December 2017 24
Data Source: Enter where IP obtains data (e.g. self-collected, GOP records or private sector).
Data Entry Frequency into PakInfo: Enter the anticipated frequency of regular data entry into PakInfo (e.g. Quarterly, Annually, etc.)
Responsible Party for Data Entry into PakInfo: Enter who will be responsible for inputting and submitting data via PakInfo.
Partners’ reports Quarterly PACE, PEEP, PREIA, GAIN PSSP, DOC632b, USDA632b, FMD
Data collection method: Enter the tools and methods to be used for data collection and indicate for each method who (IP, USAID or third party) will collect the data. (e.g. telephone survey of household sample, reading assessment administered by third-party, sign-in sheets of training participants by IP)
Qualitative and quantitative analysis; specific M&E tools (trackers/templates); through beneficiary registration, attendance sheets and training evaluation forms.
Data Analysis Plan: Enter how the data will be analyzed, including description of methodology (e.g. descriptive, comparative, qualitative or quantitative) as well as who will participate in the data analysis process (e.g. activity manager, chief of party, other stakeholders, GOP representatives, etc.)
Mixed methods using quantitative and qualitative data. Automated generation of output/ summary tables.
DATA QUALITY
Data Quality Assessment (DQA): Enter the date the DQA was conducted and the person who conducted the DQA
Date: (MM/YY) DQA completed by:
10/2017 Third Party (Monitoring and Evaluation Project with MSI)
Key Data Quality Limitations (if any) and Actions Planned to Address Those Limitations: Enter data limitations identified in the data quality assessment process related to the five quality standards, namely validity, integrity, precision, reliability and timeliness; discuss the significance of data weakness that may affect the conclusions about the extent to which performance goals have achieved; describe corrective actions planned or taken for addressing data weakness.
OTHER NOTES / NEXT STEPS
If the indicator is pending, explain why and expected date when collection will begin. As appropriate, indicate any other important information about the indicator and/or its data collection as well as actions needing to be taken.
CHANGES & UPDATES
Date (MM/YY)
Name Enter who made
updates Change or Update Made:
Reason for Change or Update:
1/2017 Nazim Ali Updated partners reporting and DQA dates
12/12/17 McHugh Slight modification to correct partner/mechanism name
Slight correction
FMD Activity MEL Plan, December 2017 25
ANNEX B: ASSURING DATA QUALITY
Data Quality Checklist and Recommended Procedures
USAID Mission Name:
Title of Performance Indicator:
Linkage to Foreign Assistance Standardized Program Structure, if applicable (i.e. Program Area,
Element, etc.):
Result This Indicator Measures (i.e., Specify the MSF Development Objective, Intermediate Result,
etc.):
FMD Data Source(s):
Partner or Contractor/ Source Who Provided the Data:
Period for Which the Data Are Being Reported:
Is This Indicator a Standard or Custom Indicator? __ Standard Foreign Assistance Indicator
___ Custom (created by USAID/Pakistan; not
standard)
Data Quality Assessment methodology:
Date(s) of Assessment:
Assessment Team Members:
USAID/Pakistan Verification of DQA
Team Leader Officer approval
X_______________________________________
S. No Elements of Data Quality Acceptable?
Comments Yes No
VALIDITY – Data should clearly and adequately represent the intended result.
1
Does the information collected measure
what it is supposed to measure? (E.g., A
valid measure of overall nutrition is
healthy variation in diet: Age is not a
valid measure of overall health.)
2
Do results collected fall within a plausible
range? (E.g., A plausible range for height
would not be 8 feet 11 inches.)
FMD Activity MEL Plan, December 2017 26
S. No Elements of Data Quality Acceptable?
Comments Yes No
3
Is there reasonable assurance that the
data collection methods being used do
not produce systematically biased data
(e.g., consistently over- or under-
counting)?
4
Are sound research methods being used
to collect the data? (E.g., if the result is a
change in perception, a participant sign-
in sheet would not be a sound research
method to collect the data.)
INTEGRITY – Data collected should have safeguards to minimize the risk of transcription error or
data manipulation.
1
Are procedures or safeguards in place to
minimize data transcription error? (E.g.,
entering the same dataset twice; training
of data staff; double-keying; rechecking
by a supervisor; taking samples and
comparing the entered data to the
original source.)
2
Is there independence in key data
collection, management, and assessment
procedures? (E.g., if the same staff
collects the data, enter it into MIS, and
review and analyze the data, then there is
no independence in the procedures, and
bias may be an issue.)
3
Are mechanisms in place to prevent
unauthorized changes to the data? (E.g.,
can any staff access the database or
public drive and change the data?)
PRECISION – Data have a sufficient level of detail to permit management decision making; e.g. the
margin of error is less than the anticipated change.
1
Has the margin of error been reported along
with the data? (Only applicable to results
obtained through statistical samples.)
2
Is the margin of error less than the expected
change being measured? (E.g. If a change of
only 2% is expected and the margin of error
in a survey used to collect the data is +/- 5%,
then the tool is not precise enough to detect
the change.)
FMD Activity MEL Plan, December 2017 27
S. No Elements of Data Quality Acceptable?
Comments Yes No
3
Is the data collection method/tool being used
to collect the data fine-tuned or exact
enough to register the expected change?
(E.g. A yardstick may not be a precise
enough tool to measure a change of a few
millimeters.)
4
Is the detail of the data aligned with
management decision making
requirements? (E.g., A percentage score
of 60.43527 may not be necessary when
analyzing data on citizen perceptions,
but it might be critical when looking at
water quality.)
RELIABILITY – Data should reflect stable and consistent data collection processes and analysis
methods over time.
1
Is a consistent data collection process used
from time to time, location to location, data
source to data source (if data come from
different sources)?
2
When the same data collection method is
used to measure/observe the same thing
multiple times, is the same result
produced each time? (E.g., A ruler used
over and over always indicates the same
length for an inch.)
3
Are data collection and analysis methods
documented in writing and being used to
ensure the same procedures are followed
each time? (This could be included in the
PIRS, the M&E plan, or in some other
handbook or guide (e.g., a guide for
uploading data into a database/MIS
system).
4 Are data limitations and quality problems
clearly described in final reports?
TIMELINESS – Data should be available at a useful frequency, should be current, and should be timely
enough to influence management decision making.
1 Are data available frequently enough to
inform program management decisions?
2 Are the data reported the most current?
FMD Activity MEL Plan, December 2017 28
S. No Elements of Data Quality Acceptable?
Comments Yes No
3 Are the data reported as soon as possible
after collection?
SUMMARY
Based on the assessment relative to the five standards, what is the overall conclusion regarding the
quality of the data?
Significance of limitations (if any):
Actions needed to address limitations prior to the next DQA (given level of USG control over data):
IF NO DATA ARE AVAILABLE FOR THE
INDICATOR
COMMENTS
If no recent relevant data are available for this indicator, why not?
What concrete actions are now being taken to collect and report these data as soon as possible?
When will data be reported?
Recommendations for Conducting Data Quality Assessments
1. Data Quality (DQ) assessor should make sure that they understand the precise definition of the
indicator by checking the Performance Indicator Reference Sheet. Please address any issues of ambiguity
before the DQA is conducted.
2. DQ assessor should have a copy of the methodology for data collection in hand before assessing the
indicator. For USAID Missions, this information should be in the PMP’s Performance Indicator Reference
Sheets for each indicator. Each indicator should have a written description of how the data being
assessed are supposed to be collected.
3. Each implementing partner should have a copy of the method of data collection in their files and
documented evidence that they are collecting the data according to the methodology.
4. DQ assessor should record the names and titles of all individuals involved in the assessment.
5. Does the implementing partner have documented evidence that they have verified the data that has
been reported? Partners should be able to provide USAID with documents (process/person conducting
the verification/field visit dates/persons met/activities visited, etc.) which demonstrates that they have
verified the data that was reported. Note: Verification by the partners should be an ongoing process.
FMD Activity MEL Plan, December 2017 29
6. The DQ assessor should be able to review the implementing partner files/records against the
methodology for data collection laid out in the PMP (for USAID Missions only). Any data quality
concerns should be documented.
7. The DQ should include a summary of significant limitations found. A plan of action, including timelines
and responsibilities, for addressing the limitations should be made.