fiinovationOpportunity Solutions, Simplified.

MONITORING &EVALUATIONFRAMEWORK

IndexMonitoring and Evaluation & it’s importance1

Monitoring and Evaluation framework in the Project Program Cycle02

Monitoring and the inclusion of RBM03

Types of Monitoring04

Tailor-Made Evaluation Studies06

Evaluation Questions & RBM08

Six key steps for M&E Planning09

Minimizing Bias and Errors10

CSR in India - Industry Overview11

Fiinovation Expertise12

Some of our Initiatives 13

Organization Brief14

Awards & Recognition15

What Does Monitoring and Evaluation Mean?

Monitoring Evaluation

Why is Monitoring and Evaluation Important?Organisations use monitoring and evaluation for two key purposes:

01

Does your projecthave an M&E plan?

Does your projecthave an M&E binder?

1

Have you developed all monitoring forms?

3

Are monitoring data regularly used duringM&E meetings or other events?

8

Are staff and partners trained on data entry and analysis?

7

Are staff and partners trained onusing the monitoring forms?

4

Have you created amonitoring database?

6

Have you conducted a qualitycheck of the monitoring data?

5

If your answer is to any of the questions above, the project team should work to complete the ‘No'step(s). This review tool has not been designed to be used in isolation and refers to the package of practices being offered at Fiinovation, and this handbook is for further guidance at each review stage.

To learn about their own activities and results, and to support internal planning and development

To be accountable to their stakeholders.

Monitoring is a routine and systematic collection of information against a plan. The information might be about activities, products or services, users, or about outside factors affecting the organisation or project.

Evaluation is about making judgements about the value of any component part of an organisation

or its products, services or benets, or about the organisation as a whole.

Organisations need evidence of their efciency and effectiveness for funders, commissioners and investors. They also need to communicate achievements to a wider public. Charities are now specically required to report the achievements of the charity over the year. However, the role of monitoring and evaluation is providing learning and improving the organisation is of equal importance.

The following eight questions shall help in creating a better understanding on the need of monitoring and evaluation:

2

Monitoring & Evaluation

M&E Framework in the Project Program Cycle

02

Initial NeedsAssessment

Log frame and indicators

M&E planning

Baseline assessment

Midterm evaluation

and/or reviews

Final evaluation

Dissemination and use of

lessons

This is done to determine whether a project/program is needed and, if so, to inform its planning.

This involves the operational design of the project/program and its objectives, indicators, means of verification and assumptions

This is the practical planning for the project/program to monitor and evaluate the log frame's objectives and indicators.

This is the measurement of the initial conditions (appropriate indicators) before the start of a project/program.These are important reflection events

to assess and inform ongoing project/program implementation.

This occurs after project/program completion to assess how well the project/program achieved its intended objectives and what difference this has made.

This informs ongoing programming. Although this has been put in the last section of the diagram but, reporting, reflection and learning should occur throughout the whole project/program cycle.

There is no magic 10-percent sampling rule. It is important to note that the sample size is not related to the size of the popula�on being sampled. A frequent mistake is to conduct surveys among 10 percent of a given popula�on; in fact, it is likely that 10 percent of the popula�on is either too many or too few households. With too many households, the survey is using excessive resources and �me; with too few households, the sample will not adequately represent the popula�on.

Monitoring & Evaluation

Monitoring and the Inclusion of RBM

ImpactŸ Measuring changes at impact level requires a longer time frame, and is therefore dealt with

by evaluation and not monitoring.

OutcomesŸ Are the outputs leading to the achievement of the outcomes?Ÿ Is there anything happening that should lead management to modify the operations

implementation plan?

Output

ActivitiesŸ Are activities leading to expected outputs?Ÿ Are activities being implemented on schedule and within the budget?

Inputs

Monitoring Questions and RBM

03

A mistake common for M&E systems is to rely solely on either observa�on data or par�cipant responses. Observa�on data alone does not provide an explana�on of prac�ces or behaviors and o�en requires large assump�ons on the part of the M&E team. Focus group data (an example of par�cipant responses) may not capture important prac�ces that par�cipants do not see as relevant and may record instead what par�cipants think data collec�on teams want to hear.

Result Based Management (RBM) is an approach to project/program management based on clearly dened results, and the methodologies and tools to measure and achieve them.

RBM supports better performance and greater accountability by applying a clear, logical framework to plan, manage and measure an intervention with a focus on the results you want to achieve.

By identifying in advance the intended results of a project/program and how we can measure their progress, we can better manage a project/program and determine whether a difference has genuinely been made for the people concerned.

Monitoring and evaluation (M&E) is a critical part of RBM. It forms the basis for clear and accurate reporting on the results achieved by an intervention (projector program).

In this way, information reporting is no longer a headache, but becomes an opportunity for critical analysis and organizational learning, informing decision-making and impact assessment.

Ÿ How do beneciaries feel about the work/intervention?Ÿ What is causing delays or unexpected results?

Ÿ Are nances, personnel and materials available on time and in the right quantities and quality?

Monitoring & Evaluation

It tracks the use of inputs and resources, the progress of activities and the delivery of outputs.

It examines how activities are delivered – the efciency in time and resources.

It is often conducted in conjunction with compliance monitoring and feeds into the evaluation of impact.

For example, a water and sanitation project may monitor that targeted households receive septic systems according to schedule.

Types of Monitoring

It tracks effects and impacts.

This is where monitoring merges with evaluation to determine if the project/program is on target towards its intended results (outputs, outcomes, impact).

It measures whether there may be any unintended impact (positive or negative).

For example, a psychosocial project may monitor that its community activities achieve the outputs that contribute to community resilience and ability to recover from a disaster.

1. Results Monitoring

2. Process (Activity) Monitoring

3. Compliance Monitoring

4. Context (Situation) Monitoring

04

It ensures compliance with donor regulations and expected results, grant and contract requirements, local governmental regulations and laws, and ethical standards.

For example, a shelter project may monitor that shelters adhere to agreed national and international safety standards in construction.

It tracks the , especially as it affects identied risks setting in which the project/program operatesand assumptions, but also any unexpected considerations that may arise.

It includes the eld as well as the larger political, institutional, funding, and policy context that affect the project/program.

For example, a project in a conict-prone area may monitor potential ghting that could not only affect project’s success but, endanger the project’s staff and volunteers.

Monitoring & Evaluation

It tracks beneciary perceptions of a project/program.

It includes beneciary satisfaction or complaints with the project/program, including their participation, treatment, access to resources and their overall experience of change.

Sometimes referred to as it often includes a stakeholder Beneciary Contact Monitoring (BCM),complaints and feedback mechanism.

It should take account of different population groups, as well as the perceptions of indirect beneciaries (e.g. community members not directly receiving a good or service).

For example, a cash-for work program assisting community members after a natural disaster may monitor how they feel about the selection of program participants, the payment of participants and the contribution the program is making to the community (e.g. are these equitable?).

6. Financial Monitoring

05

5. Beneficiary Monitoring

It tracks the sustainability, institutional development and capacity building in the project/program and with its partners.

It is often done in conjunction with the monitoring processes of the larger, implementing organization.

For example, a National Society's headquarters may use organizational monitoring to track communication and collaboration in project implementation among its branches and chapters.

7. Organizational Monitoring

It accounts for costs by input and activity within predened categories of expenditure.

It is often conducted in conjunction with compliance and process monitoring.

For example, a livelihood project implementing a series of micro-enterprises may monitor the money awarded and repaid, and ensure implementation is according to the budget and time frame.

Monitoring & Evaluation

Tailor-made Evaluation StudiesEvaluations conducted during the life of the project should provide actionable recommendations to improve the quality of the project in the time remaining. The final evaluation findings must be incorporated into the strategy design for subsequent programming. The evaluation exercises can be categorized under three broad heads as enumerated below.

According to evaluation timing According to who conducts theevaluation

According to evaluationtechnicality or methodology

Formative evaluations occur d u r i n g p r o j e c t / p r o g r a m implementation to improve per formance and assess compliance.

Internal or self-evaluations are conducted by those responsible f o r i m p l e m e n t i n g a project/program. They can be less expensive than external evaluations and help build staff capac i t y and ownersh ip . Howeve r, t he y may l a c k c r e d i b i l i t y w i t h c e r t a i n stakeholders, such as donors, as they are perceived as more subjective (biased or one-sided). These tend to be focused on learning lessons rather than demonstrating accountability.

Real-time evaluations (RTEs) a r e u n d e r t a k e n d u r i n g p r o j e c t / p r o g r a m implementation to provide i m m e d i a t e f e e d b a c k f o r mod i f icat ions to improve ongo ing imp lementat i on . Emphasis is on immediate lesson learning over impact evaluation or accountability.

Summative evaluations occur at the end of project/program implementation to assess effectiveness and impact.

Externa l o r i ndependent evaluations are conducted by evaluator(s) outside of the implementing team, lending it a degree of objectivity and often technical expertise. These tend to focus on accountability.

Meta-evaluations are used to assess the evaluation process itself. Some key uses of meta-eva luat ions inc lude: take inventory of evaluations to inform the selection of future evaluations; combine evaluation results; check compliance with evaluation policy and good practices; assess how well evaluations are disseminated and utilized for organizational learning and change, etc.

Midterm evaluations are formative in purpose and o c c u r m i d w a y t h r o u g h implementation.

Participatory evaluations are c o n d u c t e d w i t h t h e beneficiaries and other key stakeholders, and can be empowering, building their capac i t y, ownersh ip and support.

Thematic evaluations focus on one theme, such as gender or environment, typically across a number of projects, programs or the whole organization.

06Monitoring & Evaluation

According to evaluation timing According to who conducts theevaluation

According to evaluationtechnicality or methodology

Final evaluations are summative in purpose and are conducted (o ften externa l l y ) at the completion of project/program implementation to assess how well the project/ program a c h i e v e d i t s i n t e n d e d objectives.

Joint evaluations are conducted collaboratively by more than one implementing partner, and can he l p bu i l d consensus at different levels, credibility and joint support.

Cluster/sector evaluations focus on a set of related a c t i v i t i e s , p r o j e c t s o r programs, typically across sites and implemented by multiple organizations (e.g. National Societies, the United Nations and NGOs).

Ex -post e va l uat i ons a re conducted some time after implementation to assess long- term impact and sustainability.

Impact evaluations focus on the effect of a project/ program, rather than on its management and delivery. Therefore, they t y p i c a l l y o c c u r a f t e r project/program completion during a final evaluation or an ex-post evaluation. However, impact may be measured during p r o j e c t / p r o g r a m implementation during longer projects/programs and when feasible. It highlights some of the challenges in measuring impact.

Proper management of an evaluation is a critical element for its success. There are multiple resources to support evaluation management which identify the key criteria and standards that guide how we plan, commission, conduct, report on and utilize evaluations. The framework mentioned in the beginning is to be applied to all evaluation activities. It draws upon the best practices to ensure accurate and reliable evaluations that are credible with stakeholders.

07Monitoring & Evaluation

Evaluation Questions and RBM

08

Impact

Outcomes

OutputŸ Were the operations objective achieved? (effectiveness)Ÿ Did the outputs lead to intended outcome?

Activities

Inputs

Ÿ Are the benets likely to be maintained for an extended period after assistance ends? (sustainability)

Ÿ What changes did the project bring about?

Ÿ Were the operation’s objectives consistent with beneciaries’ needs and with agency policies? (relevance)

Ÿ Were there any unplanned or unintended changes.?

Ÿ Were activities implemented on schedule and within budget? (efciency)Ÿ Were outputs delivered economically?

Ÿ Were stocks of items available on time and in the right quantities and quality?

Monitoring & Evaluation

Six Key Steps for M&E Planning

Review the project/program's operational design (RBM/LFA)Identify key stakeholder informational needs and expectationsIdentify any M&E requirementsScope of major M&E events and functions

Identify the purposeand scope of the M&Esystem

1.

Develop an M&E plan tableAssess the availability of secondary dataDetermine the balance of quantitative and qualitative dataTriangulate data collection sources and methodsDetermine sampling requirementsPrepare for any surveysPrepare specic data collection methods/toolsEstablish stakeholder complaints and feedback mechanismsEstablish project/program staff/volunteer review mechanismsPlan for data managementUse an indicator tracking table (ITT)Use a risk log (table)

Plan for data collectionand management

2.

Develop a data analysis plan, identifying the: i. Purpose of data analysis ii. Frequency of data analysis iii. Responsibility for data analysis iv. Process for data analysis.

Follow the key data analysis stages: i. Data preparation ii. Data analysis (ndings and conclusions) iii. Data validation iv. Data presentation v. Recommendations and action planning.

Plan for data analysis

3.

Assess the projects/program's human resources capacity for M&EDetermine the extent of local participationDetermine the extent of outside expertiseDene roles and responsibilities at each level of the M&E system.Plan to manage project/program team's M&E activitiesIdentify M&E capacity-building requirements and opportunities

Plan for M&E humanresources and capacitybuilding

5.

Itemize M&E budget needs i.Human Resource ii.Capital Expenses

Incorporate M&E costs into the project/program budgetReview any donor budget requirements and contributionsPlan for cost contingency

Prepare the M&Ebudget

6.

Anticipate and plan for reporting: i. Needs/audience ii. Frequency iii. Formats iv. People responsible.

Plan for information utilization: i. Information dissemination ii. Decision-making and planning

Plan for informationreporting and utilization

4.

09Monitoring & Evaluation

Minimizing Bias and ErrorsMinimizing bias helps to increase accuracy and precision. Accuracy means that the data measures what it is intended to measure. For example, if you are trying to measure knowledge change following a training session, you would not just measure how many people were trained but also include some type of test of any knowledge change.

As much as we would like to eliminate bias and error in our measurements and information reporting, no research is completely without bias. Nevertheless, there are precautions that can be taken, and the first is to we encounter in our work:be familiar with the major types of bias

1. Selection Bias Selection bias results from poor selection of the sample population to measure/study. Also called design bias or sample error, it occurs when the people, place or time period measured is not representative of the larger population or condition being studied. It is a very important concept to understand because there is a tendency to study the most successful and/or convenient sites or populations to reach (which are often the same). For example, if data collection is done during a convenient time of the day, during the dry season or targets communities easily accessible near paved roads, it may not accurately represent the conditions being studied for the whole population.

2. Measurement Bias Measurement bias results from poor data measurement – either due to a fault in the data measurement instrument or the data collector. Sometimes the direct measurement may be done incorrectly, or the attitudes of the interviewer may influence how questions are asked and responses are recorded. For instance, household occupancy in a disaster response operation may be calculated incorrectly, or survey questions may be written in a way that biases the response, e.g. “Why do you like this project?” (rather than “What do you think of this project?”).

3. Processing Error Processing error results from the poor management of data – miscoded data, incorrect data entry, incorrect computer programming and inadequate checking. This source of error is particularly common with the entry of quantitative (statistical) data, for which specific practices and checks have been developed.

4. Analytical Bias Analytical bias results from the poor analysis of collected data. Different approaches to data analysis generate varying results e.g. the statistical methods employed, or how the data is separated and interpreted. A good practice to help reduce analytical bias is to carefully identify the rationale for the data analysis methods.

10Monitoring & Evaluation

1CSR Expenditure Among 10 Major Industries

HEALTH

22%RURAL DEVELOPMENT

33%

EDUCATION

23%

ENVIRONMENT

22%

2012-13

Expense on CSR in the Private and Public Sector

20 400 60 80 100

Slum Area Developent

RD Projects

Technical Incubators

PM Relief Fund

Sports

Forced Army Veterans

National Heritage

Environmental Sustainability

Reducing Inequalities

Education, Vocational Skills

Hunger, Poverty, Healthcare

955

6535

1882

8515

3070

955

5842

4258

6337

7525

2278

Pri

vate

Se

cto

rP

ub

lic

Se

cto

r

1 As per the annual reports of the top 200 firms for the year 2012-132 FICCI and Nextgen

CSR in India - Industry Overview

11Monitoring & Evaluation

Fiinovation ExpertiseA vast bank of expertise gained over the years across a wide range of sectors and research methodology. And also, the flexibility to serve a wide range of projects in the sectors of: Livelihood, Education, Health and Environment.

Defined M&E standards that help improve program quality and positively impact the people we serve. The standards followed at Fiinovation reflect the key characteristics of high-quality programs and agency culture that promotes better learning and strengthens accountability to stakeholders. These are critical elements of a high- performing, dynamic learning organization.

Triangulation of data is another key step practiced at Fiinovation, which involves collecting data from multiple sources, sometimes using multiple tools, to identify and reduce bias. If triangulation of qualitative data is not carried out, a risk of biasing or distorting the collected data exists, resulting in incorrect or incomplete information which might result in skewed understanding of the circumstances.

By collecting data from multiple sources or with multiple tools, we identify and address discrepancies or inconsistencies in the data. Triangulation often leads to additional questions or clarifications, which can be answered through follow-up interviews, discussions or exercises.

Core Values

We ensure excellent quality of the actionable reports which is intended to provide absolute and deta i l ed ins ight into the markets, products, competition and the overall perspective at an affordable price.

We ensure secrecy for all the research projects that we ca r ry ou t and ma i n t a i n confidentiality as per the code of conduct.

We guarantee authenticity, reliability and credibility of the data, facts and figures that we gather and d i ssem inate through primary and secondary research.

Aut

hent

icit

yQ

uality

Confidentiality

12Monitoring & Evaluation

13Monitoring & Evaluation

The Monitoring and Evaluation Team of Fiinovation has been involved in helping its partners to carry out eld research studies for monitoring, evaluating and assessing the impact of their proposed and ongoing/complete projects.

A few of our successfully completed assignments in the current year are mentioned below:

Fiinovation partnered with IL&FS Skills to carry out an integrated on-site assessment study for assessing the current status of the community in terms of their readiness and acceptability for skill based trainings. The study was carried out to explore the avenues that shall be available at the proposed site for Power Plant in Nana Layja (Mandvi), Gujarat. Based on the research ndings of Fiinovation, the project aims to develop skill, employment and entrepreneurial opportunities for the population of Mandvi Block in Kutch district with respect to the Nana Lajya Power Plant being put up by IL&FS Group. Fiinovation gathered adequate information through extensive research, both primary and secondary sources for the purpose of the study. The Need Assessment of the project was carried out for 3 months with assessment tools intrinsically focused on the need and acceptance of the community.

Fiinovation has also conducted a summative evaluation study of the skill training institute (Cairn Enterprise Center - CEC) of Cairn India Limited (CIL) at Barmer (Rajasthan). The study was carried out for the CSR project of CIL being implemented by IL&FS Skills with a view to evaluate the program performance and its effectiveness by collecting information that can be used for further evaluation. Fiinovation developed tailor-made evaluation parameters for assessing the outcomes of the skill training intervention. The framework used empirical evaluations aimed at elucidating and substantiating the linkages between the trainings provided and the intended or observed benets. The project required unbigoted opinion along with technical expertise which was provided by Fiinovation for assessment and recommendations for future course of action.

With its team of experts, Fiinovation has successfully conducted evaluation and assessment of Adult Literacy Centers (ALC) run by M/s JK Tyre & Industries Ltd. at Kankroli (Rajasthan), Morena( Madhya Pradesh) and Mysore( Karnataka). For this purpose, a detailed research was carried out to assess the efcacy and effectiveness of the ALCs.

Some of our

Initiatives

Afliations

Established in 2008, Fiinovation (Innovative Financial Advisors Pvt. Ltd.) has created a niche as a multi disciplinary research organization in Corporate Social Responsibility (CSR) and Sustainability domain. Headquartered in New Delhi, it has been forging partnerships between credible NGOs and corporations that are looking to implement and initiate community based interventions and initiatives. With a pool of over 4000 credible NGOs spread across the country, the motive has been to create awareness, inspire innovative thinking and action with sustainable development at its core. These NGOs are empanelled post a thorough and stringent due diligence check through secondary research and on ground evaluation.

Fiinovation is known in the development sector for enhancing quality across organizational value chain through its in-house Proposal Design & Research Laboratory for designing innovative CSR initiatives. Fiinovation also supports corporations through conducting need assessment, monitoring and evaluation as well as impact assessment of their programs.

The distinct practices of Fiinovation include CSR Portfolio Management (CPM), CSR-CSO Partnership, Initiative Design, Initiative Management, Monitoring & Evaluation, Impact Assessment and CSR audit across verticals of health, environment, education and livelihood, which contribute to INR 15 crores in the top line of nancial statement.

Fiinovation has been a knowledge partner with various industrial bodies such as Confederation of Indian Industries (CII) and Institute of Directors (IOD). With CII, the partnership emphasizes on research for promotion of afrmative action along with CSR and social enterprises. The organization has deliberated with various industrial leaders at numerous conferences conducted on corporate governance, environment management and CSR & sustainability during its path breaking partnership with IOD. Being a research organisation, Fiinovation also contributes to IOD's Master class for Directors that has a wide spectrum of subjects such as corporate governance, sustainability, HR practices, nance, corporate social responsibility, environment management and board effectiveness. Other than this, the organization has been closely associated with other forums such as NGO BOX and World CSR Congress.

Organization Brief

14Monitoring & Evaluation

Awards & Recognitions

Mr. S. Chakraborty along with Smt. Sheila Dixit at the 24th World Congress on Total Quality and Leadership

15

Fiinovation has been conferred with the “CSR Team of the Year” & “Caring Company Award” at the World CSR Day 2015. It was also the recipient of the 'Best Social Innovation: Product or Service' at the World CSR Day 2014. Apart from this, the company was bestowed with the "Manager of the Year" & "Best Enterprise of the Year - in the eld of Health, Education, Environment and Livelihood with a focus on CSR & Sustainability" Awards by the European Business Assembly at the Socrates Award Ceremony, Summit of Leaders in 2014 at Oxford Town Hall, United Kingdom.

Mr. Soumitro Chakraborty addressing the gathering at CII conference - Bridging the Gap: Fostering growth through Education & Entrepreneurship

Monitoring & Evaluation

Media Links

fiinovation

24/30, Ground Floor, Okhla Industrial Estate, Phase III, New Delhi - 110020, IndiaPhone: 011-42332200 | Fax : 011-42332205Email : info@inovation.co.in, | Website : www.inovation.co.in