How to Evaluate Extension Programmes?

P.V.K.SasidharDirector

School of Extension and Development Studies

Indira Gandhi National Open UniversityNew Delhi-110068 ( www.ignou.ac.in)

Outline (30 Minutes)

• Card Exercise on ‘Current Practices in Programme Evaluation’

• Meaning and Need for Evaluation

• Appraisal, Monitoring, Evaluation and Impact Assessment

• Evaluation Types and Designs

• How to Conduct Programme Evaluation?

(Case Study Example with Bennett’s Evaluation Hierarchy)

Card Exercise on

‘Current Practices in Programme Evaluation’

1. Have you conducted any evaluation study? If Yes,

2. Evaluation tool(s) / design(s) used ?

What is Evaluation ?

• Assessing as systematically & objectively as possible

an ongoing or completed programme / project about

their :

– Relevance

– Effectiveness

– Efficiency

– Impact, and

– Sustainability

• Stated Criteria • Point of Time

Need for Evaluation

• Extension Professionals' Responsibility -

Implementation of Programme Planning Cycle with:

– Efficiency

– Accountability

– Resource Allocation

• Funding agencies want to know :

– What did they do with the money?

– Are the programmes effective and productive ?

– Why should we continue to fund extension

programmes / projects?

– How will you improve or terminate ineffective programmes

/ projects ?

• Programme evaluation provides answers to these

questions & provides empirical evidence.

Programme Formulation

Programme Execution

Programme Evaluation

Appraisal – Monitoring - Evaluation - Impact Assessment

Appraisal - Critical examination of a programme proposal, normally before implementation

& funding w.r.t. economic viability ,technical feasibility, and / social desirability.

Monitoring EvaluationContinuous: Starts and ends with a programme. One shot operation: At a point of time (usually at

completion or mid way of programme)Required for immediate use and mid-course correction

Used for future planning/ replication/ expansion

Done by implementing personnel Usually by outside agencyQuick but covers all units In-depth; covers a sampleCorrecting / Managing Learning processSymptomatic, early warning system Diagnostic

AME focus on the processes and direct results of a project or programme.

Building on these, ‘Impact Assessment’ focuses on longer-term and wider-ranging changes

beyond the immediate results of the project or programme.

Types of Evaluation

Evaluation by focus• Formative Evaluation: During programmes development stages (Process

evaluation: /Mid term appraisals)

• Summative Evaluation: Once the programme achieves a stable state of operation

(Outcome evaluation / Ex-post evaluation)

Evaluation by agency – Internal / External

Evaluation by stage– Ongoing : During the project

– Terminal: : After completion of project immediately

– Ex-Post : After a time from completion of project

Longitudinal evaluation -repeat evaluation to study the

sustainability of results/outcomes

Evaluation Designs

1. "Pre - Project" Vs. "Post-Project" EvaluationSurvey on present situation and situation prior to project (suffers from

'memory or recall bias' of the respondents).

‘Baseline' or ‘Bench Mark’ study and ‘Post-Project’ study.

Unfortunately in most projects, baseline studies are not undertaken

or when available it is inadequate.

2. With-Without Project• Beneficiary - non beneficiary (Control Group)

Combination of both - most appropriate

How to Conduct Programme Evaluation ?

1. Writing evaluation proposal with : • Need for evaluation - economic /technical / social • Research questions • Objectives• Evaluation methodology & tool

o Bennett’s Hierarchy o LFA

• Budget2. Designing evaluation survey instrument - Ethics Committee (IRB – MSU)3. SPSS coding, pre-testing, recoding SPSS & data collection3. Data entry & analysis4. Evaluation report writing & communicating findings - Short reports / Longer

Inputs , Outputs and Outcomes

Logical Framework Approach

INPUTS OUTPUTS OUTCOMES

Program investments Activities Participation Short Medium Long-term

OUTCOMES What results for individuals, families, communities..…

SHORTLearning

Changes in • KASA• Opinion• Motivation• Behavioral intent

MEDIUMAction

Changes in •Behavior •Decision-making•Policies•Social action

LONG-TERMConditions

Changes in ConditionsSocial (well-being)Health, EconomicCivic , Environmental

Bennett’s Hierarchy of Evaluation

Provides a process by examining the chain of means (what you do) and ends (the result of your actions) through seven levels

Example : Evaluation of Training Workshop on ‘Good Practices in Extension Research and Evaluation’

Evaluation Hierarchy Measurement

Level 7 (End results)

SWOT analysis & 2 case studies on (i) Livelihood security to Groundnut farmers through assured seed supply (ii) Reviving Rice cultivation through resource efficient direct seeding technology.

Level 6 (Practice change)

Adoption behaviour of beneficiary farmers on 11 management practices for Paddy and 14 for Groundnut.

Level 5 (KASA change)

Knowledge, attitude and adoption of improved practices for Paddy and Groundnut among beneficiaries and non-beneficiaries.

Level 4 (Reactions)

Perception on training imparted to 260 Adarsha Rythus

Level 3 (Involvement)

Involvement of different stakeholders.

Level 2 (Activities)

Activities performed by KVK (OFTs, FLDs and Training programmes).

Level 1 (Inputs)

Resources used for the KVK activities.

Bennett's Hierarchy – KVK Evaluation

Integrated Contract Broiler Farming: An Evaluation Case Study in India

Research Questions1. Do contract and non-contract farmers incur significantly

different production and marketing costs and earn different marketing margins?

2. Does the provision of EASs by private CBF companies enable contract farmers to make better profits than non-contract farmers?

3. Have assured markets, competitive price and guarantee against risk resulted in successful value chain development through CBF?

4. Are the value chain developments and provision of EASs by private CBF companies really win-win situations for both integrators and farmers, or are they socially acceptable ways of exploiting the farmers?

Evaluation hierarchy

Measurement Indicators Measurement

Level 7 : End results

Socio-economic changes and impacts

SWOT parameters FGD on: selection of contract farmers; terms and conditions applicable in

CBF

Open-ended

Level 6 : Practice change

Technical advices adoption

Non-adoption, discontinuation, partial adoption and full adoption of technical advices

Scale on four- point continuum

Level 5: KASA Farmers’

perceptions Perceptions on inputs (chicks, feed, medicines and EAS) and outputs

(broiler birds, manure value and payment system) Scale on five- point continuum

Level 4 : Reactions

Farmers’ feedback Factors of motivation to do CBF and NCBF Reasons to change integrator(s) or input providers in the past two years

Open-ended

Level 3 : Outputs Technical and economic performance

Broiler birds (flock size, mortality number, birds sold, sale age, sales rate and birds lifting days)

Productivity (mortality percentage, birds sold, feed consumption and body weight)

Efficiency (FCR, sale age, weight gain/day) Economics of inputs and outputs EAS (frequency of information from various sources)

Technical and economic performance index

Level 2 : Activities Activities in CBF and NCBF

Physical and human resource activities in CBF and NCBF Survey

Level 1: Inputs Investments and Demographics

Fixed and variable costs Age, gender, education, social category, family and size, poultry occupation

and experience

Survey

Bennett's Hierarchy Applied in Evaluation

Bennett’s Hierarchy - Operationalization and

Measurement of Variables

Level 1: Inputs• Fixed and Variable Costs

• Demographic characteristics

Level 2: Activities

• Physical and human resource activities– No. of broiler poultry sheds

– Batches of poultry housed yearly

– Total labor utilized per batch (hired and family labor)

Bennett’s Hierarchy - Operationalization and

Measurement of Variables

Level 3: Outputs (Output details per batch)

• Broiler birds: Chicks housed/flock size (numbers), mortality (numbers), birds sold (numbers),

lifting age (days), sales rate (rupees (Rs) and birds lifting days (number).

• Productivity: Mortality of the birds (percent), birds sold (kg), feed consumption (kg) and body

weight (kg).

• Efficiency: FCR, mean age of sale (days) and day gain (g/day).

• Economics:

CBF - Input costs: Costs of labor, bedding material, electricity and miscellaneous expenses.

CBF - Returns on outputs : Rearing charge (RC), sale of manure and feed bags.

NCBF -Input costs: Costs of chicks, feed, medicines, labor, bedding material, electricity, EAS and

miscellaneous expenses.

NCBF – Returns on outputs: Returns through sale of birds, manure and feed bags.

– Economics of inputs and outputs were worked out per kg live chicken & per bird produced.

• EAS: Frequency with which farmers get information from various sources -- integrator, research

station, public extension staff member, public veterinary doctor, private veterinary doctor and

private poultry consultant.

Bennett’s Hierarchy - Operationalization and

Measurement of Variables

Level 4: Farmers’ Reactions

• Factors of motivation to do CBF/ NCBF

• Changes in integrator(s)/input provider(s) in the past

two years

• Reasons for terminating the contract with the

integrator or changing the input provider.

Bennett’s Hierarchy - Operationalization and

Measurement of Variables

Level 5: KASAFarmers’ perceptions on CBF and NCBF ( Inputs) :

• Supply of chicks -- cost, body weight, timely supply, strain, flock size per batch, number of batches per

year, growth rate and gap between two batches.

• Supply of feed – cost, quality, quantity and FCR.

• Supply of medicines -- cost, quality and quantity.

• Provision of EAS -- applicability of EAS, understandability of message (treatment), frequency, timeliness,

relevance, adequacy, usefulness and technical know-how of EAS provider.

Farmers’ Perceptions on CBF and NCBF (Outputs):

• Broiler birds -- number of birds produced/sold, and live weight at the time of sale.

• Manure -- quantity produced, method of disposal and economic benefit.

• Payment for sale of birds – remuneration, regularity and pricing method.

Farmers’ perceptions on the intention of EAS:

• Information only

• Information + knowledge

• Information + knowledge + skill

• Information + knowledge + skill + attitude.

Bennett’s Hierarchy - Operationalization and

Measurement of Variables

Level 6: Practice Change

Adoption of technical advice : Chicks ; Housing; Feeding practices ; Medications

Level 7: Further Changes Required

SWOT analysis

Focus Group Discussion:

– Active Participants : 2 integrators & 14 farmers

– Passive Participants :3 field vets, 4 poultry researchers & 3 extension personnel.

Key FGD questions on :

• Method of identifying potential contract farmers.

• Details to be submitted by farmers to company for agreement – personal, business

and farm particulars.

• Farm appraisal criteria by contract company.

• Terms and conditions applicable in CBF – SPC, RC, rate incentives and penalties.

Responses of FGD were cross-checked with the farm appraisal forms submitted by farmers to

the integrators and personal observations/interactions during data collection.

Evaluation •Neglected & mistakenly seen as a 'fault finding' mechanism. •Learning process & Past experiences guide us to a better future.

Conclusion

Evaluation Proposal : http://www.meas-extension.org/meas-offers/case-studies/contract-farming

Evaluation Report with Survey Instrument :http://dev.meas.illinois.edu/wp-content/uploads/2015/04/MEAS-EVAL-2015-Broiler-India-long-Sasidhar-Suvedi-June-2015.pdf

Thank you pvksasidhar@ignou.ac.in