Designing Influential EvaluationsSession 7Commissioning evaluations

Uganda Evaluation Week - Pre-Conference Workshop19th and 20th May 2014

Framework for commissioning evaluationsPlanning and scoping for evaluation, this

session explores a framework for working towards a TOR:

Evaluability assessment Working to timeframes Looking at budgets Deciding who the evaluators will be Tendering processes

2

Evaluability assessment Is a systematic process

◦ describing the structure of a programme◦ analyzing its plausibility and feasibility for

achieving objectives is there a theory of change; could one be

retro-fitted?◦ assessing suitability for in-depth evaluation◦ appropriateness to programme managers,

policymakers and programme implementers

3

Evaluation timeframesAdequate time for commissioning

and doing evaluationConsultations with all parties to

ensure TOR is realisticConsider length of programme &

time needed to conduct formative as well as summative evaluation

Be realistic and open to change4

Evaluation budgetValue for money a key concernUnderfunding as wasteful as over-

fundingBalance between cost and qualityQuality ultimately more importantBut also relevance for purpose Make sure all aspects adequately funded

including consultation with stakeholders, reporting and dissemination

Ensure evaluation design appropriate to budget as well as aims of programme 5

6

Open and closed tenders

Advantages a range of expertise keeps cost competitive stimulates providers brings in the new reduces complacency

But... inappropriate suppliers possible selling rather

than evaluation skills time and cost

consuming

Advantages fewer suppliers time to fine tune brief

with selected contractors

commissioners build on previous knowledge

less time & money on selection of evaluation team

But… lack of dialogue

w/potential others does not encourage

new ideas/evaluator

Open Closed

7

Talking money: revealing the budget

tenderers put in a price just below the budget

collusion of tenderers could lead to inflation of the budget requirement

commissioners must have clear evaluation design in mind to realistically assess proposals

no evidence that any of these happen

better to make informed decisions

time-wasting to ‘guess’

budget-ceilings facilitate quality proposals

a range of budgets allows for creative proposals

Against For

Principles of good commissioningClarity of purpose – in the brief and expectations;

aims, objectives and outputs Accessibility - of language, range of skills, the

tender process and reporting Participation - collaboration between those

engaged in research enterprise Flexibility - in design (scale of proposal matched

to funding), in methodology, in process (to adjust to field/policy changes) and scope of study

Integrity - of relationships, trust, honesty empathy, valuing different perspectives.

Transparency in open tendering, declared budgets or ball park figures, management and reporting procedures

8

The terms of reference Background

◦ Linkages and synergies

◦ Approach paper

Purpose, objectives, scope Existing sources of information Methodology

◦ Level of detail and guidance?

Skills and qualifications Logistics and procedures Outputs

◦ Communication plan

◦ Study report

Reporting & contracting arrangements Budget Annexes

◦ Steering groups/ advisors; ethics; standards; arrangements for QA

9

TaskDraft the following elements of a

terms of reference for the final evaluation of a project

Purpose, objectives, scope Approach & methodology to be used Skills and qualifications of the team Outputs

Communication plan

Programme 14 Information and National GuidanceObjectives: To regulate, coordinate and disseminate government

policies, programmes and projects to the public for economic and social accountability.

To develop and nurture a National Value system based on a shared National Vision, mindset change, beliefs and perceptions as well as popularising government policies and programmes to create nationwide awareness for socio-economic transformation.

Outputs: Meetings with various stakeholders in the media industry National and International events publicised; Public officials sensitized on ATIA; Leaders and members of the public sensate In and out of school youths mobilised and sensitized; The Barazas initiative supported Carry out cadreship training courses; Identify land for the regional training institutes. Field research, Pre-sensitization surveys and evaluations on

national activities carried out.11

Project 1078 Karamoja Integrated Development Programme (KIDP)

Objectives: To consolidate peace and security for the people of

Karamoja and strengthen coordination and harmonization of interventions both at national and regional levels.

 Outputs: Housing facilities for people in Karamoja improved. Food and nutrition security for the poor and vulnerable

households improved. Crop and livestock production and productivity

increased. The quality of education in Karamoja improved.

Clean and safe water for human consumption and for production facilities for multipurpose use provided.

TASO Support to OVCThis project is trying out new ways of providing support to OVCObjectives: Children continue in school through to complete secondary as

a result of receiving appropriate support Children who drop out or do not progress at school qualify

through apprentice training Individual OVC and their families become self-sustaining after

eduction

Activities: Selection of the most needy orphans and vulnerable children to

participate in the programme Training given to parents, guardians and teachers in working

with and caring for OVC School equipment and medical supplies provided for the

children Counselling provided for OVC at school

NAADS25-year programme, with an initial phase of 7 years; first 2 years for testing of programme approaches and concepts leading to a refinement of the design.Objectives: To promote market oriented/commercial farming (farming as a business) To empower subsistence farmers to access private extension services,

technologies and market information To create options for financing and the delivery of appropriate advisory

and technical services for different types of farmers To shift from public to private delivery of advisory services in the first five

years of implementation To develop private sector capacity and professional capability to supply

agricultural advisory servicesActivities: Deepening decentralization Market oriented farming Managing natural resource productivity Harmonization of public extension oriented programmes Development of Public Private Partnerships(PPPs) for agribusiness and

market linkageTarget groups: The principle beneficiaries are the poor farmers and in particular women,

youth and people with disabilities.

SHG/MFI ProgrammeObjectives: Job creation and income generation for women

formed in self-help groupsEmpowerment of women Improvements in family welfare

Activities:Formation of self help groupsGroups savings and loan schemesAccess to creditBusiness trainingValue chain analysis, market linkages,

vocational training

END