ipa funds monitoring and evaluation 12-16 december 2011 1 bölgesel rekabet edebilirlik operasyonel...
TRANSCRIPT
IPA Funds Monitoring and Evaluation 12-16 December 2011
1
Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına Yönelik Teknik Yardım
Technical Assistance on Institutional Building for the Implementation of RCOP in Turkey
This project is co-financed by the European Union and the Republic of Turkey
Evaluation: Evaluation: questions and questions and methodsmethods
Ankara, 1Ankara, 166 DecemberDecember 2011 2011Laura Trofin (PhD): TAT Non-Key Expert , Monitoring and [email protected]
IPA Funds Monitoring and Evaluation 12-16 December 20112
This project is co-financed by the European Union and the Republic of Turkey
We willWe will : :
1. Discuss about evaluation questions: how to design good evaluation questions
2. Go through some standard evaluation methodologies
3. Determine the methodologies we would select to carry out the evaluations determined yesterday
IPA Funds Monitoring and Evaluation 12-16 December 20113
This project is co-financed by the European Union and the Republic of Turkey
Evaluation questionsEvaluation questions
• Descriptive -(what happened?)
• Causal questions -(how and to what extent is that which occurred attributable to the programme?)
• Normative questions - are the results and impacts satisfactory in relation to targets, goals, etc?
• Predictive questions - will the measures create negative/positive effects?)
• Critical questions – e.g. how can equal opportunity policies be better accepted by SMEs?
IPA Funds Monitoring and Evaluation 12-16 December 20114
This project is co-financed by the European Union and the Republic of Turkey
What makes a good evaluationWhat makes a good evaluation question? question?
• Good evaluation questions include the following characteristics:
They are specific They are measurable They are answerable
They are realistic and reasonable They are appropriate to the local needs They can contribute to knowledge development
IPA Funds Monitoring and Evaluation 12-16 December 20115
This project is co-financed by the European Union and the Republic of Turkey
Where do evaluation questions Where do evaluation questions generally come from?generally come from?
• The first major source of evaluation questions includes the evaluation and programme stakeholders
• Most stakeholders will provide some questions if you ask them.
• You (evaluation expert) will need to work with these stakeholders to develop their questions into reasonable, testable evaluation questions.
• The second major source of evaluation questions comes from the programmes evaluator’s independent analysis of the programme/ToR.
IPA Funds Monitoring and Evaluation 12-16 December 20116
This project is co-financed by the European Union and the Republic of Turkey
Methodologies for evaluation…Methodologies for evaluation…
• According to “The Evaluation of Socio-Economic Development – The GUIDE”…
• There are 32 methodologies..!
• Strategic environmental assessment, stakeholder consultation, social surveys, regression analysis, priority evaluation, participatory methods, observational techniques, multi-criteria analysis, input-output analysis, cost-benefit analysis…
IPA Funds Monitoring and Evaluation 12-16 December 20117
This project is co-financed by the European Union and the Republic of Turkey
The main methodologies are…The main methodologies are…
1. Studying monitoring data2. Benchmarking3. Interviews4. Beneficiary/stakeholder/social surveys5. Case studies6. Focus groups7. Stakeholder consultation8. Gender Impact Assessment9. Cost-Benefit Analysis10. Econometrics
IPA Funds Monitoring and Evaluation 12-16 December 20118
This project is co-financed by the European Union and the Republic of Turkey
1. Studying monitoring data1. Studying monitoring data
• VITAL to every evaluation• You cannot properly evaluate without monitoring
data!• Financial data and physical data (performance
indicators)• Watch out for “variance” in analysing financial data• Do not read too much into performance indicator
data if programme at an early stage, particularly impact.
IPA Funds Monitoring and Evaluation 12-16 December 20119
This project is co-financed by the European Union and the Republic of Turkey
Challenges with studyingChallenges with studying monitoring datamonitoring data
• The data itself may be incorrect!• The data may be missing!• The performance indicators may be inappropriate!
(often an evaluator’s conclusion)• Monitoring templates are often too clumsy, too big,
unnecessary information• Extracting relevant data may be difficult
IPA Funds Monitoring and Evaluation 12-16 December 201110
This project is co-financed by the European Union and the Republic of Turkey
2. Benchmarking2. Benchmarking
1. Select the parts of the programme that will be subject to benchmarking
2. Identify a partner for the benchmarking3. Observe the process in the partner’s organisation4. Analyse the differences in performance.• Can be external (e.g. schools) or internal (e.g. faculties
within a university)• In the context of the Structural Funds, examples might
include comparing the operation of similar programmes in different member states or comparisons of EU funded programmes and domestic initiatives within the same country.
IPA Funds Monitoring and Evaluation 12-16 December 201111
This project is co-financed by the European Union and the Republic of Turkey
Challenges with benchmarkingChallenges with benchmarking
• For example, schools – recruitment practices, socio-economic background of pupils, better teachers go to better schools, etc.
• You have a to find a similar intervention in a very similar setting!
IPA Funds Monitoring and Evaluation 12-16 December 201112
This project is co-financed by the European Union and the Republic of Turkey
Beneficiary/Stakeholder/Social Beneficiary/Stakeholder/Social surveyssurveys
The evaluator can survey:•The final beneficiaries (e.g. participants on a training course)•Relevant stakeholders (e.g. managers of the training course)•The general population (e.g. members of a target group that did not take part in a the training course)
IPA Funds Monitoring and Evaluation 12-16 December 201113
This project is co-financed by the European Union and the Republic of Turkey
Evaluators use questionnaires Evaluators use questionnaires when…when…
• they wish to generate new, quantitative data on the programme
• they wish to reach large number of stakeholders• the population to be observed is large and
homogeneous (e.g. employers in the food processing sector, etc.);
• they have a precise and clear idea of what they want to observe.
• Any other???
IPA Funds Monitoring and Evaluation 12-16 December 201114
This project is co-financed by the European Union and the Republic of Turkey
Steps in carrying out a surveySteps in carrying out a survey
• Step 1. Designing the questionnaire • Step 2. Sampling• Step 3. Pre-test or pilot• Step 4. Administration of the questionnaire• Step 5. Codifying the data• Step 6. Analysing the data• Step 7. Reporting the data
IPA Funds Monitoring and Evaluation 12-16 December 201115
This project is co-financed by the European Union and the Republic of Turkey
Do’s and don’ts tips for good Do’s and don’ts tips for good surveys…surveys…
1. DO ask questions that can be quantified!!2. DO ask questions that are clear3. DO direct the person easily through the questionnaire (e.g. “If yes,
go to question 9, if no, go straight to question 10”)4. DON’T ask open-ended questions (e.g. “Tell us what you think
about…..”)5. DON’T ask a) too many questions on one page, b) questions that
are unnecessary, c) questions of an intrusive nature (e.g. “are you a Roma?”)
6. DON’T make the questionnaire too long7. DON’T ask questions of a “leading” nature (e.g. “Would you agree
that the programme was a success?”)
IPA Funds Monitoring and Evaluation 12-16 December 201116
This project is co-financed by the European Union and the Republic of Turkey
Challenges/problems with Challenges/problems with surveyssurveys
• Surveys• They are ONE way to gather data, but should never
be the exclusive method• Need to standardise answers leads to simplified
effects?• Telephone, face-to-face surveys very expensive• Postal surveys poor response rates• Need big enough samples!• If you have 10 categories of beneficiaries, do you
survey them all?!
IPA Funds Monitoring and Evaluation 12-16 December 201117
This project is co-financed by the European Union and the Republic of Turkey
Case StudiesCase Studies
How to carry out a case-study evaluation1. Select the cases2. Collect and process the information3. Report on the case4. Summarise (and extrapolate) the cases, where
possible
IPA Funds Monitoring and Evaluation 12-16 December 201118
This project is co-financed by the European Union and the Republic of Turkey
Case studies (contd.)Case studies (contd.)
• Of these the most fundamental is selecting the cases. Use three criteria1.Convenience/access2.Information you are trying to get from the cases 3.The extent to which you can extrapolate to provide wider insights beyond the particular case in question. •Issues to consider in selecting cases are:
1.What will count as a ‘case’? How many cases will be chosen?2.What units of analysis will be included within the case?3.How must the data be organised to allow meaningful comparisons to be made?4.What kind of generalisation is possible?
IPA Funds Monitoring and Evaluation 12-16 December 201119
This project is co-financed by the European Union and the Republic of Turkey
Challenges with case studiesChallenges with case studies
• Can you extrapolate?
• Expensive to conduct a lot of case studies
• Credibility problems with inferring causality
IPA Funds Monitoring and Evaluation 12-16 December 201120
This project is co-financed by the European Union and the Republic of Turkey
Focus groupsFocus groups
• A homogenous group of people who meet. The evaluator supplies the topics for and “chairs” the discussion.
• A variation is the workshop, a larger group with a more structured agenda
• When is it used?• analysing issues with very different opinions or complex
issues!• testing an innovative measure (ex ante evaluation)• identifying problems and needs and the improvements
required during the implementation of a programme.
IPA Funds Monitoring and Evaluation 12-16 December 201121
This project is co-financed by the European Union and the Republic of Turkey
Focus groups (continued)Focus groups (continued)
• Also good for validating quantitative data findings
• NB - Focus groups most suited to evaluations of programmes with a large “people” context (i.e. not good for transport or environmental infrastructure, good for HRD, or services-based programmes!)
IPA Funds Monitoring and Evaluation 12-16 December 201122
This project is co-financed by the European Union and the Republic of Turkey
Focus groups – how do Focus groups – how do you do it?you do it?
• Step 1 – Selection of participants• homogenous but diverse!• you may have to offer them something!
• Step 2 – Choice and training of facilitators• Don’t assume that you can do this yourself! Skills are
needed!• Don’t forget someone to take notes!
• Step 3 – Defining the interview topics• Don’t look to cover too much!• Start with the easy questions to promote discussion!
IPA Funds Monitoring and Evaluation 12-16 December 201123
This project is co-financed by the European Union and the Republic of Turkey
Focus groups – how do you Focus groups – how do you do it? (continued)do it? (continued)
• Step 4 – The discussion• “a combination of assertiveness and tact may be required,”
because…• Watch out for bullies!• Try to involve everyone!
• Step 5 – Analysis and reporting of results• compare the information given!• look for shared and divergent opinions!• code the data so as to organise the results in relation to
the objectives of the evaluation• include some direct quotes in your report!
IPA Funds Monitoring and Evaluation 12-16 December 201124
This project is co-financed by the European Union and the Republic of Turkey
Challenges with focusChallenges with focus groupsgroups
• “Hawthorne” effect
• Contrasting experiences of a minority of the participants
• Bullies!
• Straying outside the terms of the discussion
IPA Funds Monitoring and Evaluation 12-16 December 201125
This project is co-financed by the European Union and the Republic of Turkey
Stakeholder consultationStakeholder consultation
• Purpose – Ensuring that the preferences, interests and perspectives of different stakeholders are considered in the evaluation
• Why?• to allow stakeholders feel they have “ownership” of
the evaluation• to increase the likelihood that evaluation findings will
be used and acted upon• EU requirement!
IPA Funds Monitoring and Evaluation 12-16 December 201126
This project is co-financed by the European Union and the Republic of Turkey
Stakeholder consultationStakeholder consultation (continued)(continued)
• Used at 3 stages:1. Identifying the evaluation priorities
• identifying common ground and differences in what stakeholders’ want from the evaluation
• Arriving at a set of priorities, preferably through consensus.
2. Gathering data on programme performance3. Discussing possible conclusions
IPA Funds Monitoring and Evaluation 12-16 December 201127
This project is co-financed by the European Union and the Republic of Turkey
Stakeholder consultation – Stakeholder consultation – how do you do it?how do you do it?
• Step 1 – Identify the stakeholders• The Steering Committee (or commissioner) has to assist
with this!• Don’t assume that programme managers represent’ the
interests of the beneficiaries!• Step 2 – Structuring consultation with stakeholders
• How you achieve this will depend on:• how important is stakeholder consultation to the main
purpose of the evaluation?• the time and budget for the evaluation, • The number of stakeholders
• Key point - you can use other methodologies to consult with stakeholders! (e.g. focus groups, interviews, questionnaires, etc.)
IPA Funds Monitoring and Evaluation 12-16 December 201128
This project is co-financed by the European Union and the Republic of Turkey
How do you do it?How do you do it? (continued)(continued)
• Step 3 – Analysis of the data• This will not be a separate section of the report!
• Step 4 – Establishing mechanisms for ongoing consultation
• Give periodic feedback (through presentations to the Steering Committee, etc.)
IPA Funds Monitoring and Evaluation 12-16 December 201129
This project is co-financed by the European Union and the Republic of Turkey
Challenges with stakeholderChallenges with stakeholder consultationsconsultations
• The more stakeholders consulted, the more difficult the consultation process will be to manage.
• There is a limit to how many stakeholders can be consulted – which stakeholders to involve?
• Do the views of a participant equal the views of the programme manager?
IPA Funds Monitoring and Evaluation 12-16 December 201130
This project is co-financed by the European Union and the Republic of Turkey
Individual interviewsIndividual interviews – – some points…some points…
• Don’t assume people have all day! • Tell people how long you think it will last! • Explain their role in what you are trying to achieve!• Focus on a number of key issues (not more than 5 in 1
hour)• Let them read what you write!• Permission to use material from the interview must be
granted• Give them the overall results of the process at a later
date!
IPA Funds Monitoring and Evaluation 12-16 December 201131
This project is co-financed by the European Union and the Republic of Turkey
Strengths and weaknesses Strengths and weaknesses of interviews?of interviews?
• People can speak freely, no bullies!• Real insight into the issues• Allows for direct, anonymous quotes• Sensitive issues can be covered• BUT…if interviewing participants, can’t reach the
numbers that a survey can• Relies on the integrity of the interviewer• Interviewer shouldn’t be the interviewee!
IPA Funds Monitoring and Evaluation 12-16 December 201132
This project is co-financed by the European Union and the Republic of Turkey
Gender Impact AssessmentGender Impact Assessment
• estimates the different effects (positive, negative or neutral) of any policy or activity on gender
• allows changes to be proposed that will reduce any negative gender effects
IPA Funds Monitoring and Evaluation 12-16 December 201133
This project is co-financed by the European Union and the Republic of Turkey
GIA – how do you do it?GIA – how do you do it?
• Step 1 – Analysis of the present situation• Step 2 – Analysis of future trends• Step 3 – Determining priority• Step 4 – Assessing general impact of the Programme• Step 5 – Assessing specific direct impact of the
Programme• Step 6 – Assessing specific indirect impact of the
Programme• Step 7 – Defining further development• Step 8 – Defining a set of gender sensitive evaluation
indicators
IPA Funds Monitoring and Evaluation 12-16 December 201134
This project is co-financed by the European Union and the Republic of Turkey
Challenges with GIAChallenges with GIA
• If women benefit equally, but the overall number of people benefiting decreases, is that positive or negative??
• How do programmes that are targeted on men only or women only score under GIA?
• Is it okay to fund programmes that accept inequality?
IPA Funds Monitoring and Evaluation 12-16 December 201135
This project is co-financed by the European Union and the Republic of Turkey
Group exercise 4 Group exercise 4
For the types of evaluations developed under GE 3 define evaluation questions and choose the most appropriate methodological tools to carry out your evaluation
THINK ABOUT:• Purpose(s) and type of evaluation• Principles of good evaluation questions• Triangulation principle• Type and depth of information you want to collect• Strengths and challenges of each tool