1 futurreg evaluation objectives and methodology 3nd steering committee meeting malta, 28/6/2006
TRANSCRIPT
![Page 1: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/1.jpg)
1
FUTURREG Evaluation
Objectives and methodology
3nd Steering Committee Meeting
Malta, 28/6/2006
![Page 2: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/2.jpg)
2
Evaluation concepts Objectives of the Futurreg On-
going Evaluation Theory Background Evaluation Methodology Evaluation activities
![Page 3: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/3.jpg)
3
The evaluation is a systematic and objective process which values the relevance, efficiency and effectiveness of the policy, programmes and projects in relation with the established objectives (Fahernkrog et al 2002).
The evaluation has to carry out five levels of assessment:
Relevance and Coherence
Cost and Benefit, Efficiency
Effectiveness
Utility
Sustainability.
![Page 4: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/4.jpg)
4
Internal Coherence
Relevance Efficiency
Effectiveness
Utility / Sustainability
Actual/Future Needs
ObjectivesGeneral/Specific
Impact
Results /Benefit
Input/Cost
Activity Outputs
Relation of the evaluation process
![Page 5: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/5.jpg)
5
Why to Evaluate? Control and Accountability – what have
been done with all the money?
Knowledge about results – what are the effects of the activities and measures? Did we really achieve what we wanted to achieve?
Learning approach – what went well, which things went wrong, why did we (not) achieve the objectives? How can we improve working in the future?
Building link between involved actors
![Page 6: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/6.jpg)
6
ISSUES FOR FURTHER DEBATEISSUES FOR FURTHER DEBATE
•Institutional Learning in project and programme management is necessary to exploit fully the potential of evaluations.
•Training and awareness-raising for both, project and programme managers and evaluators.
•Integrate Evaluation into the project-cycle, internalise the monitoring and evaluation function.
•Free evaluation from the stigma of being an instrument for control and punishment.
![Page 7: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/7.jpg)
7
Learning approach
Taking desitions & policy makers
building linksRegional foresight capabilities
Triangle of Futurreg evaluation functions
Suggestion for
improvementImplementation
and impact
Goal control and follow-up
![Page 8: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/8.jpg)
8
to provide information and recommendations in order to improve the Futurreg project and to contribute to the development of a regional foresight culture.
Evaluation in conformity with the provision under Interreg IIIC normative
To evaluate the relevance and coherence of Futurreg project.
To evaluate the utility and sustainability of Futurreg as a project.
To derive practical lessons and realistic recommendations which are important for the dinamization of foresight capabilities at regional level.
![Page 9: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/9.jpg)
9
The Foresight can be defined as a systematic, participatory, future intelligence gathering and medium-to-long-term vision-building process aimed at present day decisions and mobilising joint actions.
(FOREN, EU, 2002)
![Page 10: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/10.jpg)
10
Complex Subject and Project, wide variety of stakeholders.
Since the project is still running, it is often not possible to identify or examine final results or impacts.
Limited visibility and sometimes indirect effects of Futurreg actions on final beneficiaries (SMEs, enterprises), leds to difficulties integrating their perspectives in the analysis to the same extent as the points of view of intermediaries.
![Page 11: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/11.jpg)
11
Main Objectives Basic sciences Key Technologies Innovation
Social
Industrial
Military defense
Caracostas y Muldur (1998)
1945 1970 2000 2020
Evolution of Innovation policy
Futures&
Foresight
![Page 12: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/12.jpg)
12
Policy makers’
environment
Valovirta ja Hjelt (2005)
Evolution and foresight: interrelations
- Impact assesment- technology assesment
![Page 13: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/13.jpg)
13
Policy makers’
environment
Valovirta ja Hjelt (2005)
Evolution and foresight: interrelations
![Page 14: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/14.jpg)
14
• Development of Evaluation Methodology• Evaluation criteria for
applications/implementation projects• Development of indicators to evaluate
each component of the project• Development of a questionnaire for the
partners• On-going evaluation• Questionnaire of satisfaction for the actual
users of the application• Interviews with the regional actors benefit
of the project• Ex-post evaluation report
![Page 15: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/15.jpg)
15
• Methodological tools for evaluation:
– Desk research (background literature, work programmes, project implementation reports, former foresight projects evaluations, studies, statistics, etc.)
– Work research (Interviews with partnership and select stakeholders at regional level, On-line Questionnaire).
– Permanent relationship with partner (meeting, e-mail, phone). Contrast dynamics and consent.
![Page 16: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/16.jpg)
16
• Evaluation Framework of the project:
Evaluation Topics
Evaluation Questions
Indicators Data Sources
Data gathering (Methods)
Main Additional
Relevance
Effectiveness
Efficiency
Effects and Impacts
![Page 17: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/17.jpg)
17
CONCLUSIONCONCLUSION
•Evaluation will give new insights on how to improve.
• Evaluation will give you information on which measures work and which don’t.
•Evaluation gives the public institutions a proof of what you have done and achieved, and legitimise what you do at regional level.
![Page 18: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/18.jpg)
18
The contribution of Futurreg Evaluation:
Ensures the RELEVANCE, FEASIBILITY and SUSTAINABILITY of a project
Facilitates a dialogue / OWNERSHIP Ensures that fundamental questions are
asked and weaknesses are analysed Defines the key elements & the settings of a
project Identifies measurements/ indicators of the
projects achievements Systematic common sense (helping to adapt
expectations to planned expenditure to time frames)
![Page 19: 1 FUTURREG Evaluation Objectives and methodology 3nd Steering Committee Meeting Malta, 28/6/2006](https://reader035.vdocuments.site/reader035/viewer/2022062519/56649e855503460f94b88623/html5/thumbnails/19.jpg)
19
Structure of the Report
• Executive Summary
• Introduction
• Evaluation Methodology
• Analysis of:– Futurreg deliveries– Futures toolkit– Regional foresight capabilities impact
• Conclusions and Recommendations on the Overall Futurreg