1 international conference evaluation: evidence-based tools for decision-making future cohesion...
TRANSCRIPT
1
International Conference
Evaluation: Evidence-based Tools for Decision-making
Future Cohesion Policy: Implications for Monitoring and Evaluation
Budapest May 24-25, 2011
Dr Jim FitzpatrickManaging DirectorFitzpatrick Associates
122 Ranelagh VillageDublin 6Tel: +353 1 6280084Fax: + 353 1 6219771
2
issues in future Cohesion Policy affecting Monitoring and
Evaluation (M+E)
emerging parameters of 2014-20 period
the “new” Logical Framework
challenges in implementing new approaches
some practical suggestions
TOPICS
3
EU STRUCTURAL AND COHESION FUNDING
20 years + of multi-annual programmes
M+E central to the programming package
Objectives, priorities, targets/indicators
Agreed strategies, programmes
Regular monitoring
Formal evaluation
Gradual development of M+E practice
2014-20 will involve significant changes
1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
CEE entry
4
2014-20: INCREASING THE EFFECTIVENESS OF COHESION POLICY
the main focus of evaluations “tended to be on processes and financial implementation rather than on the actual results of programmes”, (Ex Post Evaluation
of Cohesion Policy Programme 2000-06, Synthesis Report April 2010)
a Member State-Commission National Strategic contract where Member States (or Regions) “commit themselves to quantified and verifiable objectives”. (An
Agenda for a Reformed Cohesion Policy , Barca Report, April 2009)
“specific binding conditionality in the areas directly linked to cohesion policy would be agreed with each Member State and/or Region – depending on the institutional context…” (Investing In Europe’s Future: Fifth Report on Economic, Social and
Territorial Cohesion, Nov. 2010)
Work of High Level Group on Future Cohesion Policy, Conditionality Task Force (Feb.-April 2011), Evaluation Unit/Network, Hungarian Presidency High Level Conference (March 31/April 1).
5
A CONFLUENCE of Related STRANDS
Monitoring And
Evaluation
ThematicConcentration
BindingContracts
Conditionality
Better Information
Better Evaluation/
CFA
Performance/ Results
measuring outcomes
6
provides goals
provides themes
regulatory requirements
evaluation will inform content/progress
ex ante, ongoing, ex post
EMERGING NEW ARCHITECTURE 2014-20
* Possibly for Cohesion Fund, ERDF, ESF, EAFRD, EEF
Europe 2020 Goals
Common Strategic Framework (CSF) ?*
Regulations
Partnership and Investment Contracts*
Operational Programmes
Themes/Axes (from menu)
7
STRENGTHENED CONDITIONALITY?
already is various conditionality: regulatory (inc. M+E); strategic. Infrastructure planning; institutional
types of conditionality:
“macroeconomic” – part of Stability/Growth Pact
“ex ante” – preconditions (see existing categories)
“structural reform” – structural/admin reform milestones
“performance” – Programme/EC objectives
incorporated in Partnership Contracts?
possible financial incentives/sanctions?
if implemented, could have implications for Monitoring/Evaluation
8
Other Factors
Monitoring and Evaluation
ProgrammingStrategy
Needs ThematicObjective
IntendedResult
Contribution - ImpactPolicy AllocatedINPUTS
TargetedOUTPUTS
ActualINPUTS
AchievedOUTPUTS
ActualResult
NEW LOGICAL FRAMEWORK
outcomes monitored/impacts evaluated
inputs/outputs “contribute” to outcomes
outcome indicators should be “responsive” to intervention
Source: V. Gaffey, Acting Director, Policy Development, DG Regio, Intervention Evaluation Conference, Budapest, May 24-25, 2011.
9
RE-DEFINING THE LOGIC
Source: based on DG Regio, Concepts and Ideas: Monitoring and Evaluation in Practice of European Cohesion Policy 2014+, Draft, 30 March 2011
Old: Inputs Outputs Results(Short/medium)
IMPACTS(Long term)
Monitoring Evaluation
Monitoring
New: Inputs Outputs Results/ outcomes
Evaluation
Impact
10
CRITERIA FOR A GOOD OUTCOME INDICATOR
Reasonable: capturing the essence of an outcome according to a reasonable argument about which
features of the outcome they can and cannot represent
Robust: reliable, statistically and analytically validated, and, as far as practicable, complying with
internationally recognised standards and methodologies;
Responsive to policy: linked in as direct way as possible to the policy interventions for whose
assessment they are used, while not being subject to manipulation;
Normative: having a clear and accepted normative interpretation (i.e. there must be agreement that a
movement in a particular direction or within a certain range is a favourable or an unfavourable result);
Feasible: built, as far as practicable, on available underlying data, their measurement not imposing too
large a burden on Member States, on enterprises, nor on the citizens;
Debatable: timely available to a wide public, with room being built for public debate and for their own
revision when needed and motivated.
Source: F. Barca, P. McCann, Outcome Indicators and Targets – Towards a Performance Oriented EC Cohesion Policy, High-
level Group on Future Cohesion Policy, Meeting No. 8, 15 Feb. 2011
RRR-NFD instead of SMART! The challenge is in the “Third R”
12
CHALLENGES REGARDING OUTCOME/RESULT INDICATORS
Conceptual: the causation problem remains? a lot of judgement?
Communication: can we be confident this will address the issue? trade-off between communication and robustness
Capacity: who is going to define the indicators? A specialist task. Will need forensic
precision
Time-lags: when will attributable outcomes arise?
Conditionality: M+E insufficiently robust to support binding conditionality
13
SOME SUGGESTIONS FOR COMMISSION/MEMBER STATES
skills, capacity, technical assistance, technical support
M+E as communication tools? Simplify language? Consistent terminology, e.g.
“outcomes” v “results”
revisit institutional context, inc. Monitoring Committees, Managing Authority
Nature of Commission Guidelines and technical support
use of regular Peer Review Groups, (“boots on the ground”) ongoing Evaluators
merge Monitoring and Evaluation function in Member States
greater role for Eurostat/National Statistical Offices (but avoid pure context
indicators)
nature of Multi-annual Evaluation plans