1

EU Strategy for the Baltic Sea Region

Evaluation: Setting Outcome Indicators and

Targets

Seminar: 15 March 2011, La HulpeVeronica Gaffey

Acting Director and Head of EvaluationDG for Regional PolicyEuropean Commission

2

Objectives and Structure of the Strategy

• 4 Cornerstones: to make this part of the EU: ►environmentally sustainable,

►prosperous

►accessible and attractive

►safe and secure

• 15 Priority Areas• about 80 Flagship Projects• wide range of Actions

3

Specific Features of the Strategy

• integrated framework (many related policies)• wide range of Objectives at the level of the Strategy and Priority Areas• many Actions contribute to more than one Objective• concerns over 100 million people in 8 Member States• different Member States responsible for co-ordinating various Priority Areas and

Actions• absence of own funding source - variety of other funding instruments (major source:

Structural Funds)

4

Added Value of the Strategy• Strategy: co-ordination platform of various policies being

implemented in the Baltic Sea Region• Goal: co-ordinating an integrated use of existing co-operation

patterns, financial resources and know-how of various actors to achieve objectives of several policies in the Baltic Sea Region (Strategy’s Objectives) in a more effective and efficient way

▼Added Value

no one Member State can address this goal alone

5

Terminology• Objective: something you plan to achieve/change; expressed as intended

Outcome• Inputs: are the resources • Outputs: describe “physical” effect of spending resources - what the

money purchased • Outcomes: reflect targeted change influenced by intervention and

external factors; evaluation can help identify Impact (the contribution of the Inputs and Outputs to actual Outcomes)

• Indicator: variable providing quantitative or qualitative (development trend) information on a phenomenon (Output or Outcome); it can include Target (value + measurement unit + timelines)

• Baseline: value of Indicator before intervention

6

Theory of Change and Intervention Logic

• What does the Strategy aim to influence/change (Objectives expressed as Outcomes)?

• How can we be sure that Flagship Projects and Actions will contribute to achieving the Objectives of the Strategy?

• How the Strategy contributes to intended or observed Outcomes?

• How will we know when we have been successful?

7

Challenges: Monitoring and Evaluation (I)

• complexity of Objectives • cross-sectoral and cross-border framework• absence of own direct funding • shared co-ordination arrangements

What do we try to capture?

1. It would be difficult to use Output Indicators (the Strategy does not have its own funding source) although we could refer to Outputs identified under other funding instruments which support the Strategy (it would give us some idea of various interventions)

8

Challenges: Monitoring and Evaluation (II)

What do we try to capture?

2. Capturing Outcomes should be possible: they should express changes, taking place in the Baltic Sea Region, which are influenced by the Strategy and other external factors – challenge would be to identify relevant statistics or launch appropriate surveys

3. We should be able to provide evidence that the Strategy has an Impact

But, how should we understand this Impact?

9

Challenges: Monitoring and Evaluation (III)

• Impact: not as a contribution of the Inputs and Outputs to actual Outcomes - establishing clear Cause-Effect Relations between them would be difficult (the Strategy does not have its own source of financing)

• But, Impact as a contribution of the Strategy - through its Co-ordination Processes - to achieving various policies’ objectives in the Baltic Sea Region (Strategy’s Objectives: Outcomes) In other words, Impact = Added Value of the StrategyIt would not be possible to quantify this kind of Impact although we should be able to provide a qualitative evidence on it by using evaluation techniques

10

Providing Evidence on Impact: Practical Steps

Defining Outcomes Indicators at the level of the Strategy and for each Priority Area and demonstrating their inter-relationships

Clarifying Baseline situation in relation to these outcomes indicators and monitoring annually their development

Identifying Flagship Projects and Actions which should contribute to actual Outcomes under each Priority Area

Evaluating how and to what extent the Strategy (through its Co-ordination Processes - dependencies between mechanisms, e.g., tasks, peoples’ behaviour, and context) has influenced observed Outcomes: evaluating Added Value of the Strategy

11

The process…

• How to proceed?– Some external expertise can be built in (e.g. to

clarify the logic of intervention, outcomes and their baseline situation, to review the current monitoring system and to accompany preparations of an evaluation exercise)

– Who should ensure the annual monitoring of outcome indicators at the level of the Strategy and Priority Areas? Should Member States take ownership of the process under their respective Priority Areas?

– Who should be in charge of evaluating the impact of the Strategy and when?

12

Thank you for your attention