monitoring and evaluation (m&e) method for climate · 1 definitions adapted from undp, 2009,...
TRANSCRIPT
Methodological Guide
Monitoring and evaluation (M&E) method for climate change adaptation policies
LocaL authorities
relevance
sustainability
effectivenessutility
French Environment and Energy Management Agency
2 3
CONTEXT 41 • Adaptation policy definition and issues 4
2 • Definition and principles of territorial adaptation policy M&E 4
3 • Opportunities and challenges for adaptation policy M&E 7
PRESENTATION OF THE M&E METHODOLOGY 9
1 • Philosophy and objectives 9
2 • Scope and targets 9
3 • Methodological approach 9
4 • Methodology structure 10
ADAPTATION M&E TOOLBOX 12Prefiguring M&E in the local authority 12
Method sheet : M&E preparation and design 12
Method sheet : Managing M&E 14
Building my M&E system 17
Method sheet : The logical framework approach, a tool for structuring adaptation policy M&E 17
Method sheet : Different types of adaptation policy indicators 25
Method sheet : Help with choosing and measuring adaptation indicators 29
Method sheet : Setting up the indicator monitoring plan 32
Method sheet : Taking into account territorial observations on climate change 33
Implementing M&E in the local authority 35
Method sheet : Monitoring the adaptation policy 35
Method sheet : Evaluating the policy and communicating the results 37
ANNEXES 47Annexe 1 : Problem and solution diagrams for planning climate change adaptation 47
Annexe 2 : Help with selecting adaptation strategies and actions 50
GLOSSARY 55
1
2
3
INDEX
STRATÉGIE
PLAN D’ACTION
4 5
Adaptation complements attenuation. Whereas attenuation aims at reducing the causes of climate change, adaptation seeks to contain the effects through strategy and actions designed to reduce vulnerability and increase the resilience of socio-economic and natural systems in the face of climate-related disruption.
Adaptation requires an initial vision of the observed and potential consequences of a territory’s future climate: that is the goal of the diagnostic stage.
An adaptation strategy will define an array of political, technical, institutional, social and behavioural orientations to limit negative impacts and benefit from opportunities arising from the change. This strategy can be structured either by sector or more transversally and broken down into strategic and/or operational objectives and an action programme.
In practice, however, implementing adaptation is complex.
Our understanding of climate change and its impacts is only partial and surrounded by uncertainties. Local actors are often ill-equipped for decision making in a changing climate. Indeed climate change is a dynamic, ongoing process and adaptation policy has to be implemented in an unstable and particularly uncertain context. Adaptation therefore is not a one-off action aimed at moving from one stable situation to another, it requires strategic flexibility.
ADAPTATION POLICY DEFINITION AND ISSUES
DEFINITION AND PRINCIPLES OF TERRITORIAL ADAPTATION POLICY M&E
CONTEXT
Definition of monitoring and evaluation
Monitoring and evaluation (M&E) are two essential stages in the proper conduct of public policies. They involve two different but complementary notions that are worth clarifying here.
Monitoring is an ongoing process of collecting and analysing information on policy progress and
execution. Generally, monitoring indicates whether the activities are moving in the right direction and at the right pace and informs on costs and the budget.
It is a management tool for those in charge, the operational decision-makers and provides valuable indications on the progress of the action to assess the state of completion and plan any necessary adjustments.
1 2 Definitions adapted from UNDP, 2009, Guide méthodologique d’élaboration des politiques sectorielles.
Monitoring is based on a a system of information and analysis (“monitoring system”) with quantitative and qualitative indicators.
Evaluation is a strategic instrument for managing public action. It is a governance tool for measuring and analysing effects.
It is a one-off exercise that provides a systematic, objective assessment of policy design, implementation and outcomes. Evaluation indicates to what extent and why public intervention has obtained its objectives and generated the anticipated impacts on the target beneficiaries. Evaluation can be used to assess policy at different levels with a variety of criteria (sustainability, policy consistency, effectiveness of the action, impact on the territory) that can be tailored to the specific features of the policy and time of the evaluation.
Monitoring and evaluation are therefore both intrinsically linked to policy implementation: monitoring data provide input for the evaluation process and the evaluation provides feed back for the monitoring process.
There are, however, various distinctions to be made: In terms of management, monitoring is generally managed in-house, whereas evaluation can be outsourced or opened up more widely to stakeholders and citizens in the territory;
In terms of objectives, monitoring refers to policy implementation whereas evaluation focuses on the impact and results of the intervention;
In terms of time frame and dynamics: monitoring refers to policy implementation in the short term and is a continuous process that is constantly updated, whereas evaluation is a one-off exercise that generally assesses medium and long term effects.
Different M&E levels
In general, three different M&E levels are applicable to territorial policies. M&E must make it possible to observe and evaluate over time:In this “transitional regime” context, adaptive
management is imperative. This flexible gradual “step-by-step” approach to management is based on a process of continuous learning and implementation of flexible policies and measures that are regularly evaluated and adjusted in the light of developments in scientific understanding and changes in climate and socio-economic conditions.
Figure 1 : Different M&E levels
M&E level What is being measured
Mechanism to put in place
Associated measurement
indicators
TERRITORY
State of the territory in relation to specific features of the policy.
Observation of the context is the basis for a diagnostic of the territory which must be regularly updated to keep abreast of developments.
Territorial observation mechanism
Policy status and context indicators
STRATEGY
Impacts of the policy on the territory, on policies.
This is the strategic evaluation. It consists in establishing strategic objectives and verifying, over time, that they are achieved. Policy M&E
mechanism
Policy goal, strategy and outcome indicators.
ACTION PLAN
Implementation and visible results of the programmed actions.
This is the monitoring of policy implementation.
Action, process and resource monitoring
indicators.
Source: Référentiel pour l’évaluation des politiques territoriales, MEDDTL (2011)
Conception de la stratégie et du programme
d’actions
et suivi de la politique de la stratégie et du
6 7
Different policy M&E times
It is recommended to prefigure the M&E mechanism at the territorial policy design stage. This approach helps to structure consideration and construction of the territorial action plan and to establish specific objectives.
Furthermore, as noted above, whereas monitoring takes place continuously during policy implementation,
The diagram below provides an overview of the different policy M&E times:
ANTICIPATED BENEFITS
There are many advantages with this approach for adaptation policies. Indeed, M&E can be used to:
Promote continuous improvements and adaptive management
M&E promotes this dynamic by enabling a local authority to form a strategic vision of its policy, monitor progress and learn lessons from the successes and failures so that it can fine tune and improve the policy, the challenge being to increase resilience as feedback from the experience and the knowledge base expand.
Assess good management and outcomes
Local authorities face obligations of result and effective management of their policy. They must be able to justify that the resources allocated to actions are aligned with needs and produce the desired results. M&E must contribute to this reflection and direct resources to the most efficient levers.
Reinforce transversal governance of adaptation
A transversal adaptation policy requires a close relationship between the different sectors, decision levels and actors involved (water, biodiversity, health, land-use planning, etc.). M&E provides a common frame of reference to deal with this issue and promotes stakeholder synergies and analyses. It also facilitates policy coordination and alignment at different territorial levels so that the degree of transversality and cooperation during implementation can be assessed.
Reinforce consistency between adaptation and attenuation policies and other environmental policies.
In addition to transversal governance, adaptation policies must seek synergies with attenuation policies and other environmental policies. Attenuating greenhouse gas emissions and adaptation are two aspects of climate policies with complex boundaries and relationships. Poor adaptation may have feedback effects on energy consumption and greenhouse gas emissions (e.g. intensive air-conditioning to combat high temperatures). In contrast, certain adaptation actions provide co-benefits in terms of attenuation: for example, improved insulation of buildings can help to reduce energy consumption (heating in winter, air-conditioning in summer) and can also reduce people’s vulnerability to intense heat.
Selecting actions with co-benefits is a priority especially as the extent of climate change is heavily dependent on attenuation policies at different levels (local, national and international) and on development choices in the future. Implementation of an M&E system enables local authorities to consider the relevance of their choice ofactions beforehand (selection criteria for the intrinsic characteristics of adaptation policies and in relation to synergies with other policies) while also providing them with the means to subsequently verify integration of the issues.
Figure 2 Different monitoring and evaluation times for territorial policies
Territorial observation mechanismTERRITORY
POLICY CYCLE
IMPLEMENTING MONITORING AND EVALUATION
Diagnostic of vulnerabilities
and forecasts for the territory
Strategy and action plan
design
Policy implementation and monitoring
Diagnostic update and review of
the strategy and action plan
Populating M&E
Prefiguring and constructing M&E
Source : Adapted by TEC from ARGOS/RAEE (2012) and MEDDTL (2011)
In-itinere evaluation
Ex-post evaluation
OPPORTUNITES AND CHALLENGES INADAPTATION POLICY M&E
evaluation generally intervenes at two key moments: mid-way through the policy cycle (in-itinere evaluation) and at the end of the process (ex-post evaluation). At the mid-way point, evaluation aims to assess the initial outcomes and good governance and possibly redirect the strategy. At the end of the process, it provides a comprehensive strategic overview of the effects of the policy so the next cycle can be prepared based on a philosophy of continuous improvement.
In view of the dynamic, uncertain and non linear nature of climate change and its consequences, adaptation processes should be redesigned at regular intervals to take into account recent trends in climate change and structure of the territory. More than ever before, these policies need an M&E mechanism so the strategy can be adjusted in real time.
8 9
CHALLENGES
The intrinsic characteristics of climate change adaptation policies present several challenges for making an M&E system operational.
Lack of feedback and measurement units
The still exploratory nature of adaptation policies means that feedback on the evaluation of adaptation plans remains scanty. Few local authorities have formulated and effectively implemented their adaptation policies. Consequently, it is difficult to draw any general lessons on the relevance and effectiveness of the chosen orientations and actions. It is possible, however, to establish selection criteria for choosing actions which, tailored to the issues in each local authority area, guard against irreversible solutions and are sufficiently flexible to enable fine-tuning of the policy.
The same can also be said for the formulation of adaptation policy indicators. The lack of perspective on adaptation implementation makes it impossible to propose “ready-to-use” indicators acknowledged to be effective for M&E. There is also a further degree of complexity due to the lack of measurement unit. Indeed, while the effects of an attenuation policy can be measured with a common measurement unit (CO2 emissions) there is no universal indicator for evaluating the effects of adaptation policies.
Attributing responsibilities
It may prove difficult in practice to attribute a territory’s improved resilience specifically to an adaptation policy. A complex set of factors affect the state of a population and a territory, including socio-economic trends and cultural and climate factors. Furthermore, any relevant evaluation of adaptation policy outcomes requires the ability to posit and monitor a number of hypotheses and risks in achieving the objective and above all, to correlate results with territorial observations.
Evaluation of adaptation policies therefore requires several levels of analysis and must be based on both indicators and detailed analysis of key factors for the success or failure of the measures.
Time scales and climate change uncertainty
Adaptation consists in taking measures to protect against climate variability and observed change but also to ensure long term adaptation. In both cases, evaluation entails a number of risks and difficulties. Some measures, like the production of artificial snow in ski resorts can help to compensate for the increased variability of snowfall. However, this type of measure can also encourage maladaptation by maintaining an activity with increasingly higher economic and environmental costs.
Similarly, given the inertia in some sectors, local authorities are also currently faced with making choices on issues like construction and infrastructure design to adapt to a future risk whose full extent is unknown. It is therefore difficult to measure the short term benefit of these actions and establish a long term objective in the face of a moving climate target.
In view of these measurement difficulties, suitable for evaluating the extent of a system’s resilience, flexibility and reversibility, rather than focusing on the mere achievement of quantitative objectives to measure the real impact.
The proposed method is based on a proven M&E system for territorial policies and projects, already currently used by French local authorities, especially in the context of evaluating their Agenda 21. It is also increasingly being recommended abroad as theappropriate M&E framework for adaptation policies and programmes.This system takes the logical framework approach,
PHILOSOPHY AND OBJECTIVES
This methodology is a mechanism to help produce, structure and operationalise an M&E process for climate change adaptation policies.
It must enable local authorities to locate and monitor the progress of their policy and evaluate performance
SCOPE AND TARGETS
The methodology is flexible and integrative.
METHODOLOGICAL APPROACH
PRESENTATION OF THE M&E METHODOLOGY
with a view to continuous improvement and policy consistency.By offering local authorities a common reference framework for their M&E, the proposed methodology promotes the exchange of experience and collective learning in the field of still largely “exploratory” public policies.
It is directed at all territories (region, department, federation of municipalities, etc.) whatever their size, level of obligation, geographical situation (mountain, coast, rural, urban, etc.), their field of competences and their priority sectors of activity (industry, agriculture, tourism, etc.).
The methodology provides help with M&E system design. Thus all local authorities, whatever their level of progress towards adaptation can commit to an M&E mechanism tailored to their needs.
This method can be used as an ad hoc M&E system with the “Objectif ’Climat” tool developed from this present methodology guide or as a one-off or regular methodology input for defining and integrating specific adaptation indicators into pre-existing M&E mechanisms.
This method is directed first of all at the people responsible for promoting the in-house adaptation process, most commonly those in charge of the territorial climate energy plan (PCET). The transversal nature of adaptation strongly encourages the involvement of all the relevant local authority services, partners and/or actors at different moments in the M&E process to ensure success.
sometimes called the results framework, common framework or logical impact diagram depending on the local authority. This framework can be used to formalise and clarify the policy goal, objectives or strategic and operational orientations and attached action programme. It offers a vision of what is desirable for the territory or a given sector in terms of adaptation.
10 11
Prefiguring M&E in the local authority
Building the M&E system
Implementing M&E in the local authority
1 2 3
Instructions for use
Demo version of the tool
Methodological guide
Tool use procedure sheets
Case study implementing the methodology
PROCESS
OBJECTIF’CLIMAT
TOOL
Summary scorecard
Home
M&S system design
Monitoring scorecard
Local authority information
Main menu
Indicator monitoring plan
Logical framework 1,2,3 etc.
Introduction to the logical framework
Action and indicator selection
module
Organising management
Self-evaluation grid and results
Indicators are associated with each level of the policy so it can be monitored and evaluated. The issue of monitoring and then evaluation rests on the choice and sharing of relevant indicators suitable for reporting on policy impacts and progress.
Use of this methodology in the the field of territorial adaptation policies is experimental and and feedback from local authorities will be used to improve it.
In addition to the logical framework that structures the method, the proposed approach can guide local authorities at each stage of the M&E process. The main stages of M&E implementation in territorial policies are as follows:
Prefiguring M&E in the local authority
community
Building the M&E system
Implementing M&E in in the local au-
thority
1 2 3
Amélioration continue
This stage is intended to help local authorities decide on M&E system design according to their needs and characteristics and to guide them
with organising management.
This stage aims to implement M&E by proposing a scorecard for monitoring the indicators and a standard
self-evaluation grid for the adaptation policy, which can be used to review and
improve the M&E system.
The proposed methodology takes the form of two complementary modules: this present methodological guide and the “Objectif ’ Climat” software tool with instructions for use that follows the steps presented above.
The method sheets provide methodological advice on how to organise, introduce and successfully implement M&E: how to build the logical framework, how to organise management, how to choose and define adaptation indicators, etc. This methodology can be used independently or in parallel with the computerised form according to local authority needs.
The “Objectif’ Climat” computer tool has been designed using the methodology proposed here. It is a tool for integral management of the M&E process, but certain modules can be used on their own if the local authority already has an M&E system or if the adaptation process is insufficiently advanced for any report. In particular, Objectif ’Climat provides:
2 tabs for prefiguring the M&E system: one for organising management and monitoring of the process, the other for selecting system design according to the approach chosen by the local authority.
1 tab introducing the logical framework approach with precautions and use recommendations and a standard logical framework matrix.
I help module for choosing indicators and preliminary evaluation of actions proposed following the logical frameworks
I tab proposing a detailed indicator monitoring plan with all the logical frameworks in the same table. It enables local authorities to plan monitoring.
3 M&E implementation tabs comprising:
1 • an indicator scorecard for collecting and
monitoring progress and outcomes in all the frameworks,
2 • a tab presenting a summarised scorecard.
3 • a tab with a transversal self-evaluation grid for the policy based on a system of evaluative questions to complete and notes, automatically shown as a graph to facilitate the final evaluative judgement and ultimately enable further adjustment of the system.
The computer tool also has instructions for use which present the tool and detail tab by tab the operating principles, filling-in procedure, advice and points to watch out for when using the tool.A demonstration version of the tool is also provid-ed. It gives a concrete example of the methodology in practice.
Figure 3: M&E implementation stages
This stage is at the heart of the approach. It is intended to prepare local authorities to implement adaptation
policy M&E: define indicators, the monitoring plan and the logical analytical framework in order to assess their policy.
1 2 3
Figure 4: General architecture of the methodology
M&E method sheets
METHODOLOGICAL STRUCTURE
Although the method presented here is based on a classical evaluation system, in the light of
experience in France and abroad, it provides methodological guidelines for adapting the
common framework, choice of indicators and evaluation to specific adaptation issues.
This present methodological guide has been designed as a toolkit for adaptation M&E. It is organised around M&E milestones in the form of “method” sheets.
d’une politique d’adaptationolitique d adaptation
1
2
Adaptationstrategy
3
Action plan
4
Monitoring and evaluation
5
Decision to commit to an
adaptation policy
Identification of territory
vulnerability to climate change
12 13
This stage is intended to help local authorities prepare and organise M&E in their territory by providing them with the components of methods for designing the system and organising management of the process.
PROCESS
The following questions must be answered: what should (or can) the local authority monitor and evaluate? What resources are available for that purpose?
Report on the context and define the scope of M&E
Before embarking on an M&E process, there must be a simplified overview of the state of progress of the process in the local authority. This will help to determine which levers and sectors should be the focus of M&E.
NOTE The Objectif ’Climat tool
provides help with choosing the design of the M&E system in relation to how far the local
authority’s adaptation process has progressed. A simple questionnaire provides
useful recommendations.
Some questions can help to take stock of the situation:
It is important to have least identified the consequences of climate change before embarking on an M&E process (see figure opposite). Local authorities that do not have a vision of their vulnerabilities and
PREFIGURING M&EIN THE LOCAL AUTHORITY
METHOD SHEET M&E preparation and design
The sheets provided in this part are:
Method sheet: M&E system preparation and design
Method sheet: Organising M&E management
ADAPTATIONM&E TOOLKIT
1
1
Where is the local authority in the
traditional cycle of an adaptation process?
Figure 5: Simplified adaptation policy cycle
issues, can initially use the pre-diagnostic tool “Impact’Climat” developed by ADEME which provides an overview of the set of initial impacts that may affect the territory or local authority competences.In parallel, they can begin to establish the bases for M&E by defining key directions and actions on the matter of adaptation governance and management (appointing an adaptation advisor benchmark, launching an in-house or external diagnostic study, etc.) followed by indicators to measure progress in the implementation process.
The territorial diagnostic must be crossed with local authority competences to determine key levers and sectors to prioritise for M&E. For example, if a local authority has a mandatory and practically exclusive competence to manage flooding risk and has found greater vulnerability to this issue, it must pay particular attention to this sector in its adaptation policy M&E. If greater vulnerability is identified in an area where its competence is more restricted, it can still choose indicators (either for example because the vulnerability is likely to impact one of its sectors, or because it is considered fundamental to observe the progress made by other local authorities) but fewer of them. A hierarchical list of key and secondary
sectors/themes is needed.
In order to monitor and evaluate a policy there must be a strategy (based on strategic and/or
operational orientations) and an action plan. M&E must in effect enable observation of progress in plan implementation and evaluation of whether the actions are enabling achievement of the local authority’s strategic objectives. If a local authority does not have any, it must first work to formalise its strategy and action plan to enable M&E. As this task takes time, it could concentrate first on one or two priority sectors in addition to defining objectives on the issue of transversal governance. To help with this exercise, there is the usual “problem and solution diagram” method which is a simple method for summarising the vulnerabilities of a territory and adaptation planning. This type of diagram helps to clarify the objectives and anticipated outcomes of the policy (vision of what is “desirable”). There is an example in the annexe to this document.Local authorities that already have a strategy and an action plan at the prefiguration or implementation stage can directly address introduction of the M&E framework for all or part of the strategy, necessarily taking into account the available resources and inputs. The common analytical framework proposed in this methodology is the logical framework (see method sheet “ The logical framework approach, a structuring tool for adaptation policy M&E “).
What are the priority issues for the territory (or for the
studied sector)?
Has the local authority got a strategy
and an adaptation action plan for these priority sectors and issues?
This sheet is provided to help guide local authorities towards an M&E system tailored to the progress of their adaptation policy, to their desired priorities and to the available inputs and resources.
14 15
Is the local authority’s adaptation policy part of a more global policy framework (a climate, energy, transport policy
for example) ?
If it does, is it relevant
for analysing adaptation policy?
Organising adaptation policy M&E with what already exists
Definition of the scope of M&E must be followed by examination of the operational resources avail-able for implementation in relation to community objectives. A decision has to be made on whether it is more relevant to move towards ad hoc man-agement of the adaptation policy or whether to in-tegrate it in a pre-existing analytical system or study it in conjunction with another policy component.
The following questions should be asked:
Check on the inputs allocated to the M&E cycle
In addition to choosing how to manage the process, par ticular attention should be paid to the inputs allocated to M&E. In fact, the introduction of M&E, the procedures for gathering and monitoring indicators and the types of evaluation can quickly take up significant financial and human resources. It is advisable therefore to target the M&E process in relation to inputs and, for example, define a few strategic indicators for priority sectors rather than a long list of indicators that will be impossible to monitor over time due to lack of resources.
Here are some checkpoints: human resources (a service or person dedicated to M&E, number of hours project managers dedicate to M&E, etc.) technical and financial resources (procedures for gathering existing indicators, resources allocated to M&E in the local authority, etc.)
This sheet is intended to guide local authorities on how to organise and manage each M&E stage. It provides an approach and components of methods which must obviously be adapted to the local context and each local authority’s objectives.
It helps to fur ther understanding of the different roles and responsibilities for each stakeholder according to whether the focus is on monitoring or evaluation.
PROCESS
Due to its transversal nature, creating an adap-tation process as well as its M&E generally re-quires the mobilisation of many stakeholders in and outside the local authority. They are mobil-ised at different stages of the process and are more or less numerous according to the scope and resources the local authority assigns to the process.
Does the local authority already have an M&E mechanism?
METHOD SHEETOrganising M&E management2
We propose below a standard process that can be tailored to the specific features of your local author-ity. By way of example, it may be useful for a given local authority to review the composition of an ex-isting M&E body for the climate, energy and trans-port plan rather than create an ad hoc structure.
Designate a person in charge of managing the process in-house.
Good practice in a climate change adaptation pro-cess involves the definition of management led by a contact person, usually the climate project manager, who supports and directs adaptation policy. This person must be in a position to inform elected of-ficials and competent services of his or her actions in order to coordinate transversality of the actions defined in the strategic plan.
The climate project manager is often identified as the contact person for M&E of the local au-thority’s adaptation plan. This contact person centralises the tools and information and has contacts for the entire M&E process.
Implementing operational management of monitoring
Monitoring provides information for assessing poli-cy achievements and planning any necessary adjust-ments. It can be taken in hand by the climate project manager but requires effective participation from the services involved with the adaptation policy (water authority, urban planning and development, health, etc.). These services will help to define the relevant indicators, collect them regularly, analyse their progress and then transmit them to the con-tact person.
Local authorities can decide to create a more for-mal body, a monitoring committee for example, to formalise this management. This committee can meet at key monitoring moments, that is, at the start of the indicator selection process, then at key stages in the policy process (periodic monitoring reviews) to measure progress and state of advancement of the climate change adaptation policy. The commit-tee does this using a scorecard and a monitoring report produced by the contact person identified beforehand. It also proposes operational adjustments (credit lines, choice of indicators, etc.). This monitor-ing committee, comprising the contact person and operational managers, may also include key elected officials as well as stakeholders from the territory.
Workshops to coordinate the monitoring of in-dicators should be organised regularly between the project manager and service contact per-sons designated by depar tment heads for the indicator monitoring plan. These meetings can be informal but must be regular and the decisions that emerge must be noted in the indicator mon-itoring plan.
Implementing strategic control of evaluation
Evaluation is the strategic control instrument for public action. It is a governance tool that can be used to measure and analyse the effects of the policy.
An evaluation committee is essential for mak-ing key decisions linked to policy evaluation. A par ticipative evaluation committee must bring together, in addition to the contact person and local authority representatives, elected officials, par tner institutions and actors from civil society (socio-economic actors, businesses, associations, etc.). Indeed, the success of climate change ad-aptation policies is often conditional upon the degree of stakeholder commitment and par tici-pation. Par ticipative evaluation can therefore be a real asset for the local authority.
The evaluation committee is constituted prefer-ably when policy evaluation is implemented and validates the indicators chosen by the monitoring committee. It can issue an evaluation opinion before the policy starts. It can then be mobilised midway through the strategy to assess performance of the climate change adaptation policy and decide on corrective measures (for example, after 2 years). Ultimately, it can decide whether to outsource the evaluation or to carry out the final evaluation in house. To complete its missions successfully, the evaluation committee uses the indicator scorecard and an evaluation grid and generally issues an eval-uation report.
SUGGESTION
The ARGOS/RAEE guide provides useful advice on organising a participative evaluation
committee (number of people, number of meetings required to prepare the evaluation
with the committee, etc.). Have a look!
1 ARGOS/RAEE, March 2012, Guide méthodologique : l’évaluation au service du pilotage d’un Plan Climat-Énergie Territorial.
16 17
The aim here is to help local authorities assess their climate change adaptation policy. To this end, there must be a general vision of the policy beforehand:
The major issue here resides in policy (re)formulation (strategies and action plans are often presented in very different ways in a framework document) and transcription to a common analytical framework in order to visualise the policy “impact chain”. The logical framework tool presented here will enable this work and serve as the basis for the entire M&E process.
What are the expected results of any particular
action?
Formalise an in-house procedure for managing monitoring and evaluation
Implementing an in-house procedure to help the contact person organise monitoring and evaluation of the policy. See a standard example below:
Figure 6: Example procedure that can be implemented in the in the local authority
Periodic monitoring review
OBJECTIVE: Operational monitoring of the adaptation policy
ACTORS: Monitoring committee Climate project manager, department heads, other key persons from the services
TOOLS: Monitoring plan, monitoring sheet, scorecard, annual monitoring report.
OBJECTIVE : Evaluation of the adaptation policy in order to decide on corrective measures
ACTORS: Evaluation committee Elected officials, climate project manager, managers and department heads and key persons from services, and any external stakeholders
TOOLS: Scorecard and evaluation report designed using an evaluation grid
PROCEDURE:
1 • Monitoring report and presentations prepared by the climate project manager;
2 • Analysis of indicators, scorecard and monitoring report by the monitoring committee;
3 • Decisions and indications of corrective measures concerning execution and achievement of the desired results;
4 • Climate project manager integrates these measures in the execution plans
PROCEDURE:
1 • Evaluation committee analyses the indicators and the scorecard;
2 • Evaluation (external and/or in house) based on an evaluation grid: formulation of evaluative opinions and recommendations (corrective measures);
3 • Presentation of evaluation results and recommendations by the evaluation committee and/or during public workshops;
4 • Joint validation by the contact elected official(s) and the evaluation committee of the chosen corrective measures;
5 • Integration of corrective measures in policy planning (review of the logical framework, knowledge base, objectives, action plans, resources, etc.)
This stage is at the heart of the approach
It is designed to prepare local authorities to implement the M&E system for their adaptation policy. It provides a method for defining indicators and a logical analytical framework so that local authorities can assess their adaptation policy.
BUILDING THEM&E SYSTEM
The sheets provided in this part are:
Method sheet: The logical framework approach, a tool for structuring adaptation policy M&E
Method sheet: The different types of adaptation policy indicators
Method sheet: Help with choosing and measuring adaptation indicators
Method sheet: Setting up the indicator monitoring plan
Method sheet: Taking into account territorial observations on climate change
2
METHOD SHEETThe logical framework approach, a tool for structuring adaptation policy M&E
1
What is the purpose of the policy?
What are the conditions for policy success and risks to
achievement?
Communicating in the territory
In order to further appropriation of the climate change adaptation policy and the final evaluation as widely as possible Workshops for possible public workshops can be organised. These workshops can be used to de-liver content and integrate remarks and opinions from the territory’s inhabitants and key entities for decision making and corrective planning.
Evaluation
18 19
The logical framework matrix is organised as follows:
LOGICAL FRAMEWORK: DESCRIPTION AND DEFINITION
The logical framework is a planning method that allows local authorities to formalise and organise the intervention logic behind their territorial policy. It takes the form of a matrix that summarises the general plan for the adaptation process by logically presenting key policy components and their cause and effect relationships (based on the level of inputs and actions until the strategic outcomes and more long term expected effects are achieved).
It is intended to provide a vision of what is desirable for the territory or a given sector as regards adapta-tion. It is also an informative reference document for the different actors involved in the policy such as elect-ed officials and local authority agents, their institutional partners, civil society representatives, professional or-ganisations, etc.
The logical framework tool focuses on policy M&E and therefore concerns the “Strategy” and “Action Plan” for the policy.
Figure 7: Standard logical framework matrix
The formulation of intervention levels may vary depending on the local authority. Certain strategy documents or adaptation action plans may speak of strategic “orientation” rather than strategic “objective”. The important thing is to place the different policy components at the correct level in the framework regardless of the headings or semantic fields employed.
Construction of the logical framework requires local authorities to collect, format and if necessary complete the available information on territorial issues and definition of their adaptation strategy.
For local authorities with a relatively mature strategy and action plan, this work will essentially involved transferring these elements to the framework format. Nevertheless, significant work on reformulation and prioritisation of actions may be necessary. We advise using the examples provided in the demo version of Objectif ’Climat.
Figure 8: Basis for the approach, the reference logical framework
THEME 4: PROMOTING A CULTURE OF CLIMATE RISK IN HAUTE-NORMANDIEObjective 4.2: sensitise elected officials and decision-makers to adaptation issues
Action 4.2.1: train government and local authority services in climate risks for the territory
Transition from adaptation strategy...
Operational objective:Develop the ability of agents in charge of validating urban planning documents (SCOT (land develop-ment), PLU (town planning), etc.) drawn up by local authorities to take climate risks into account in their analysis of the development projects in the documents. Government officers must be trained to compare territorial projects with projected climate developments around key questions to be taken into account: Does the development project affect the water cycle at the level of the watershed area (soil sealing and runoff downstream)? Are flood zone expansion areas (and exposure of existing housing and facilities to flooding risk) subject to change? etc.
Shown in rows are:
- the description of each intervention level
- the nature of the indicators for each level,
- success factors, that is, key factors determining the smooth progress of the policy which are worth monitoring and managing throughout implementation.
:
M&E level
Intervention logic
Description Indicator by M&E level
Factors for success
THE STRATEGY
Goal
Describe the policy's medium or long term adaptation objective at territorial and/or sector level
Example
Promote a culture of climate risk in Haute-Normandie
Effect indicatorExample
Expertise and action capability of actors in adaptation matters
Example
Actors are convinced of the reality of climate change
Strategicobjective
Describe the intended strategic outcome(s) of the policy
-
Example
Sensitise elected officials and territorial decision-makers to adaptation issues.
Strategic impact indicators
Example
Number of sensitised persons (training and information campaign)
Example
Support from elected officials
Operational objective
Describe the tangible elements the policy must produce to achieve the strategic objective
Example
Train government and local authority services involved in climate risk management to analyse and implement projects
Operational outcome indicator
Example
Number of persons trained
Example
Motivation and availability of targeted audiences
ACTIONPLAN
Action
Describe the different tasks required to obtain the expected operational outcomes
-
Examples
Government inter-service training
Inter-community training
Action completion indicator Examples
Amount of training provided Quality of training provided
Example
Inputs (optional)
Describe the human, technical and financial resources mobilised to implement the actionsExamples
Available budget External technical assistance
Input indicators
Examples
Amount of subsidies obtained Number of trainers mobilised
Example
Funding requests approved
Availability of required expert profiles
Training contenttailored to needs
The column shows 5 levelsof intervention: 1. Goal2. Strategic objective(s) 3. Operational objective(s)4. Actions5. Inputs (optional)
M&E level Transcribed intervention level
TERRITORY (EXCLUDING METHODOLOGY)
Territorialand climate
context
Changes in climate extremes
Context indicator (climate and socio-economic)Examples
Frequency of extreme hydro-meteorological events
Population ageing rate
STRATEGY
Goal Promote climate risk culture in the territory
Medium and long term Impact indicator Example
Expertise and action capability of actors in adaptation matters
Strategicobjective 1
Sensitise elected officials and territorial decision-
makers to adaptation issues.
Strategic indicators for specific policy impacts
Example
Nb de personnes sensibilisées (training and information campaign)
Operationalobjective 1.1
Train government and local authority services involved in climate risk management to
analyse and implement projects
Operational outcome indicatorExample
Number of persons trained
ACTION PLAN
Action 1.1.1 Government services training Action completion indicator
Examples
Amount of training provided Quality of training provided
Action 1.1.2 Local authority training
Inputs 1.1.1.1 Mobilised budgetInput indicatorsExamples
subsidy % obtained Number of trainers mobilised
Inputs 1.1.1.2 Trainers
A-Intervention logic B-Description C-Indicators D-Success factors
GOALReduce damage to property and persons due to increased 10-year floods
Loss amount
Number of affected persons Good inter-service coordination Political determination
STRATEGICOBJECTIVE 1
Limit economic and human impacts in flood zones.
Evolution over 5 years of the value of construction in a flood-risk area
Evolution over 5 years of the number of inhabitants living in a flood-risk area.
Support from elected officials and users
STRATEGICOBJECTIVE
1.1
Prevention instruments are reinforced and fully applied in risk areas.
% of the population at risk benefiting from prevention measures
Effective application of identified prevention measures
ACTION 1.1.1 Introduction of a IntegratedRelocation Pilot Programme that includes climate projections
Existence of an Integrated Relocation Pilot Programme
Favourable discussion among elected officials
Technical assistance from the government Ability to evaluate the impact of climate change on flood risk
Cooperation with the "Natural risk management" service
ACTION 1.1.2 Updating urban development plans
Extent to which current and future climate risks are taken into account in the development plan (scale from 1 to 5)
Ability to evaluate the impact of climate change on the hazard and the spatial extent of flooding.
Cooperation with the "Town planning" service
ACTION 1.1.3Improving procedures for appraising and controlling building permits.
% of building permits that include prevention measures
Cooperation with the "Town planning" service
ACTION 1.1.4
Implementation of measures to restore and protect wetlands with a role in flood control.
Surface area of restored/conserved wetlands
Commitment from the farming profession
Eligibility of measures for the programme of agro-environmental measures
Compatibility with land development and town planning.
Specific indicators for adaptation issues
Other local authorities whose policy design is less advanced will need to identify the vulnerabilities and issues they wish to act on in order to define the adaptation objectives and identify the corresponding solutions and actions.
To help them with this exercise, local authorities can use the Impact’Climat tool developed by ADEME and the “Problem and solution diagram” method which are two simple methods for analysing a territory’s vulnerability and for planning adaptation (see Annexe 1).
The diagram thus produced will enable them to clarify the goals and expected outcomes of their policy (vision of what is “desirable”), test its internal consistency and decide between different intervention strategies. These components can then be reported in the framework matrix and appropriate indicators can be chosen to measure them.
Figure 9: Example of a logical framework for water management and adaptation to climate change
DEVELOPING A LOGICAL FRAMEWORK
The framework is generally developed using a five-step approach more or less completed by existing local authority documents according to the state of advancement of the adaptation process:
1 • Thorough examination of the situations and data on the diagnostic on which the local authority wishes to act;
2• Formulation of the desired medium and long term strategic adaptation goals and objectives;
3• Formulation of operational outcomes and actions intended to lead to the desired outcomes;
4• Identification of success factors for achieving the objectives5• Choice of indicators for measuring the achievement or
otherwise of objectives and outcomes and to monitor actions ( maximum 2 to 3 indicators per intervention level).
... to formulation of the M&E framework for the policy
20 21
SUGGESTION
Local authorities can choose to build several logical frameworks for the
different sectors in their adaptation policy framework (town planning, water management, etc.) or they may choose to act on only one of
the sectors involved. In all cases the strategy must be used to produce
your framework.
Goal
STRATEGIC OBJECTIVES
OPERATIONAL OBJECTIVES
ACTIONS
1
2
3
4
9
10
11
12
8
7
6
5
The completion procedure requires a dual vertical and horizontal logic and in general follows the sequence shown below (see the figure below).
Column B Stages 1 to 4
The local authority star ts by describing in column B (from top to bottom) the different levels of policy intervention respecting a certain hierarchy between them. This hierarchy should be based on a causal relation of the type “if-then” summarised as follows:
NOTE As in the previous exam-ples, the input level can be added. It is, however, proposed as optional in
the methodology to avoid making the process too
cumbersome.
IF we wish to contribute to the adaptation goal THEN the strategic objective must be achieved.IF we want to achieve the strategic objective, THEN certain outcomes from the operational objectives must be achieved. IF we wish to achieve these operational objectives, THEN certain actions must be carried out.
Stage 1
Explain the “Goal”: the goal is the ultimate outcome, that is, the intended medium and long term effects which the policy seeks to fur ther. There are generally two types of effects: reinforcing the local authority’s adaptation ability or reducing its vulnerability to climate change.
Stage 2
Formulate “Strategic Objectives”(maximum two per field/topic):strategic objectives are the expected impacts of the policy at strategic level which, in combination, will contribute to the adaptation goal.
Stage 3
Define “Operational Objectives” (maximum three per strategic objective)Describe the results the policy must produce to achieve the strategic objective.
Stage 4
Inform on the “Actions” to be implemented (maximum three per operational objective): these are the main adaptation actions identified in the action plan required to achieve the expected outcomes.
Here is a list of questions to guide completion of each stage of the logical framework.
GOAL
What is the ultimate long term policy goal in terms of “reinforcing adap-tation capability” or “reducing local authority vulnerability”?
What effect indicators can be used to measure the policy’s contribution to the adaptation goal ?
What factors and conditions external to the policy are required to achieve this goal?
STRATEGIC OBJECTIVES
What strategic objectives must the policy achieve in order to contribute to the goal?(Please provide a numbered list - maximum of 2 strategic objectives)
What indicators can be used to measure in detail the extent to which each strategic objective has been achieved?
What external conditions must be taken into account to ensure the strategic objectives are achieved in the allotted time?
OPERATIONAL OBJECTIVES
For each strategic objective, outcomes are the “products” of the actions implemented. All outcomes contribute to completion of the associated strategic objective.What are the expected outcomes? (Please provide a numbered list - maxi-mum of 3 operational objectives for each strategic objective)
What indicators can be used to verify and measure that the policy has achieved the expected results?
What external conditions must be taken into account to ensure the outcomes are achieved in the allotted time?
ACTIONS
What actions are to be implemented and in which order, to obtain the expected outcomes?(Group and number actions by operation-al objective, maximum of 3 actions per operational objective)
What indicators can be used to appreciate advancement of the actions?
What external conditions must be taken into account to implement the planned actions within the allotted time?
INPUTS (OPTIONAL)
What human, technical and financial inputs have to be mobilised in order to complete the actions?
What indicators can be used to evaluate the inputs deployed?
What external conditions must be taken into account to gather the planned inputs in the allotted time?
A-Intervention logic B-Description
Figure 10: Questions for completing the frameworkFigure 11: Logical framework construction process
C-Indicators D- Success factors
22 23
A-Intervention logic B-Description C-Indicators D- Success factors
:
Policy M&E level Indicators by M&E level
THE STRATEGY
GoalEffect indicators (adaptation capacity/
vulnerability reduction)
Strategic objectiveStrategic impact indicators
Operational objective Operational outcome indicators
ACTION PLAN Action Action implementation indicators
Inputs Input indicators
24 25
Column D Stages 5 to 7
The matrix is completed by identifying the success factors (from bottom to top). These are the exter-nal conditions likely to favour or hinder the success of the adaptation policy, that is: “What conditions external to the policy must be met to guarantee achievement of the anticipated objectives and effects? “ The relationship between success factors and each level of intervention in the framework is illustrated by the figure below.
IF actions are taken AND if success factors identified by the local authority are confirmed, THEN the operational objectives can be achieved, etc.
The factors deemed critical for the policy but with weak or fragile prospects of achieving them are risks which should also be monitored, evaluated and processed throughout implementation.
Stages 9 to 12
After determining the intervention philosophy and success factors for the adaptation policy (columns B and D in the matrix), a set of relevant indicators for measuring and repor ting on process performance must be defined in column C. This is the hear t of the system. There is a type of indicator for each level of intervention. To formulate and choose the indicators correctly, refer to the method sheets “Different types of indicators for adaptation policies” and “Help with choosing and measuring adaptation indicators”.
This sheet contains conceptual and operational specifications on the concept of indicators and their specific features for adaptation policies to help with the choice of relevant indicators for each M&E level.
PRESENTATION
Adaptation objectives are of a changing nature. Adaptive management requires a dynamic understanding of territory adaptation as a sequence of interventions and graduated response (adaptive path) whose implementation is scaled in time not only in relation to the nature of the oppor tunities and threats and the speed at which they occur, but also to the results of the experience.
In this context, indicators are essential tools. They promote learning about adaptation as well as stakeholder dialogue and information in the face of changes which are still poorly understood, while permitting local authorities to move their adaptation process forward. M&E indicators can be used, at regular intervals, to take stock and thus develop local adaptation plans and strategies as well as risk management mechanisms deployed by local authorities.
CATEGORIES OF ADAPTATION INDICATORS BY INTERVENTION LEVEL
The proposed categories of indicators are as follows:
NB : This sheet does not deal with the issue of context indicators at territorial level, because it is outside the M&E mechanism, and is the subject of a specific method sheet “Taking into account territorial observation of climate change”.
SUGGESTION
If a local authority does not have a clear vision of these actions or if it wishes to verify their rele-vance, it can refer to the method sheet in the annexe “Help with selecting adaptation strategies and actions” which gives specific examples of adaptation options and measures in different local
authority sectors and fields of competence.Furthermore, the Objectif ’Climat tool provides tool has a module to help with selecting actions: it
can be used to evaluate and prioritise interventions using a multi-criteria approach.
Figure 12: Numbering elements in the logical framework
Operational objective 1.1
Operational objective 1.2
Action 1.1.1 Action 1.1.2 Action 1.2.1
Strategic objective 1
Figure 13: Account taken of success factors in the construction of the logical framework
Goal
Strategic objective
Operational objectives
Actions Success factors
Success factors
Success factors
+
+
+
METHOD SHEETThe different types of adaptation policy indicators2
In contrast to attenuation which can be measured quantitatively by greenhouse gas emissions or equivalent tonnes of CO2, so far there is no universal measurement of adaptation to climate change. Monitoring adaptation requires different types of indicators at different levels of policy M&E (territory, strategy, action) and is based on quantitative and qualitative data. The aim is to be able to assess both the adaptation process and the outcomes.
Column C
Source : Commission Européenne (2004), Manuel de gestion de projets.
Throughout the sequence (preceding page), for greater clarity it is recommended to relate the corresponding objectives, outcomes and actions hierarchically using the applying the following numbering.
26 27
• Adaptive capacity indicators
Adaptive capacity covers everything that ena-bles a territory to adapt to a positive or nega-tive change. Adaptive capacity can be assessed using the tools available to a local authority and other actors in a territory to deal with climate change.Adaptive capacity indicators correspond to components that can be used to monitor and evaluate the existence of these tools or capacity in the territory to mobilise them to deal with change.
Examples of adaptive capacity indicators:
Number of agents with competences in the field of climate change adaptation.
Existence of prevention and safeguard plans
Adapted agricultural surface area
Development level of early warning systems
Budget allocated for adaptation actions
• Vulnerability indicators
Periodic monitoring and evaluation of the vulnerability of a territory to climate change requires indicators that enable simplified up-dating of the diagnostic of the local authority’s vulnerability. An answer must be found to the question “are we about to become more or less vulnerable? “These indicators will therefore be used to measure the anticipated reduction in vulner-ability to climate change in a territory as a result of the adaptation process. These indica-tors measure territory exposure and sensitiv-ity to climate change.
Strategic impact indicators
These indicators are intended to measure changes flowing directly from the policy in each of its main strategic fields. Specific impacts of the policy on socio-economic, environmental, political and even institutional aspects of adaptation are assessed. They can be transversal indicators, for various sectors at a time, as well as indicators for cer tain sectors.Note that effect and impact indicators can sometimes be very similar, except that impact indicators measure more tangible changes directly attributable to policy action.
Effect indicators
They are used to evaluate the direct and indirect effects of the policy on the two main goals of any local ad-aptation policy, that is, reinforcing resilience and reducing vulnerability in the territory. They must be related to territorial context indicators in the M&E.
Operational outcome indicators
These indicators inventory immediate outcomes and results of the policy, measuring the degree of achievement. Operational outcome indicators are generally quantifiable components defined by a quantitative measurement unit or by its evolution. Qualitative or rating indicators for outcomes can also be used.
There are generally two type of outcome indicators:
Number of environmental restoration sites
Methodological guides on adaptation (yes/no)
Progress in sensitisation actions (1-weak, 2-moderate, 3-satisfactory, 4-very satis-
factory)
Input indicators
These are transversal or sector indicators used to measure the technical, human, financial or other inputs for the adaptation policy. Whenever possible, input indicators are expressed as an economic unit (for example full-time equivalents dedicated to the adaptation process) in order to facilitate aggregation ad comparison.
The indicator categories presented above are not to be seen as watertight concepts with clearly defined boundaries.
In practice, a certain degree of flexibility is needed in the use of this typology which must be tailored to the particular features of the territory and its adaptation policy. Thus cer tain types of indicators can easily apply to different categories especially in relation to M&E needs and the way the policy is structured.
Examples of vulnerability indicators:
Surface are of flood risk areas
% of built surface exposed to landslide risk
Urban surface area at risk
Location of farm land
Agricultural surface areas at risk
Amount of flood-related damage
Number of persons affected
Expenditure on road maintenance
% reduction of yields in dry years
Evolution of agricultural water use
Number of extreme weather-related incidents
in the electricity network
Examples of strategic impact indicators:
% of the population with a protection plan Level of decision-maker sensitivity to climate change.
Number of hotels adapted to new summer conditions
Existence of prevention rules or orders at local authority level
Number of farmers with insurance against extreme weather events
Urban compactness
• Sector indicators:
These indicators are attached to a given sector and populated with sector data.
Examples of sector indicators:
Lines of elevated dikes (km)
Existence of a heat island map
Green roof surface area (m²)
Lines of re-sized piping in the rainwater network (km)
Number of flood protection works
Examples of achievement indicators
Number of vulnerability studies
Number of workshops
Number of communication materials
Number of partnership agreements
Examples of input indicators Subsidy amount (€k)
Community investment amount (€k)
Cost of works (€k)
Full-time equivalent involvement from the local authority
Budget voted in favour of adaptation (€k)
Service contracts amount (€k)
• Coverage indicators
These indicators cover transversal actions or have a territorial dimension
Examples of coverage indicators:
Number of actors (local authorities, agencies, companies, inhabitants) engaged in adaptation activities
Number of trained agents
Number of territorial policies and plans which include climate change analysis
Number of projects re-evaluated or modified in view of the potential impacts of climate change.
Achievement indicators
They inform on the various actions implemented to produce the expected outcomes for operational objectives and their completion rate.
28 29
METHOD SHEETHelp with the selection and measurement of adaptation indicators3
Adaptation policy M&E must be based on qualitative and quantitative indicators which must be simple to use and permit unequivocal measurement of what is in the policy’s sphere of influence. These indicators must be accom-panied by schedules, types of responsibility sharing and accurate information sources and must contribute to the evaluation of performance and decision making.
It is therefore important to have quality criteria and a measurement method for evaluating each indicator to ensure its relevance.
SELECTING AND CONSTRUCTING THE INDICATORS
In order to fully assess policy effectiveness and impact, it is often necessary to combine quantitative and qualitative indicators. Nevertheless, too many indicators, insufficiently targeted on policy objectives is a pitfall to be avoided because it would slow down the M&E process and ultimately impair legibility. It is therefore advisable not to have more than 3 indicators for each intervention level (goal, strategic objective, operational objective, action and input).These indicators can be defined with commonly used quality criteria:
Monitoring and evaluating climate change adaptation in the territories: Methodological components taken from international experience (ADEME) : this collection provides a large number of examples of indicators based on international experience where French local authorities can find inspiration for their own process.
Climate change adaptation, 12 action sheets for local authorities (ADEME, Languedoc- Roussillon Regional Office). This collection also proposes indicators for monitoring adaptation actions.
Indicators of the territory’s vulnerability to climate change (ADEME)
OECD guide (2012) “Monitoring and Evaluation for Adaptation: Lessons from Development Cooperation Agencies” provides proposed indicators for major adaptation options (research, risk reduction, sensitisation etc.) based on the experience of international agencies.
FURTHER INFORMATION ON ADAPTATION INDICATORS
SMART METHOD
A good indicator must respond to several quality criteria commonly used internationally under the term “SMART”. To be SMART, an indicator (and its target value) must be:• SPECIFIC: it must be directly related to the component it is supposed to measure ;• MEASURABLE : whether qualitative or quantitative, it must be capable of objective measurement in the sense that the information must be accessible and the measurement results must be the same whoever takes the measurement;• ALIGNED: it must be relevant and meet information needs;• REALISTIC: it must be possible to achieve the indicator and its target values at an acceptable cost;
• TIME-RELATED: it must apply to a clearly defined time period and include deadlines (e.g. by 2020, mid-term, end of project, etc.)
“Reinforce the expert and action capabilities of technical services in the field of climate change adaptation.”
The intention is to define a strategic outcome indicator and its target value: Specify the indicator: Classification level...
Specify the quantitative objective: …at least 50 % of clean-up service teams…
Specify the qualitative objective: … show improvement in the issue of adaptation to climate change…
Specify the time horizon: …within two years…
Define the spatial coverage where possible: …in all intercommunal associations
Example of construction of an indicator for the following strategic objective:
30 31
The criteria below can be used by local authorities as an additional check list for choosing indicators tailored to the need for adaptation policy M&E.
Validity: does the indicator permit measurement of change in terms of adaptive capacity, vulnerability reduction, climate risks or any other key element in the adaptation policy? Acceptability: do the different stakeholders agree on the definition and relevance of the indicator and what it is supposed to measure ?
Reliability: Does the indicator correctly reflect the changes we wish to measure? Sensitivity: it is sufficiently correlated with the phe-nomena to be measured in the short, medium and long term? Meaning: can variations in the indicator give rise to contradictory interpretations? Attribution: how can variations in the indicator be attributed to policy outcomes ? Utility: will the information collected make a useful contribution to the adaptive management process and clarify the decision?
MEASUREMENT OF ADAPTATION INDICATORS (INITIAL STATE/TARGET OBJECTIVEAfter defining the indicators, there must be com-ponents that will situate the state of advancement of the adaptation policy in relation to the objec-tives. It must also be possible to establish a ref-erence state, corresponding to time zero of the policy and a target objective expected at the end of the implementation period for each indicator.
The final status will take the form of a quantified and/or assessable objective describing the desired future situation with the indicator in question.
Initial state: before any M&E process, it is essential to know “where we are star ting from” before determining “where we want to go”. The local authority must therefore find out as much as possible about the status of each indicator at the star t of its policy or one of its implementation cycles. This star ting point, assessed quantitatively or qualitatively in relation to the nature of the indicator, will provide a reference state against which the progress and impact of the policy can be evaluated. This reference state can be either an observed state at the launch of the policy (what is the current situation?), or the future state of an indicator in the absence of any adaptation policy (what would be the situation if no policy were implemented?), in order to compare what is comparable.
Target objective: when the reference situation is known, it is impor tant to define the targets the policy seeks to achieve at the end of the considered period. With these targets the nature and extent of the desired effects and changes can be expressed in measurable terms. They constitute useful watch points throughout the M&E process. Other intermediate states may also be established to serve as milestones during implementation.
Focus on measuring qualitative indicators (binary logic and progression)
Given the lack of a specific measurement unit for adaptation policies, qualitative indicators that can assess the progress in the adaptation process are currently the most commonly used category of indicators.
Although quantitative indicators are generally easy to measure, it is impor tant to assess qualitative indicators that can measure the progress of the adaptation policy. They can be measured on scales which describe each stage of advancement.
Thus for all qualitative indicators (e.g: “the adaptation policy is formalised in a strategic document”), two choices are possible:
Another example of a progression continuum: a local authority wants to monitor the degree of progress in adaptation policy implementation. The local authority defines the stages which it associates with numeric thresholds to enable measurement of progress.
Example of a progress indicator: indicator NI188 in the United Kingdom
In 2008, an indicator of progress in local adaptation processes was defined at national level in the group of performance indicators for local government and regional bodies in the United Kingdom concerning sustainable development.The scale was scored as follows: Level 0: the process has begun Level 1: elected officials have signed an agreement to act against climate change and a diagnostic has been made of the impacts of climate change Level 2: a comprehensive diagnostic of climate change-related risks is available Level 3: a comprehensive action plan has been defined Level 4: the plan has been implemented, M&E types have been defined
WATCH POINTS
Whenever possible, a target must be chosen before the M&E mechanism, that is, when the action plan is being designed.
The process of defining targets is a critical aspect of the planning process because it has direct implications for the financial, technical and human resources to be mobilised. For that reason, this work requires the participation of different technical and institutional actors from the local authority as well as political validation.
Targets must be correctly designed by crossing several types of considerations such as the importance of certain adaptation issues, the urgency of action, nature of the available inputs, etc. Targets must be neither unattainable nor insufficiently ambitious.
Targets may change. One M&E objective is precisely to assess the degree of completion and possibly redefine targets depending on the effectiveness of the actions implemented.
0 No progress
1 A little progress
2 Underway
3 Objective achieved
4 Objective exceeded
0 No diagnostic Weak
1 Vulnerability diagnostic made Moderate
2 Strategy and action plan validated Satisfactory
3 Action plan implemented
Very satisfactory
Source : ADEME, Suivre et évaluer l’adaptation au changement climatique dans les territoires : Éléments méthodologiques tirés de l’expérience internationale
1 • Binary logic: the indicator is achieved or not, put 0 at the star t and 1 at the end (e.g. existence of climate scenarios, risk prevention plan, etc.) ;
2 • Gradual logic: gradual logic can be applied to a qualitative indicator in relation to the state of progress. For example:
Note that given the intrinsic characteristics of adaptation policies (complex, multi-factor nature of the issues at stake, ongoing climate change, uncertainties, etc.) these targets must be analysed in regard to the territory context.
:
32 33
DESCRIPTION
The monitoring plan must necessarily take in the different levels of policy intervention (strategic objectives, operational objectives, actions and any inputs) as well as the set of related indicators. Each indicator must be associated with a procedure of collection, analysis and diffusion of the information thus produced. This procedure can be based on methods and tools that the local authority already has or they can be formalised in a new scorecard. It must be discussed and validated with the heads of the services involved and/or the monitoring committee put in place by the local authority.
Data collection is intended to identify how the information for the indicators will be collected. It seems important to identify for this purpose:
the owner of the data: in house in local authority services (indicators produced by the local authority, partnerships already in place with bodies, etc.), external (observatories, laboratories, universities, statistics institutes, etc.);
the person responsible for collection in the local authority: define who will be in charge of recovering the data;
the collection method: survey, calculation, questionnaire, qualitative evaluation by experts, etc.;
collection frequency: quarterly, twice yearly, annually, etc.;
associated costs and resources.
Analysis refers to how the information and results are interpreted. An inventory is made for each indicator:
those in charge of the analysis (who may be different from those in charge of collection): strategic offices, technical services, etc.;
analysis frequency: at the start of the programme, each year, mid-term, at the end of the programme, for example;
addressees of the analysis (project leaders, elected officials, monitoring committees, etc.).
The monitoring plan must be kept by the person in charge of process M&E in the local authority.
Observation of the territory in relation to climate change is indissociable from adaptation policy M&E. Indicators of key changes in the territory in relation to the main climate change issues identified must be monitored. These indicators do not provide information directly on the policy but rather on the territorial context (socio-economic, environment and climate). They do not populate the M&E system for the policy as such but are there to compare evaluation results.
DESCRIPTION
Observation of the territory (socio-economic parameters, evolution of flooding, etc.) informs on climate and territorial dynamics likely to interact with the policy and can alert to changes in the context and the implications for adaptation. It can aid understanding of the reasons for the success or otherwise of a strategic indicator and enable identification of critical thresholds beyond which the policy is no longer sufficient and requires fine tuning or even significant modifications.For example, an objective to reduce flooding-related property damage (with changes in the actual amount of damage as the only indicator) may be achieved during the policy cycle not because of the effectiveness of the policy itself but simply because there has been no flooding in the territory during the period in question. In contrast, an objective to reduce vulnerability to mortality caused by high temperatures cannot be achieved, not because the policy is not effective but because socio-economic (growth in the number of elderly people and precarious populations for example) or climate trends (unexpected acceleration of warming) have changed considerably making the original reduction objective obsolete.
Thus observation of context data must fur ther understanding of policy outcomes and furthermore, help to confirm the correct policy path in relation to developments in factors external to the local authority.
For each major goal, it is therefore useful to associate a restricted number of context indicators for collection and analysis throughout the M&E process even if they do not have a target objective.
CHOOSING A SET OF CONTEXT INDICATORS
Monitoring observation of the territory is based on the definition of a set of indicators. They must be chosen in relation to the main climate change issues and vulnerabilities identified in the territory, the territory’s intrinsic characteristics (socio-economic and physical) and policy objectives. The number of indicators must not be too large (5 to 10 in total).
As far as possible indicators that can cover several issues and vulnerabilities at the same time must be identified: for example, monitoring trends in high temperature episodes is useful for health sectors, infrastructures, town planning and organisation of the territory, forests, etc.
Three categories of context can be distinguished:
Climate indicators
These indicators are intended to observe effective changes in the state and variability of the climate and will gradually provide a historical record from which to assess medium and long term trends.
Some local authorities already use this type of indicators for their obligatory competences (e.g. monitoring the heat wave index for health services, forest fire index monitoring for forest services) and they collaborate directly with data providers (Météo France (the French meteorological office) in particular). On this point, it is equally possible to work with Météo France to specify a seasonal climate report tailored to the territory. Beyond that, existing observatories must be used as much as possible as they already have basic relevant indicators: the French national observatory of climate change effects (ONERC), regional and local observatories for particularly vulnerable areas,
METHOD SHEET Implementation Indicator monitoring plan4
NOTE The Objectif ’Climat tool proposes a monitoring plan that can be used to
formalise the data collection and analysis procedure for
each indicator defined in the adaptation policy.
METHOD SHEETTaking into account territorial observation of climate change5
This sheet aims to provide the components of the method for structuring the monitoring plan and its role in the M&E system.
34 35
(climate change observatories in Savoy and the Pyrenees for example).
Indicators of physical impacts of climate change
They are intended to mark the effects of climate change on the territory by observing trends in cli-mate-dependent phenomena and the identified asso-ciated risks (climate is not the only driver of change). These indicators are generally already monitored by some local authority services (environment, ag-riculture, water departments, etc.) by generalist or specialised agencies (Agence Régionale Pour l’Envi-ronnement, Agence de l’Eau, Agence de l’Air, Office des forêts etc.), and research laboratories (INRA; CEMAGREF, etc.). Key topic sectors include: water resources, biodiversity, natural risks, forest and sylvi-culture, the coast, livestock and agriculture.
Indicators of risk factors other than climate change
These indicators involve territory dynamics other than climate change which are factors for controlling territory exposure and sensitivity and can thus help to explain positive or negative trends in an identified vulnerability and the effectiveness of the associated policy. These indicators deal with societal and economic trends (behaviour in the face of risk, structure and age of the population, activities in the territory) and urban dynamics.
Examples of climate indicators:
Heat wave monitoring index (Météo-France)
Forest fire weather Index (Météo-France)
Number of days of very heavy rainfall
Water temperature at a measurement station
Accumulated average rainfall at a measurement station
Number of days of snow
Examples of impact indicators: Population trends (number and geographical distribution) in a characteristic animal species in the territory recognised as a climate change marker (INRA (French National Institute for Agricultural Research), ARPE (French Regional Environment Agencies) ...).
Trends in specific tree populations characteristic of the territory and recognised as a climate change marker (INRA, ARPE...)
Trends in the presence of mosquitoes (inter-departmental agreement on mosquito control, etc.)
Trends in ten-year flood periods. Low flow of a stream or river Heat island intensity Number of days on ozone alert
Examples of risk factor indicators: % of people over the age of 75
Average distance to an air-conditioned space
Memory and perception of flooding risk
Built surface area located on a flood risk area
Trend in average price per square metre
This stage aims to implement both monitoring and evaluation by proposing in particular a monitoring method and standard evaluation grid for the adaptation policy.
Method sheet: Monitoring the adaptation policy
Method sheet: Evaluating the policy and communicating the outcomes
3
METHOD SHEETMonitoring the adaptation policy
1
Monitoring enables local authorities to allocate and use resources, assess the progress of the actions and measure achievement of the outcomes in relation to the initial provisions. It must permit the detection of any differences or deviations and facilitate decision making on whether to continue with or adjust the policy (adjust objectives, redirect actions, etc.).
This sheet is intended to provide local authorities with method components for the monitoring pro-cess and to prepare the evaluation.
IMPLEMENTING THE SCORECARD FOR MONITORING INDICATORS
A scorecard can be used to monitor policy progress continuously and thus assess the extent of achievement of objectives and action programmes.
To measure progress, it seems necessary to define for each indicator at least:
the measurement unit;
indicator code: if it is not numeric, it might be appropriate to associate it with a numerical code to facilitate data collection and analysis (e.g. : ID1, ID2, etc.) ;
value measured by the indicator (expressed quantitatively even if the indicator is qualitative, for example, based on a predefined scale) for this reference value will not change over time.
expected value (forecast) of the indicator for the target objective and its time horizon (e.g. end of the action programme). This value can be revised upwards or downwards if the adaptation plan is revised during the programme. As with the choice of indicators, the attached target objectives are generally validated by the monitoring committee and/or evaluated by the local authority.
effective measurement of the indicator (completed), updatable at various moments of the programme in relation to the collection frequency defined in the monitoring plan for each indicator. In order to ensure dynamic, regular updating of the scorecard, it is impor tant to associate collection dates and future dates. This indicator value will determine the rate of completion of the end objective for each level of intervention.
measurement of indicator progress rate: indicates by simple calculation, the differential between the measured and expected values of the indicator ;
3 Refer to the sheet “ Help with the selection and measurement of adaptation indicators for methodological components to
define a target objective.
IMPLEMENTING M&E IN A LOCAL AUTHORITY
The sheets proposed in this part are:
FURTHER INFORMATION ON TERRITORY INDICATORS
The French National Observatory of Climate Warming Effects (ONERC) proposes a certain number of thematic climate change indicators (forest, biodiversity, health, etc.) : http://www.developpementdurable. gouv.fr/Indicateurs-du-changement,29789.html
Impact’Climat user guide (ADEME) also proposes potential thematic indicators of climate change.
Source : Based on UNDP, 2009, guide méthodologique d’élaboration des politiques sectorielles.
Month/Quarterly/Year
Name of the Serviceand the people in charge of collectionand/or analysis
Programme
Intervention field Code : theme 1
Observations:
Strategicobjective Code 1 Title
Indicator Code Name
Indicator measurementInitial
Text including methodology, analysis and any recommendations
Forecast Completed
Observations
Operational objective Code 1.1 Title
Indicator Code Name
Indicator measurementInitial Forecast Completed
Observations Text including methodology, analysis and any recommendations
Action Code 1.1.1 Title
Indicator Code Name
Indicator measurementInitial Forecast Completed
Observations Text including methodology, analysis and any recommendations
order to better assess adaptation policy progress (the an-ticipated indicator value and effective measurement and analysis of the indicator in the chosen time hori-zon should also be provided). This stage can represent an additional workload for the local authority that must be assessed in relation to its resources and inputs dedicated to policy M&E.
36 37
Evaluation is the second pillar of any continuous improvement process and adaptive management of an adaptation process. Evaluation is intended to assess policy performance systematically and objectively in order to learn lessons and correct objectives, inputs and methods in the light of the results of the experience.
DESCRIPTION
Evaluation is a strategic steering instrument for public action and in this regard it is a crucial stage in the life cycle of a local adaptation policy. It generally intervenes in three major moments of implementation.
Before (ex-ante evaluation)Evaluation takes place beforehand to support definition of the strategy and adaptation action plan to help with the choice of goals, objectives, actions and inputs. This is the particular objective of the Climat Practic tool developed by ADEME.
During (mid-term or in itinere evaluation):Evaluation is therefore a management tool for measuring policy performance from the
PERIODIC MONITORING REVIEWS
To measure and analyse adaptation policy progress, monitoring can be reviewed periodically (annually for example) with the monitoring committee or the various operational managers involved in the
indicator analysis (differences or achievements of observed objectives etc.), assessment (consequences, levels of seriousness, possible risks) and possible decisions (envisaged corrective actions, increase in the objectives, etc.).
Whereas indicators can be analysed during collection, assessment and decisions should generally be the object of extended validation, for example by the monitoring committee.
Finally, the local authority can choose to add inter-mediate objectives to some or all the indicators in
REGULAR COLLECTION AND UPDATING OF MONITORING DATA
When the monitoring plan (cf. sheet “ Implementing a indicator monitoring plan “) and the scorecard are in place, the person in charge of monitoring can collect and enter the indicators on the scorecard in a relatively continuous manner in relation to the identified deadlines (collection frequency, stage objectives, planned monitoring or evaluation committees ). A detailed monitoring sheet can be used for this collection with the different stakeholders.
Example sheet:
Figure: 14 Example indicator monitoring sheet METHOD SHEETEvaluating my policy and communicating the outcomes
2
collection and assembled in a less formal body. The project manager can use the scorecard and prepare a presentation and a periodical monitoring report for this purpose, presenting the state of progress of the indicators (possibly supported by graphs or scorecard extract), analysis of progress or any delays observed from the operational perspective.
NOTE By way of illustration the Objectif ’Climat
tool provides an automated scorecard
with all these characteristics.
perspective of implemented actions and as well as governance of the process. It permits continuous adjustment of the process.
After (ex-post evaluation) At the end of the process, it is intended to provide a complete strategic overview of the effects of the policy and to prepare the next cycle from a perspective of continuous improvement
The recommendations in this present sheet are directed mainly at these last two stages.
The evaluation process is an integral par t of the continuous improvement process (cf. Figure 15) and is organised around three main elements:
1 • Evaluation in the strict sense, which measures achievement of objectives and impact,
2 • Communication of the outcomes to decision-makers and stakeholders, who provide recommendations to clarify the decision on necessary adjustments
3 • Planning review, so that strategic and operational adjustments can be integrated into policy monitoring or the next cycle.
EVALUATION
Review / correction
ImplementationCommunication to decision-makers
Evaluation Monitoring:
Planning
38 39
PERFORMANCE EVALUATION (IN THE STRICT SENSE)
This stage consists in casting a critical eye over different aspects of the adaptation policy in order to assess effects, efficiency, quality of governance, appropriation perspectives, etc. This work, carried out transversally and with full stakeholder par ticipation ¯ will provide a global vision of policy performance and thus provide an evaluative opinion resulting in recommendations for continuous improvement of the process.Evaluation of performances is based on an inquiry process intended to highlight strong and weak points and identify room for progress and corrective measures. This analysis is generally based on a variety of evaluation criteria. The methodology guide
provides an analysis grid based on 6 criteria:
Adaptive management: Does the policy take into account the main principles of climate change adaptation? How does it take into account future climate uncertainties? What place has it reserved for learning and innovation?
Consistency: is the adaptation policy consistent with already existing policy? Relevance: are the policy objectives and inputs aligned to the issues in the territory? Governance: What are the links between actors? Do the stakeholders participate in decision making? Does the project have political support?
Effectiveness: Based on the diagnostic of vulnerability of the territory, what actions have been taken or are in progress? Have they enabled achievement of policy objectives?
Utility: What risks in the territory have been effectively reduced? What are the long term impacts and perspectives for policy effects?
Sustainability: Are the technical, financial and institutional capabilities in place to provide the policy with a long term perspective? Will policy effects subsist beyond local authority intervention?
4 What is known as “participative” evaluation is nowadays the object of broad consensus in the field of evaluating territorial sustain-able development projects . The present methodology also encourages application of the principles and methods of this approach to adaptation policies and invites the reader to refer to existing documentation on the subject.
Different themes and sub-themes are attached to each of these criteria with several evaluative questions to orient the local authority on these issues. The table in Figure 16 provides examples of evaluative questions available to local authorities.
A local authority does not have to respond to all the evaluative questions proposed here.
It is advisable to choose a maximum of 5 to 6 evaluative questions that the local authority deems especially relevant.
Figure 15 Evaluation in the continuous improvement process Local authority responses to the chosen evaluation questions must come from a self-evaluation or in house evaluation process, that is, essentially from the actors involved in the policy, whether they are the originators or the direct and indirect beneficiaries of the policy.These responses must be the result of perspective and comparison of the different points of view expressed by territorial actors, especially through the various participative evaluation mechanisms which can be mobilised in order to reduce differences between value judgements and highlight points of convergence and divergence between stakeholders. The aim is to overcome the inherent subjective nature of this type of exercise and formulate a shared evaluative opinion endorsed by policy managers.Whatever criteria and evaluative questions are chosen, the responses must be based on a set of data and information collected and formatted beforehand:
vulnerability diagnostic data
strategy and action plan data,
data from the monitoring system itself, scorecard indicators and observation of the territory.
data in studies, status reports, minutes from meetings, workshops and any other available source of information.
It is also important to listen to key policy actors (public sector contractors, service providers, institutional and associational partners, direct and indirect beneficiaries, etc.) to gain a fresh look at policy performance and put the collected data into perspective Similarly, the local authority may have access to complementary studies in the form of questionnaires, interviews or field visits to supplement certain information that is missing or to provide deeper examination.To ensure an objective evaluation, responses to the questions may be scored qualitatively. These scores can then be used to represent the strengths and weaknesses of the process and to produce graphs and messages for the public at large.
These questions must be chosen in relation to the level of policy progress and be agreed by the M&E body set up by the local authority.Questions must be adapted to the territorial context, to the degree of implementation of the process and reflect the main dimensions that the local authority wants to find out about. For example, the local authority may want to ask about effectiveness and sustainability criteria in the context of an ex-post evaluation, given the necessary perspective required to respond.
40 41
CRITERION: ADAPTATION CHARACTERISTICDoes the local adaptation policy take into account the major principles of climate change adaptation?
Resilience to uncertainty
To what extent do the local adaptation policy and actions take into account climate change uncertainties?Are the objectives and measures effective under the different climate change scenarioshighlighted for the territory?
Progressive/flexiblereversible in time
Can the implementation policy be adjusted or redirectedat an acceptable cost in relation to developments in climate change and scientific knowledge?
“No regrets” measures and strategiesDoes the policy help to reduce vulnerability to climatechange with immediate benefits, regardless of future climate trends?
LEARNING FUNCTION
Local authority exchange and learningIs there time to meet and collectively exchange and reflect on the local adaptation policy and its performance?
Adaptation training Does the policy involve ongoing training of agents, elected officials and key actors in adaptation approaches and tools?
Exchange of experiences and good practice
Is the policy based on exchange of experiences and research into good adaptation practices especially through participation in national and/or international networks?
Trends in philosophy and action
Does the policy lead to changes in territorial development approaches, decisions and practices in the local authority and its partners?
INNOVATION PROCESS
Experiments/ innovative actions
Does the policy encourage experimentation with approaches and innovative adaptation measures to better prepare for the effects of climate change?
Capitalisation Are the innovations and their effects on adaptation part of a capitalisation process that can enrich and improve the policy over time?
Theme Sub-theme Examples of evaluative questions
ADAPTIVE MANAGEMENT
Figure 16: Examples of evaluative questions CRITERION: CONSISTENCYIs the adaptation policy consistent with existing policy?
EXTERNAL CONSISTENCY
OF THE CLIMATE CHANGE
ADAPTATION POLICY
Aligned with orientations at other territorial levels
To what extent is the climate change adaptation policy consistent with the strategic orientations of other territorial levels (Department, Region, Inter-region, etc.)?
Aligned with government orien-tations
To what extent is the climate change adaptation policy aligned with the strategic orientations of the government?
Organisation with other territorial adaptation programmes and projects
Is the climate change adaptation policy well coordinated with the other adaptation processes in the territory (water agency, national forest service, energy sectors, etc.)?
INTERNAL CONSISTENCY
OF THE CLIMATE CHANGE
ADAPTATION POLICY
Internal logic of the policy
Are the components in the climate change adaptation policy defined and organised logically, respecting a cause and effect relationship?
How do each of the different objectives logically and ultimately contrib-ute to the adaptation policy?
Suitability of inputs and tools
To what extent are the technical, human and financial inputs mobilised by the policy consistent with its ambitions?
Theme Sub-theme Examples of evaluative questions
CRITERION: RELEVANCE
Are your policy objectives and inputs tailored to the issues in the territory?
RELEVANCE OF THE CLIMATE
CHANGE ADAPTATION
POLICY
Relevance with regard to the territory’s adaptation issues
Are the themes addressed by the local climate change adaptation policy relevant in relation to the main vulnerabilities of the territory and its people?
Quality of the vulnerability analysis
Is the adaptation policy based on adequate analysis of vulnerabilities?
Theme Sub-theme Examples of evaluative questions
Specific adaptation issues
42 43
CRITERION: GOVERNANCE
OPERATIONAL MANAGEMENT
Organisation To what extent are the roles of the different partners sufficiently well-identified?
Project management Is the role of project manager consistent with its competences and ambitions?
Local authority project team and services
Do the services involved fully perform their tasks? Do they have sufficient technical and human capabilities?
Transversality Has the policy permitted coordination of climate change adaptation actions, competences, and strategies?
Adaptive approach to policy conduct
Has the policy adopted an adaptive, flexible approach to the conduct of the actions? Have corrective measures been taken when constraints, risks or important needs have emerged?
STRATEGIC STEERING
Political support Does the policy have the benefit of strong political commitment from elected officials?
Strategic supervision and orientation
Do the steering committee and other decision-making bodies engage in their role of supervising and strategically orienting the policy?
Inter-institutional coordination Does policy management ensure the necessary coordination between the different institutional partners involved?
TAKING INTO ACCOUNT
TERRITORIAL STAKEHOLDERS.
Stakeholder participation in policy definition
Have stakeholders been involved in defining the climate change adaptation policy and actions?
Stakeholder participation in implementation
To what extent is policy governance based on effective participation from the various categories of actors involved?
Theme Sub-theme Examples of evaluation questions
CRITERION: EFFECTIVENESS
To what extent has the climate change adaptation policy achieved its objectives?
COMPLETION OF OBJECTIVES AND
ACTIONS
Achievement of strategic objectives To what extent have policy objectives been achieved?
Achievement of operational objectives Have the operational outcomes been achieved?
Implementation of actions
Have the planned actions been carried out according to expectations?
Have the actions implemented produced the anticipated effects?
Could greater impact have been achieved using different instruments or measures?
Relevance of responses to the identified issues
Have the actions in the climate change adaptation policy helped to meet the intended objectives?
Perspective on policy outcomes To what extent have policy outcomes been compared to developments in the territorial and climate context during the implementation period?
CONSIDERATION OF BRAKES ANDOPPORTUNITIES
BrakesTo what extent have economic, institutional or climate brakes (e.g. more brutal or drastic changes in certain weather hazards) been analysed and managed throughout implementation?
OpportunitiesTo what extent have opportunities, including climate change-related opportunities (e.g. more favourable conditions for certain economic activities, etc.) been analysed and taken into account during policy implementation?
Theme Sub-theme Examples of evaluative questions
CRITERION: UTILITY
What are the long term impacts and perspectives for policy effects?
Theme Sub-theme Examples of evaluative questions
Specific adaptation issues
CONTRIBUTION TO TERRITORY ADAPTATION
Impact on the territory’s vulnerability to climate change (activities, property, people, etc.)Does the policy contribute or is it heading in the right direction to contribute towards reducing territory vulnerability?
Are the expected and unexpected effects of my adaptation policy globally satisfactory from the perspective of the population and any indirect beneficiaries of my policy?
Impact on adaptation abilities Is the policy contributing or is it on track towards contributing towards reinforcing adaptation capabilities in the territory and its actors?
IMPACTS ON TERRITORIAL AND SECTOR
POLICIES
Impact on the content of other policies Has the policy enabled local planning and development policies to take greater account of the impacts of climate change (e.g.: land development and town planning etc.) ?
Impact on programming and investment operations Has the local climate change adaptation policy led to a re-evaluation or positive re-direction of certain particularly vulnerable investments or projects (e.g. construction of transport infrastructures, urban development schemes, etc.)?
IMPACTS ON POPULATIONS
AND THE GENERAL PUBLIC
Impact on the general public Has the policy contributed to greater sensitisation of the general public to climate change adaptation issues?
Impact on climate risk perception and attitudes Has the policy led to changes in perception and behaviour in favour of greater prevention and preparation in the face of certain major climate risks?
44 45
CRITERION: SUSTAINABILITY
Has the basis for a sustainable climate change adaptation policy been laid?
INSTITUTIONAL AND TECHNICAL SUSTAINABILITY
Sustainability of policy benefits and impacts Will policy effects subsist beyond local authority intervention?
Long term reinforcement of technical, administrative and financial capabilities
Has the policy enabled development of the necessary institutional, technical and administrative capabilities for a long term adaptation policy (e.g. functions of observation and hazard forecasting, risk evaluation and response planning and organisation, coordination, etc.)?
Reinforcement of human resources Have human resources been trained to support a long term process?
Local policy networks Are “network institutions or actors able to perpetuate or multiply policy benefits?
ECONOMIC AND FINANCIAL
SUSTAINABILITY
Taking long term costs into account
Are the costs of the climate change adaptation policy under control and can they be sustained in the long term?
Sustainable financing Are the institutions responsible for introducing the policy ready to assume their financial and economic responsibilities?
Implementation of financing mechanisms
Has the policy implemented financial inputs or mechanisms to enable the process to continue in the long term?
SOCIAL SUSTAINABILITY
Social acceptance To what extent does the general public support and accept the policy?
Long term integration of adaptation principles by stakeholders in the territory.
To what extent has the policy been appropriated by stakeholders in the territory (citizens, businesses, associations...) ?
Multiplier effectDo any local initiatives that contribute to territory adaptation have long term stakeholder support? Is there an adaptation network in the territory outside the policy?
POLITICAL SUSTAINABILITY
Sensitisation and awareness of decision-makers
What is the degree of awareness and mobilisation of political decision makers on adaptation issues after the policy?
Political support What type of support and how much involvement has there been from elected officials?
ENVIRONMENTAL SUSTAINABILITY
Taking environmental impacts into account
To what extent has the policy sought to minimise the potential impacts on the environment?
Compatibility with attenuationAre the measures promoted by the local climate change adaptation policy compatible and consistent with the attenuation strategies and actions for the territory?
COMMUNICATION OF OUTCOMES AND RECOMMENDATIONS FROM THE EVALUATION
Evaluation does not just analyse the benefits and shortcomings of a policy. It resolutely focuses on action in order to feed into the ongoing improvement process. Its ultimate aim is to maintain and multiply the process of change supported by the territorial adaptation process.
Performance evaluation findings will therefore be formalised and presented so that they contribute to political and technical managers’ considerations of any adjustments and reorientations. They will take the form of an evaluative opinion that will explicitly highlight the sticking points and agreement and disagreement between actors questioned on policy implementation and the way it has been conducted. This opinion will be accompanied by proposals for improvement to be included in the programming and execution of the actions.
Different types of mechanisms are available to local authorities for the process of communicating outcomes and decision making. Thus they can:
Constitute a permanent M&E body for their policy;
Implement ad hoc evaluation groups for the themes to be evaluated :
Use “expert evaluation facilitators” to guarantee the process and successful execution of the evaluation;
Share outcomes in an extended way in territorial assemblies or any other mode of participation involving the general public.
The methodology guide “L’évaluation au service du pilotage d’un Plan Climat-Énergie Territorial” (Evaluation for managing a climate and energy territorial plan), produced by RhonAlpEnergie Environnement and Cabinet Argos proposes a set of principles and methods to bring the various actors into the evaluation process and implement a suitable framework for the operational and strategic valuation of the evaluative work.
PLANNING REVIEW
When changes recommended by the evaluation have been studied, compared and finally accepted by those in charge, the project leaders and agents in charge of the policy must then include them in the planning of objectives, actions and inputs. This is the final stage in the process which will be used to consolidate, adjust or establish a new course for the adaptation path pur-sued by the territory until the next cycle.
Specific adaptation issues
Theme Sub-theme Examples of evaluative questions
46 47
Problem and solution diagrams (still called “problem trees” and “solution or objective trees”) can be useful tools for helping local authorities move beyond their diagnostic of the consequences of climate change towards the formulation of objectives and adaptation actions.
The problem diagram is intended to analyse the negative aspects of a situation and the cause and effect relations between different problems. In particular, it is about identifying and then relating:
the main problem to be solved (in general, increased vulnerability of sectors or territories in the face of climate change) ;
sub-issues linked to the main problem (reduced water resources, increased loss of human life due to high temperatures, for example ;
root causes of these problems (increased exposure to the hazard, lack of adaptive ability to deal with problems, etc.)
When the tree has been created and verified, a problem-solving strategy must be proposed which can be based either on an existing adaptation strategy or formulated in house or outsourced by the local authority.
Below is an example concerning vulnerability of water resources
ANNEXE 1 Problem and solution diagrams for planning climate change adaptation
1
ANNEXES
48 49
Fight against eutrophication of
water masses
Reduction in economic and human issues in risk areas
Reduced exposure to 10-year floods
Urban development in flood risk areas controlled by greater
control over building permits.
Accrued effectiveness of protection works
Redesign and reinforcement of
worksThe atlas of flood risk areas is reviewed on the basis of improved understanding of climate change.
Strategic objectives Reduce flood-related damage to property and persons.
Preservation of water resource quality
Maintenance of water resource availability (supply and demand balance)
Core issue
Main causes Increased flood-related damage to property and persons
Degradation of water resource quality
Reduction in water resource availability(supply-demand imbalance)
The territory’s water resources and hydrology are affected by climate change
Increase in economic and human issues in risk areas. Increased exposure to 10-year floods
Insufficient control of urban development in flood risk areas
Accrued ineffectiveness of protection works
The municipality does not have an updated flood protection plan
Undersized works
Insufficient maintenance of works
Atlas of flood risk areas does not reflect the impact of climate change on
the reference hazards.
Degradation of wetlands and loss of natural flood expansion areas
Concentration of organic and toxic
contaminants
Change in the physico-chemical
properties of ground and
surface waters.Algae and bacterial blooms
Reduced oxygen and eutrophication
Increased temperature
of ground and surface water
bodies
Reduced groundwater recharge
Reduced minimum flows
Increased water demand
Root causes
Change and intensification of the water cycle: increased evapotranspiration, concentration of rains, reduced annual rainfall, increased hydrological variability, etc.
FROM THE PROBLEM DIAGRAM...
...TO THE SOLUTION DIAGRAM Goal Improve quantitative management and qualitative management of water resources and the extremes
Operational objectives
Actions The municipality has an updated flood protection plan.
Restoration and protection of wetland areas and natural flood expansion areas
Works audit and maintenance
programme.
Change and intensification of the water cycle: increased evotranspiration concentration of rains, reduced annual rainfall, increased hydrological variability, etc.
Maintain physico-chemical properties
of ground and surface waters.
Fight against organic and toxic
contaminants
Monitoring and control of factors conditioning water
temperature
Development of preventive and
curative techniques against algal and
bacterial proliferation
Review plans and rules on freatic groundwater management
Maintenance of minimum flows
Control over water demand (watersaving measures, efficient use)
50 51
This sheet proposes an overview of possible strategies and actions and guidance on selection criteria so that so that local authorities can prioritise their actions, verify their resilience and relevance in relation to climate change and the local and specific issues.
DEFINITION AND TYPE OF ADAPTATION STRATEGIES AND ACTIONS
Adaptation actions serve two main objectives of climate change adaptation policies: direct reduction of vulnerability (reduction of exposure and sensitivity to weather hazards) and the reinforcement of capabilities (implementation of an urban cooling mechanism with low environmental impact, for example) .They can be very diverse according to the chosen strategies and impacts already observed by local authorities: technical, legislative and regulatory, institutional or deepening knowledge, sensitisation and communication, etc. They also concern a large number of sectors and topics; planning, urban development, forestry, water resources management,
ANNEXE 2Help with selecting adaptation strategies and actions2
Below are types of adaptation actions according to major strategic orientations:
Major strategic orientations
Figure 17: Type of strategies and adaptation actions
Source : Translated into French and adapted from Lamhauge, N. E. Lanzi and S. Agrawala (2012), Monitoring and Evaluation for Adaptation: Lessons from Development Cooperation Agencies, OECD Environment Working Papers, No. 38, OECD Publishing.
These strategic orientations can also be specified by sector. Below is a panel of examples of actions for sectors considered as particularly vulnerable to climate change:
Figure 18: Type of strategy and types of adaptation action by sector
Sectors
Flood control and prevention
Agriculture
Fisheries
Forestry
Natural risk prevention and
protection
Water and sanitation
Health
Develop research and improve understanding of climate, impacts and adaptation
Reinforce transversal coordination of climate change actions
Educate, train and sensitise
Integrate climate change in institutions and policies
Reduce climate risk with technical actions and management
Source: Translated from the French and adapted from Lamhauge, N. E. Lanzi and S. Agrawala (2012), Monitoring and Evaluation for Adaptation: Lessons from Development Cooperation Agencies, OECD Environment Working Papers, No. 38, OECD Publishing.
Actions to develop studies on climate change, scenarios, impacts and their modelling, tools and equipment necessary for improved understanding of vulnerability and possible adaptation strategies
Studies of the impact of climate change on flooding risk, probabilistic flooding risk forecasts
Meteorological studies of rainfall pattern forecasts
Studies of the impact of climate change on fisheries
Research into the impact of climate change on certain tree species
Climate scenarios for forecasting extreme weather events
Research into the impacts of climate change on water quality
Research into the impact of climate change on health
Coordination between decision makers, individuals and social security
Sensitisation campaigns on the importance of vaccinations
R&D incentives, creation of codes, insurance, disease control requirements
Vector control programmes, disease eradication programmes, R&D in vector control, vaccinations
Coordination between water policy authorities and research institutes
Education on the importance of potable water use
Water quality certificates, water pricing
Water distribution, water conservation, seawater desalination infrastructures
Coordination between natural risk management authorities and local actors
Training in early warning systems and protection against natural risks
Building codes, zone planning, insurance systems
Deployment of early warning systems, insurance, zone planning, improved natural disaster management
Coordination between forest managers and inhabitants
Training in forest management
Regulating deforestation
Diversification of species, planting to prevent soil erosion
Coordination between industry, research and decision-makers
Training in aquaculture
Fishing quotas, regulating fishing gear
Adaptation of stock to variation, targeting species, changing species
Coordination between farmers and investigators into crop resistance to drought
Training in the use of irrigation technologies
Water pricing, water yield requirements, R&D incentives, incentives to adopt more efficient technology, insurance
Agricultural insurance (for harvests), rainwater collection
Coordination between citizens, flood prevention authorities and the government
Training in early warning systems, sensitisation to possible future impacts of climate change on flooding risk
Zone planning, insurance systems
Dykes and coastal protection, salt water intrusion barriers, relocation of activities
Develop research and improve understanding of climate, impacts and adaptation
Reinforce transversal coordination of climate change actions
Educate, train and sensitise
Integrate climate change in institutions and policies
Reduce climate risk with technical actions and management
Actions to create networks and links between institutions, participation in debates and dialogues between stakeholders and policy makers, to reinforce the sharing of good practices favouring shared learning and adaptive management.
Actions to disseminate information on climate change-related risks, training intended to reinforce the capabilities of actors, sectors and institutions, education focused on profound changes in behaviour (climate change education in schools for example)
Actions to integrate climate change in legislative and regulatory frameworks (e.g: risk prevention plan), in institutions (e.g. creation of a climate/energy cell) and in sector policy processes.
Actions to reduce exposure and sensitivity to hazards in the territory, like conserving water in the ground, growing techniques, designing protection infrastructure, abandoning areas that are particularly vulnerable to the hazards of submersion/flooding, etc.
Description of types of action;
health, etc. and can treat issues tranversally or by sector and theme of interest.
Given the exploratory nature of adaptation polices, uncertainties over climate and impacts and a lack of feedback on policy effects, the approach to defining adaptation options in the various existing national and local level plans is generally prudent and gradual.
1 • Chosen actions are generally intended to improve knowledge of forecasts and impacts, reinforce coordination between different actors, sensitise and train;
2 • In parallel, complementary options to reinforce expertise and action capabilities in institutions are implemented;
3 • They are followed by institutional and regulatory actions and technical and management measures, mainly of the no regrets type which have not yet been used much for adaptation.
52 53
FURTHER INFORMATION ON
A selection of guides and a catalogue of actions is provided to offer guidance with your choice of adaptation actions:
Adapting to climate change, 12 local authority action sheets (ADEME, Languedoc-Roussillon Regional Office). This compilation of actions covers a broad spectrum from the field of local authority competences and has been designed as a tool box, for use by elected officials, mayors and presidents of public inter-municipality cooperation establishments as well as their regional directors.
Produce and implement an adaptation strategy or action plan in a territory: Methodological components drawn from international experience (ADEME): this document presents the experience of ten territorial local authorities at international level which have produced climate change adaptation strategies.
Adaptation tool box proposed by ALTERRE-Bourgogne in partnership with ADEME who provides in particular sector sheets on impacts, vulnerabilities and adaptive measures
The European climate change adaptation platform (Climate-Adapt) offers a database on different adaptation options for priority sectors (health, urban development, etc.) or impacts identified by territories (drought, flooding, etc.).
CRITERIA FOR CHOOSING ADAPTATION ACTIONS
To enable action when faced with uncertainty and maintain a certain degree of flexibility and reversibility in the choice of measures and thus permit changes in policy as knowledge improves and experience accu-mulates, a variety of selection criteria are proposed. Actions must be:
Robust: that is they must operate within as large a range of future climates as possible and not only for one type of scenario (e.g. investment in safety margins during changes in water drainage networks);
“No regrets”,: that is, they must help to reduce vulnerability to climate change while delivering immediate benefits, regardless of future climate change and uncertainties (e.g. improved building standards to deal with higher temperatures);
Flexible/reversible: that is, it must be possible to make adjustments at acceptable costs as the future is revealed and understanding improves (e.g. reducing water demand rather than introducing additional reservoirs);
In synergy with attenuation objectives and other environmental policies in order to avoid the risks of conflict and incompatibility (e.g.: building insulation generates a double dividend in terms of adaptation and attenuation);
NOTE The Objectif ’Climat tool
developed by ADEME offers a simplified module to help
select adaptation actions based on the elements introduced
here which can be tailored to the local authority’s needs.
Favouring financial and institutional local authority mechanisms that are less costly and carry less inertia in the face of uncertainties (e.g. in the face of flooding risks, prefer to develop an early warning and evacuation system rather than building dykes, long term perspective).
Other criteria can also help decision making:
Social acceptability of the measures;
Costs/benefits that can be qualitatively estimated in certain cases or through a more in-depth study.
In all cases, the final choice of adaptation measures must be the object of consultation with the different services and actors involved in their implementation. Local authorities can build and use a grid to help select actions based on the above criteria.
54 55
Adjustment of natural or human systems to a new or changing environment. Adaptation to climate change requires changes in natural or human systems in response to harmful stimuli or permits the exploitation of beneficial opportunities. There are various types of adaptation, in particular anticipated and reactive adaptation, public and private adaptation and autonomous and planned adaptation (GIEC, 2001). Adaptation is in addition to and not in place of reductions in green house gases.
ADAPTATION
A weather hazard is a weather or weather-related event which is likely to happen (with a more or less high probability) that can cause damage to populations, activities and environments. Hazards may be either trends or weather extremes.
HAZARD
Attenuation designates the actions intended to limit the breadth of climate change by reducing direct and indirect greenhouse gas emissions. It involves in particular reduced energy consumption, improved energy performances and significant use of renewable resources.
ATTENUATION
Methodological tool for evaluating public policies and programmes, permitting the identification of evaluation issues and indicators.
LOGICAL FRAMEWORK
Ability of a system to adjust to climate change (including climate variability and extreme weather) in order to attenuate potential effects, exploit opportunities or deal with the consequences (GIEC, 2001).
ADAPTIVE CAPACITY
Climate change refers to statistically significant variations in the average state of the climate or of its variability persisting over long periods (generally decades or more) (GIEC, 2001).
CLIMATE CHANGE
Policy sustainability is evaluated by the long term effects and sustainability of outcomes and their effects.
SUSTAINABILITY
GLOSSARY
56 57
The effectiveness of a policy is evaluated by evaluating the outcomes of the action in relation to the objectives.
EFFECTIVENESS
The efficiency of a policy is evaluated by evaluating the outcomes of an action in relation to the inputs allocated to it.
EFFICIENCY
Evaluation is the strategic steering instrument for public action. It is a governance tool that can be used to measure and analyse the effects of the policy.
EVALUATION
Nature and extent to which a system is subject to significant changes in climate over a certain duration (a time horizon of 10, 20, etc. years).
EXPOSURE
A systematic process of continuous improvement of policies and management practice based on lessons drawn from the outcomes of previous policies and practices (UN, 2001). Adaptive management is a way of acting in a context of uncertainty like climate change.
ADAPTIVE MANAGEMENT
Expression of the degree to which a value (the future state of a weather system, for example) is unknown. Uncertainty may be due to a lack of information or disagreement over what is known, or even over what can be known. It may have a variety of origins, ranging from quantifiable errors in data to concepts or terminology with ambiguous definitions and forecasts or projections of human behaviour. Uncertainty can thus be represented by quantitative measures ( a range of values calculated by different models, for example) or by quantitative statements (reflecting the opinion of a group of experts). (See GIEC, 2001).
UNCERTAINTY
Evaluating impacts is to evaluate the direct, indirect and induced effects of policy outcomes.
IMPACT
Quantified or qualitative information or set of information that enables observation of the progress or outcomes of a policy.
INDICATOR
Any changes in human or natural systems which inadvertently increase vulnerability to climatic stimuli: adaptation which increases vulnerability instead of reducing it (GIEC).
MALADAPTATION
Relevance is evaluated by examining the alignment between policy objectives and the specific features of the situation on which it is supposed to act.
RELEVANCE
The Territorial Climate and Energy Plan (PCET) is a territorial sustainable development plan whose main goal is to combat climate change. Instituted by the French national climate plan and taken up by Grenelle 1 law and the Grenelle 2 bill, it is the territory’s framework for commitment.
TERRITORIAL CLIMATE AND ENERGY PLAN
Ability of an ecosystem to absorb disturbances while maintaining its structure and be-haviour. It can be defined as the ability to protect from these disturbances, to self-organ-ise and increase the ability to learn and adapt (Berkes, 2007).
RESILIENCE
The regional climate air and energy scheme (or SRCAE) is one of the major regional schemes created by Grenelle I and Grenelle II (Article 681) as part of the follow up to the 2007 Grenelle Environment Forum. It also includes some of the content from European legislation on climate and energy at regional levels
REGIONAL CLIMATE AIR and ENERGY SCHEME
Component concerning one of the divisions of public action within a local authority or government, generally corresponding to a structuring of the economy.
SECTOR
Sensitisation to climate change refers to the proportion in which a component (local authority, organisation...) exposed to climate change is likely to be favourably or un-favourably affected by a hazard. The effects or impacts of a hazard may be direct (for example changes in agricultural outputs linked to a change in the average value, range or variability of temperature) or indirect (for example damage caused by increasingly frequent flooding in coastal areas due to higher sea levels). (see OECD, 2010).
SENSITIVITY
Monitoring is a continuous process of collecting, analysing, using and communicating information on the state of policy progress and execution (UNDP, 2009).
MONITORING
Compilation of information and periodic statements to find out about and make known the progress and effects of a programme or policy. The term M&E underlines the need to not separate evaluation from the compilation of monitoring indicators.
M&E
Degree to which a system is exposed to the adverse effects of climate change or of being negatively affected. Vulnerability depends on the nature, extent and rate of climate change to which a system is exposed, as well as on its sensitivity and adaptive capacity. (GIEC, 2001)
VULNERABILITY
58 59
PERSONAL NOTES
ADEME20, avenue du Grésillé BP 90406 I 49004 Angers Cedex 01
www.ademe.fr
8723
-
Dec
embe
r 20
13 -
Lay
out
& il
lust
ratio
ns :
RC
2C
ADEME IN SHORTThe French Agency for the Environment
and Energy Control (ADEME)
participates in implementing public
policies in the fields of the environment,
energy and sustainable development.
To promote progress in environmental
process, the agency provides businesses,
local authorities, public powers and
the general public with its expertise
and advice. It also helps with project
funding and implementation research in
the following fields: waste management,
ground conservation, energy efficiency
and renewable energies, air quality and
the fight against noise.
ADEME is a public establishment under
the umbrella of the French Ministry
of Ecology, Sustainable Development
and Energy and the Ministry of Higher
Education and Research. www.ademe.fr
With this guide, the French Environment and Energy
Control Agency (ADEME) provides a methodology for
monitoring and evaluating climate change adaptation
policies. This methodological guide has been designed
as a tool box, organised in the form of “method” sheets
that provide methodological advice on organising,
introducing and successfully implementing monitoring
and evaluation. The proposed methodology is intended
to be accessible to all local authorities and can be
used either as an integral M&E mechanism or as an
occasional or regular aid to integrate adaptation into
their pre-existing M&E mechanisms.
Liberty Equality Fraternity
F R E N C H R E P U B L I C
MINISTRY OF ECOLOGY, SUSTAINABLE DEVELOPMENT
AND ENERGY
MINISTRY OF ECOLOGY, HIGHER DEVELOPMENT AND RESEARCH
French Environment and Energy Management Agency