is monitoring and evalution a challenge urban mobility
TRANSCRIPT
Microsoft Word -
Challenge_Description_WP5_20140128_final.docxWhy is monitoring and evalution a challenge
in sustainable urban mobility planning?
Authors: City of Dresden Dr Kerstin Burggraf Institute for Transport Studies University of Leeds Dr Astrid Gühnemann Contributions from: Institute for Transport Studies University of Leeds Prof. Anthony D May OBE FREng January 2014
2
2 What is meant by monitoring and evaluation? ............................................................................. 3
4 What research and information is available on monitoring and evaluation? ............................... 5
5 How does CH4LLENGE address monitoring and evaluation? ........................................................ 8
6 Further reading ............................................................................................................................ 10
1 Aims of CH4LLENGE
The EU cofunded project CH4LLENGE addresses significant barriers for the wider takeup of SUMPs
in Europe. In a joint undertaking together with research and resource institutions, the project will
support European cities at different stages of advancing the takeup of SUMPs. Building on previous
experiences and lessons from earlier and ongoing national and European SUMP initiatives, the
consortium has identified common challenges which pose significant barriers in the wider takeup of
SUMPs in Europe. The project will work on innovative and transferable solutions for four SUMP
challenges.
The following sections explain in detail why monitoring and evaluation in particular is a challenge for
the takeup of SUMPs in European cities.
2 What is meant by monitoring and evaluation?
Monitoring and evaluation can be applied as management tools both for individual measures and for
the overall SUMP planning process. A related process, appraisal, is used in developing proposals for
individual measures and for the overall SUMP.
A very short description about it is the following:
monitoring: continuous data collection and data analyses during implementation,
evaluation: systematic determination of measure’s merit and significance during and after
implementation with conclusions,
appraisal: evaluation of the impacts and worth of measures before implementation.
4
Monitoring and evaluation activities should be conducted on a regular cycle, although their
frequency might vary with evaluation taking place at longer time intervals. They are important tools
in the development and implementation of Sustainable Urban Mobility Plans (SUMPs) that serve the
purpose of timely identification of problems, potential successes and the need for readjustment of a
SUMP and its instruments. Providing regular information to decision makers, potential funding
bodies and local stakeholders can help to convince them that a SUMP has or will deliver benefits to
the community, provides value for money and is worth continuing, or requires modifications to be
successful.
Appraisal (exante evaluation) is conducted in the development of proposals for a measure or an
overall SUMP, to assess whether the proposals will be effective and represent value for money, or
need enhancement. Evaluation (expost evaluation) takes place after implementation of a measure
or an overall SUMP, and is used to assess whether the measure (or SUMP) has been effective, and
does represent value for money, or whether it needs modification or enhancement.
Key steps in monitoring, appraisal and evaluation are
Definition of objectives
For appraisal (exante evaluation)
o Determining a dominimum base against which to assess the proposal
o Predicting the effects of the proposal
For evaluation (expost evaluation)
o Measuring the before conditions
o Measuring the after conditions
Analysis, interpretation and, if appropriate, assessing value for money.
The “Guidelines for Developing and Implementing a Sustainable Urban Mobility Plan”
(www.mobilityplans.eu/docs/SUMPguidelines_web0.pdf) give the following description:
„Monitoring and evaluation need to be built into the plan as essential management tools to keep
track of the planning process and measure implementation, but also so that you can learn from the
planning experience, understand what works well and less well, and to build the business case and
evidence base for the wider application of similar measures in the future.”
Typical barriers towards an effective use of evaluation and monitoring are lack of financial and staff
resources as well as gaps in technical knowledge with regards to defining performance indicators,
data retrieval and collection, data preparation and data understanding.
A key element in Sustainable Urban Mobility Planning is evaluation and monitoring progress. Local
authorities often underestimate the need for monitoring and evaluation in SUMP. This challenge 4
will provide guidance on monitoring and evaluating both measures and the whole SUMP
development process for application by the cities.
The objective is to provide methodologies and feasible ways to assess change and evaluate the
effectiveness of a SUMP.
5
Figure 1: SUMP monitoring and evaluation process (CH4LLENGE, City of Dresden)
3 Why is monitoring and evaluation important for SUMPs?
Cities get knowhow and advice for local implementation for monitoring and evaluation. Monitoring and evaluation of measures and the SUMP process have benefits to efficiency of planning processes and implementation of measures,
higher quality of a SUMP itself and the SUMP process,
assess and raise the quality of measures and measure bundles and packages,
fill the gap between the objectives, the plan and its implementation,
quality management for all partners: planners, operators, politicians etc.,
save resources.
4 What research and information is available on monitoring and
evaluation?
There are broad information on monitoring and evaluation available from recent European projects:
Projects/source Topics covered Link
CIVITAS MIMOSA (2013)
Handbook “Evaluation matters a practitioners guide to sound evaluation for urban mobility measures”
http://www.civitas.eu/docs/Civitas MimosaCityBrochure.pdf
http://www.citymobil project.eu/site/en/documenten.ph p
ADVANCE (20112014)
ADVANCE develops, tests and applies an Audit Scheme to assess the quality of sustainable urban mobility planning
www.advanceproject.org
QUEST (20112013)
Quest is a Quality Management tool to help small and mediumsized cities to set up and further develop their sustainable mobility policies and actions with assistance of an external auditor. Recommendations with regard to Urban Mobility Assessment from the review of approaches to evaluation (Deliverable 3.1)
http://www.quest project.eu/index.php?id=7
CIVITAS ELAN (2012)
ELAN experiments at all stages from project planning and implementation, to monitoring and evaluation.
http://www.civitas initiative.org/content/elan
www.mobilityplans.eu/docs/SUMP guidelines_web0.pdf
http://www.cost.eu/domains_acti ons/tud/Actions/356
MAX (2009) MaxSumo. Guidance on how to plan, monitor and evaluate mobility projects
MaxSumo offers an opportunity to effectively plan, monitor and evaluate mobility projects and programmes aimed at behavioural change. Available in EN, DE, ES, FR, NL, PL, PT, SE
www.epomm.eu/index. phtml?ID1=2359&id=2359
DISTILLATE (2008)
Project C Indicators: Product C1: Designing a monitoring strategy to support effective delivery of sustainable transport goals; Product C2: Advice on selecting indicators for sustainable transport; Product C3: Monitoring across sectors and spatial levels for sustainable transport: a good practice guide
www.distillate.ac.uk, http://www.its.leeds.ac.uk/project s/distillate/outputs/products.php
Measuring indicators, p. 59 f., 61 Evaluation methods (Cost effectiveness, Costbenefit analysis, least cost planning, multiple criteria analysis), p. 79
www.osmoseos.org/ documents/316/GUIDEMAPSHand book_web[1].pdf
GUIDEMAPS (2004) Handbook, Volume 2: Fact Sheets
Measuring indicators, p. 70 f. Tools for tracking progress, p. 73 Measuring outcome indicators, p. 76 Post implementation evaluation, p. 78
www.osmoseos.org/ documents/316/GUIDEMAPSHand book_web[1].pdf
PROPOLIS (20002003)
PROPOLIS focused on developing methodologies and tools for assessment of urban sustainability and on evaluation of different land use and transport policies
http://www.transport research.info/web/projects/projec t_details.cfm?ID=4385
http://www.konsult.leeds.ac.uk/p ublic/level1/sec00/index.htm,
PROSPECTS (2003), A Methodological Guidebook
Appraisal and evaluation, p. 25 f., 33 ff. Implementation and monitoring, p. 27 ff. Basics of CBA, p. 99 f.
www.ivv.tuwien.ac.at/forschung/p rojekte/international projects/prospects2000.html
PROSPECTS (2002), Evaluation tools (Deliverable 2)
Covering a wide range of methods and tools for evaluation. See whole document.
www.ivv.tuwien.ac.at/forschung/p rojekte/international projects/prospects2000.html
These sources highlight a number of barriers to the effective adoption of monitoring and evaluation:
Many cities have some experiences in monitoring of measures. But they do not have
experiences in monitoring the SUMP process. There are very few experiences with
evaluation, either for measures or for the SUMP process. The level of experience with
monitoring and evaluation activities varies strongly between cities with only limited tradition
in preparation of SUMPs where urban transport planning is often either strongly
infrastructure based or part of landuse planning and cities which have a longestablished
tradition of transport planning (see ADVANCE project).
Differing definitions exist for the indicators to be monitored ranging from indicators for
particular impacts (e.g. environmental impacts in COST356), over those for the evaluation of
particular types of measures (e. g. for Advanced Transport Systems in MAESTRO or CityMobil
or for mobility management measures in MAXSumo) to complete indicator sets (e. g. in
DISTILLATE). However, in practice there is little consistency in what is monitored in different
cities or over time.
8
In several cities there are data available, but there is no connection to the SUMP and its
objectives. The data are used for the whole development, not for certain measures. The
cities do not really have a monitoring and evaluation strategy but they see the need of it.
Monitoring and evaluation is not very high on the political agenda.
Involving stakeholder groups in the evaluation process and using monitoring data to inform
the public are of growing importance but seen as difficult and potentially prone to pressure
from interest groups (see e. g. DISTILLATE project conclusions). Examples from Dutch
planning demonstrate how stakeholders could be involved at different steps of the
evaluation process (Macharis et al., 2013).
There are only limited experiences on how monitoring and evaluation should be managed
and who should do it (the city administration with focus on measures or an external
body/consultant with focus on SUMPprocess).
Monitoring and evaluation need capacities and also money (for behaviour surveys e. g.). But
the costs of monitoring and evaluation are much less than the costs of inefficient measures
and ineffective strategies.
5 How does CH4LLENGE address monitoring and evaluation?
The level of using monitoring and evaluation in practice in European cities differs. All CH4LLENGE
cities reported that evaluation and monitoring are seen as important by the administrations, but
political commitment is sometimes low and generally a lack of manpower and budget can be
observed.
The cities need a work method to be more efficient in data collection, monitoring and evaluation
(continuously every one, two, or more years). They need a standardised method/an algorithm. This is
reflected by the results of the CH4LLENGE survey and the workshops. The Leeds workshop
participants agreed that a list of indicators with core indicators and recommendations for the most
important indicators would be most helpful. It should be differentiated between output indicators
and intermediate indicators. Further, the cities need a guideline to practice monitoring and
evaluation (methods of measurement, time scales etc.).
Generally, it should be differentiated between large scale monitoring and evaluation in big cities and
smaller scale monitoring and evaluation in smaller cities. It should also be differentiated between
monitoring and evaluation of small measures (as a more qualitative one) and of bigger measures/
measure packages/ SUMPs (as a more quantitative one). Financing of monitoring and evaluation
tasks needs to be addressed, planned and realised. Data collection itself is usually unproblematic, but
many cities lack resources to analyse the collected data subsequently. Therefore, the cities need “pro
evaluation” arguments for the politicians, the political discussion and for the administration itself.
Also an efficient institutional cooperation is required for monitoring and evaluation.
9
The participating cities will overcome these barriers by piloting strategies and develop clear handson
guidance for decision makers and practitioners in local authorities in form of practical resource kits.
The work has started with a survey of the nine partner cities and the 30 follower cities which gives
information about the local situation, the level of practical application and of problems of monitoring
and evaluation.
Then key evaluation parameters and core indicator sets will be identified and local monitoring and
evaluation programmes will be defined.
In local pilots the advanced cities will set up monitoring and evaluation programmes for SUMP
processes and for selected measures with support of the project experts. Piloting will be applied to
selected measures (already implemented or currently being implemented outside the project’s
budget) or to the entire SUMP process (both only for optimising cities that already have a SUMP in
place).
Based on pilots in the cities CH4LLENGE will identify lessons and develop a SUMP evaluation and
monitoring kit helping other European cities to plan their own evaluation and monitoring processes
in the framework of their own SUMP.
The new experiences and knowledge gained in this CH4LLENGE workpackage monitoring and
evaluating progress will help to qualify local SUMP processes and develop further the European
Guidelines.
Figure 2: Further qualifying SUMP monitoring and evaluation (CH4LLENGE, City of Dresden)
For more information join us on www.sumpchallenges.eu
Barham, P. et al. (2012) State of the Art of Urban Mobility Assessment. QUEST Quality management toll for Urban Energy efficient Sustainable Transport. Deliverable 3.1 http://www.quest- project.eu/files/upload/files/QUEST_State_of_the_Art_of_urban_mobility_assessment.pdf (last accessed 18/12/2013)
Beukers,E. et al. (2012) Why Cost Benefit Analysis is perceived as a problematic tool for assessment of transport plans: A process perspective, Transportation Research Part A: Policy and Practice, Volume 46, Issue 1, January 2012, Pages 68-78
Bickel, P. et al. (2004) HEATCO - Developing Harmonised European Approaches for Transport Costing and Project Assessment: Deliverable 5 - Proposal for Harmonised Guidelines. http://heatco.ier.uni-stuttgart.de (last accessed 18/12/2013)
Department for Communities and Local Government, UK (2009) Multi-criteria analysis: a manual. https://www.gov.uk/government/publications/multi-criteria-analysis-manual-for-making- government-policy (last accessed 18/12/2013)
HM Treasury (2003) The Green Book – Appraisal and Evaluation in Central Government. 2003 edition, updated July 2011. https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in- central-governent (last accessed 18/12/2013)
ITS (no date) KONSULT Decision Maker’s Guidebook. http://www.konsult.leeds.ac.uk/public/level1/sec01/index.htm (last accessed 18/12/2013)
Lautso, K. et al. (2004) PROPOLIS – Planning and Research of Policies for Land Use and Transport for Increasing Urban Sustainability. Final Report, second edition, February 2004 http://www.transport-research.info/web/projects/project_details.cfm?ID=4385 (last accessed 18/12/2013)
Lichfield, N. (1996) Community Impact Evaluation. London: UCL Press
Litman, T. (2011) Sustainability and Livability: Summary of Definitions, Goals, Objectives and Performance Indicators. Monograph Victoria Transport Policy Institute. http://www.vtpi.org/sus_liv.pdf (last accessed 18/12/2013)
Litman, T. (2012) Well Measured: Developing Indicators for Sustainable and Livable Transport Planning. Monograph Victoria Transport Policy Institute. http://www.vtpi.org/wellmeas.pdf (last accessed 18/12/2013)
Mackie, P. & Worsley, T. (2013) International comparisons of transport appraisal practice: overview report. Report for the Department for Transport. https://www.gov.uk/government/publications/international-comparisons-of-transport-appraisal- practice (last accessed 18/12/2013)
Marsden, G. et al. (2005) Improved Indicators for Sustainable Transport and Planning. DISTILLATE Deliverable C1 – Sustainable Transport Indicators: Selection and Use. Leeds, York, 2005 http://www.distillate.ac.uk/outputs/reports.php (last accessed 18/12/2013)
11
Marsden, G. et al. (2006a) CityMobil - Towards advanced transport for the urban environment: Evaluation framework. Deliverable 5.1.1 http://www.citymobil-project.eu/site/en/documenten_deliverables.php (last accessed 18/12/2013)
Marsden, G. et al. (2006b) Selecting indicators for strategic performance management. in Management and Public Policy 2006. Transportation Research Board Natl. Research Council, Washington, pp. 21-29, 85th Annual Meeting of the Transportation-Research-Board, Washington, 22-26 January.
Mouter, N. et al. (2013), Attitudes towards the role of Cost–Benefit Analysis in the decision-making process for spatial-infrastructure projects: A Dutch case study, Transportation Research Part A: Policy and Practice, Volume 58, December 2013, Pages 1-14
Nelson, D., Sharkow, D. (1995) Least-Cost Planning: A Tool for Metropolitan Decision-Making, Transportation Research Record, #1499, National Academy of Sciences/Engineering, Washington, D.C., 1995
Nijkamp, P. & van Delft , A (1977) Multi-Criteria Analysis and Regional Decision-Making. Studies in Applied Regional Science. Springer Verlag
Odgaard, T. et al. (2005) Current practice in project appraisal in Europe – Analysis of country reports. HEATCO Deliverable 1 http://heatco.ier.uni-stuttgart.de/hd1final.pdf (last accessed 18/12/2013)
Tavistock Institute & AECOM (2010) Guidance for transport impact evaluations: choosing an evaluation approach to achieve better attribution. Report for the Department of Transport. http://webarchive.nationalarchives.gov.uk/20110202223908/http://www.dft.gov.uk/pgr/evalua tion/evaluationguidance/transportimpact/ (last accessed 18/12/2013)
van der Voet, M. et al. (2013) Sustainable Urban Transport Planning. The QUEST management tool and the ADVANCE Audit Scheme – a review of two practical tools to improve mobility planning. Discussion paper. http://www.quest-project.eu/files/7/d-8.1-quest-advance-discussion-paper.pdf (last accessed 18/12/2013)
Vickerman, R. (2007) Cost-benefit analysis and large-scale infrastructure projects: state of the art and challenges, Environment and Planning B: Planning and Design, 34, p. 598-610
Wefering, F. et al. (2013) Guidelines. Developing and Implementing a Sustainable Urban Mobility Plan. December 2013. http://mobilityplans.eu/docs/file/guidelines_developing-and-implementing-a- sump_final_december_2013.pdf (last accessed 18/12/2013)
German literature:
Landeshauptstadt München - Referat für Stadtplanung und Bauordnung, Mentz, Horst; Erfolgskontrollen und Monitoring der Verkehrsentwicklungsplanung München, Dresden, 2012/10/15 http://www.dresden.de/de/03/verkehr/verkehrsplanung/verkehrsentwicklungsplanung/vep/045_O effentliche_Veranstaltungen.php
12
Forschungsgesellschaft für Straßen- und Verkehrswesen e. V.; Hinweise zur Anwendung von Qualitätsmanagement in kommunalenVerkehrsplanungsprozessen, FGSV Verlag, Köln, 2007
Authors: City of Dresden Dr Kerstin Burggraf Institute for Transport Studies University of Leeds Dr Astrid Gühnemann Contributions from: Institute for Transport Studies University of Leeds Prof. Anthony D May OBE FREng January 2014
2
2 What is meant by monitoring and evaluation? ............................................................................. 3
4 What research and information is available on monitoring and evaluation? ............................... 5
5 How does CH4LLENGE address monitoring and evaluation? ........................................................ 8
6 Further reading ............................................................................................................................ 10
1 Aims of CH4LLENGE
The EU cofunded project CH4LLENGE addresses significant barriers for the wider takeup of SUMPs
in Europe. In a joint undertaking together with research and resource institutions, the project will
support European cities at different stages of advancing the takeup of SUMPs. Building on previous
experiences and lessons from earlier and ongoing national and European SUMP initiatives, the
consortium has identified common challenges which pose significant barriers in the wider takeup of
SUMPs in Europe. The project will work on innovative and transferable solutions for four SUMP
challenges.
The following sections explain in detail why monitoring and evaluation in particular is a challenge for
the takeup of SUMPs in European cities.
2 What is meant by monitoring and evaluation?
Monitoring and evaluation can be applied as management tools both for individual measures and for
the overall SUMP planning process. A related process, appraisal, is used in developing proposals for
individual measures and for the overall SUMP.
A very short description about it is the following:
monitoring: continuous data collection and data analyses during implementation,
evaluation: systematic determination of measure’s merit and significance during and after
implementation with conclusions,
appraisal: evaluation of the impacts and worth of measures before implementation.
4
Monitoring and evaluation activities should be conducted on a regular cycle, although their
frequency might vary with evaluation taking place at longer time intervals. They are important tools
in the development and implementation of Sustainable Urban Mobility Plans (SUMPs) that serve the
purpose of timely identification of problems, potential successes and the need for readjustment of a
SUMP and its instruments. Providing regular information to decision makers, potential funding
bodies and local stakeholders can help to convince them that a SUMP has or will deliver benefits to
the community, provides value for money and is worth continuing, or requires modifications to be
successful.
Appraisal (exante evaluation) is conducted in the development of proposals for a measure or an
overall SUMP, to assess whether the proposals will be effective and represent value for money, or
need enhancement. Evaluation (expost evaluation) takes place after implementation of a measure
or an overall SUMP, and is used to assess whether the measure (or SUMP) has been effective, and
does represent value for money, or whether it needs modification or enhancement.
Key steps in monitoring, appraisal and evaluation are
Definition of objectives
For appraisal (exante evaluation)
o Determining a dominimum base against which to assess the proposal
o Predicting the effects of the proposal
For evaluation (expost evaluation)
o Measuring the before conditions
o Measuring the after conditions
Analysis, interpretation and, if appropriate, assessing value for money.
The “Guidelines for Developing and Implementing a Sustainable Urban Mobility Plan”
(www.mobilityplans.eu/docs/SUMPguidelines_web0.pdf) give the following description:
„Monitoring and evaluation need to be built into the plan as essential management tools to keep
track of the planning process and measure implementation, but also so that you can learn from the
planning experience, understand what works well and less well, and to build the business case and
evidence base for the wider application of similar measures in the future.”
Typical barriers towards an effective use of evaluation and monitoring are lack of financial and staff
resources as well as gaps in technical knowledge with regards to defining performance indicators,
data retrieval and collection, data preparation and data understanding.
A key element in Sustainable Urban Mobility Planning is evaluation and monitoring progress. Local
authorities often underestimate the need for monitoring and evaluation in SUMP. This challenge 4
will provide guidance on monitoring and evaluating both measures and the whole SUMP
development process for application by the cities.
The objective is to provide methodologies and feasible ways to assess change and evaluate the
effectiveness of a SUMP.
5
Figure 1: SUMP monitoring and evaluation process (CH4LLENGE, City of Dresden)
3 Why is monitoring and evaluation important for SUMPs?
Cities get knowhow and advice for local implementation for monitoring and evaluation. Monitoring and evaluation of measures and the SUMP process have benefits to efficiency of planning processes and implementation of measures,
higher quality of a SUMP itself and the SUMP process,
assess and raise the quality of measures and measure bundles and packages,
fill the gap between the objectives, the plan and its implementation,
quality management for all partners: planners, operators, politicians etc.,
save resources.
4 What research and information is available on monitoring and
evaluation?
There are broad information on monitoring and evaluation available from recent European projects:
Projects/source Topics covered Link
CIVITAS MIMOSA (2013)
Handbook “Evaluation matters a practitioners guide to sound evaluation for urban mobility measures”
http://www.civitas.eu/docs/Civitas MimosaCityBrochure.pdf
http://www.citymobil project.eu/site/en/documenten.ph p
ADVANCE (20112014)
ADVANCE develops, tests and applies an Audit Scheme to assess the quality of sustainable urban mobility planning
www.advanceproject.org
QUEST (20112013)
Quest is a Quality Management tool to help small and mediumsized cities to set up and further develop their sustainable mobility policies and actions with assistance of an external auditor. Recommendations with regard to Urban Mobility Assessment from the review of approaches to evaluation (Deliverable 3.1)
http://www.quest project.eu/index.php?id=7
CIVITAS ELAN (2012)
ELAN experiments at all stages from project planning and implementation, to monitoring and evaluation.
http://www.civitas initiative.org/content/elan
www.mobilityplans.eu/docs/SUMP guidelines_web0.pdf
http://www.cost.eu/domains_acti ons/tud/Actions/356
MAX (2009) MaxSumo. Guidance on how to plan, monitor and evaluate mobility projects
MaxSumo offers an opportunity to effectively plan, monitor and evaluate mobility projects and programmes aimed at behavioural change. Available in EN, DE, ES, FR, NL, PL, PT, SE
www.epomm.eu/index. phtml?ID1=2359&id=2359
DISTILLATE (2008)
Project C Indicators: Product C1: Designing a monitoring strategy to support effective delivery of sustainable transport goals; Product C2: Advice on selecting indicators for sustainable transport; Product C3: Monitoring across sectors and spatial levels for sustainable transport: a good practice guide
www.distillate.ac.uk, http://www.its.leeds.ac.uk/project s/distillate/outputs/products.php
Measuring indicators, p. 59 f., 61 Evaluation methods (Cost effectiveness, Costbenefit analysis, least cost planning, multiple criteria analysis), p. 79
www.osmoseos.org/ documents/316/GUIDEMAPSHand book_web[1].pdf
GUIDEMAPS (2004) Handbook, Volume 2: Fact Sheets
Measuring indicators, p. 70 f. Tools for tracking progress, p. 73 Measuring outcome indicators, p. 76 Post implementation evaluation, p. 78
www.osmoseos.org/ documents/316/GUIDEMAPSHand book_web[1].pdf
PROPOLIS (20002003)
PROPOLIS focused on developing methodologies and tools for assessment of urban sustainability and on evaluation of different land use and transport policies
http://www.transport research.info/web/projects/projec t_details.cfm?ID=4385
http://www.konsult.leeds.ac.uk/p ublic/level1/sec00/index.htm,
PROSPECTS (2003), A Methodological Guidebook
Appraisal and evaluation, p. 25 f., 33 ff. Implementation and monitoring, p. 27 ff. Basics of CBA, p. 99 f.
www.ivv.tuwien.ac.at/forschung/p rojekte/international projects/prospects2000.html
PROSPECTS (2002), Evaluation tools (Deliverable 2)
Covering a wide range of methods and tools for evaluation. See whole document.
www.ivv.tuwien.ac.at/forschung/p rojekte/international projects/prospects2000.html
These sources highlight a number of barriers to the effective adoption of monitoring and evaluation:
Many cities have some experiences in monitoring of measures. But they do not have
experiences in monitoring the SUMP process. There are very few experiences with
evaluation, either for measures or for the SUMP process. The level of experience with
monitoring and evaluation activities varies strongly between cities with only limited tradition
in preparation of SUMPs where urban transport planning is often either strongly
infrastructure based or part of landuse planning and cities which have a longestablished
tradition of transport planning (see ADVANCE project).
Differing definitions exist for the indicators to be monitored ranging from indicators for
particular impacts (e.g. environmental impacts in COST356), over those for the evaluation of
particular types of measures (e. g. for Advanced Transport Systems in MAESTRO or CityMobil
or for mobility management measures in MAXSumo) to complete indicator sets (e. g. in
DISTILLATE). However, in practice there is little consistency in what is monitored in different
cities or over time.
8
In several cities there are data available, but there is no connection to the SUMP and its
objectives. The data are used for the whole development, not for certain measures. The
cities do not really have a monitoring and evaluation strategy but they see the need of it.
Monitoring and evaluation is not very high on the political agenda.
Involving stakeholder groups in the evaluation process and using monitoring data to inform
the public are of growing importance but seen as difficult and potentially prone to pressure
from interest groups (see e. g. DISTILLATE project conclusions). Examples from Dutch
planning demonstrate how stakeholders could be involved at different steps of the
evaluation process (Macharis et al., 2013).
There are only limited experiences on how monitoring and evaluation should be managed
and who should do it (the city administration with focus on measures or an external
body/consultant with focus on SUMPprocess).
Monitoring and evaluation need capacities and also money (for behaviour surveys e. g.). But
the costs of monitoring and evaluation are much less than the costs of inefficient measures
and ineffective strategies.
5 How does CH4LLENGE address monitoring and evaluation?
The level of using monitoring and evaluation in practice in European cities differs. All CH4LLENGE
cities reported that evaluation and monitoring are seen as important by the administrations, but
political commitment is sometimes low and generally a lack of manpower and budget can be
observed.
The cities need a work method to be more efficient in data collection, monitoring and evaluation
(continuously every one, two, or more years). They need a standardised method/an algorithm. This is
reflected by the results of the CH4LLENGE survey and the workshops. The Leeds workshop
participants agreed that a list of indicators with core indicators and recommendations for the most
important indicators would be most helpful. It should be differentiated between output indicators
and intermediate indicators. Further, the cities need a guideline to practice monitoring and
evaluation (methods of measurement, time scales etc.).
Generally, it should be differentiated between large scale monitoring and evaluation in big cities and
smaller scale monitoring and evaluation in smaller cities. It should also be differentiated between
monitoring and evaluation of small measures (as a more qualitative one) and of bigger measures/
measure packages/ SUMPs (as a more quantitative one). Financing of monitoring and evaluation
tasks needs to be addressed, planned and realised. Data collection itself is usually unproblematic, but
many cities lack resources to analyse the collected data subsequently. Therefore, the cities need “pro
evaluation” arguments for the politicians, the political discussion and for the administration itself.
Also an efficient institutional cooperation is required for monitoring and evaluation.
9
The participating cities will overcome these barriers by piloting strategies and develop clear handson
guidance for decision makers and practitioners in local authorities in form of practical resource kits.
The work has started with a survey of the nine partner cities and the 30 follower cities which gives
information about the local situation, the level of practical application and of problems of monitoring
and evaluation.
Then key evaluation parameters and core indicator sets will be identified and local monitoring and
evaluation programmes will be defined.
In local pilots the advanced cities will set up monitoring and evaluation programmes for SUMP
processes and for selected measures with support of the project experts. Piloting will be applied to
selected measures (already implemented or currently being implemented outside the project’s
budget) or to the entire SUMP process (both only for optimising cities that already have a SUMP in
place).
Based on pilots in the cities CH4LLENGE will identify lessons and develop a SUMP evaluation and
monitoring kit helping other European cities to plan their own evaluation and monitoring processes
in the framework of their own SUMP.
The new experiences and knowledge gained in this CH4LLENGE workpackage monitoring and
evaluating progress will help to qualify local SUMP processes and develop further the European
Guidelines.
Figure 2: Further qualifying SUMP monitoring and evaluation (CH4LLENGE, City of Dresden)
For more information join us on www.sumpchallenges.eu
Barham, P. et al. (2012) State of the Art of Urban Mobility Assessment. QUEST Quality management toll for Urban Energy efficient Sustainable Transport. Deliverable 3.1 http://www.quest- project.eu/files/upload/files/QUEST_State_of_the_Art_of_urban_mobility_assessment.pdf (last accessed 18/12/2013)
Beukers,E. et al. (2012) Why Cost Benefit Analysis is perceived as a problematic tool for assessment of transport plans: A process perspective, Transportation Research Part A: Policy and Practice, Volume 46, Issue 1, January 2012, Pages 68-78
Bickel, P. et al. (2004) HEATCO - Developing Harmonised European Approaches for Transport Costing and Project Assessment: Deliverable 5 - Proposal for Harmonised Guidelines. http://heatco.ier.uni-stuttgart.de (last accessed 18/12/2013)
Department for Communities and Local Government, UK (2009) Multi-criteria analysis: a manual. https://www.gov.uk/government/publications/multi-criteria-analysis-manual-for-making- government-policy (last accessed 18/12/2013)
HM Treasury (2003) The Green Book – Appraisal and Evaluation in Central Government. 2003 edition, updated July 2011. https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in- central-governent (last accessed 18/12/2013)
ITS (no date) KONSULT Decision Maker’s Guidebook. http://www.konsult.leeds.ac.uk/public/level1/sec01/index.htm (last accessed 18/12/2013)
Lautso, K. et al. (2004) PROPOLIS – Planning and Research of Policies for Land Use and Transport for Increasing Urban Sustainability. Final Report, second edition, February 2004 http://www.transport-research.info/web/projects/project_details.cfm?ID=4385 (last accessed 18/12/2013)
Lichfield, N. (1996) Community Impact Evaluation. London: UCL Press
Litman, T. (2011) Sustainability and Livability: Summary of Definitions, Goals, Objectives and Performance Indicators. Monograph Victoria Transport Policy Institute. http://www.vtpi.org/sus_liv.pdf (last accessed 18/12/2013)
Litman, T. (2012) Well Measured: Developing Indicators for Sustainable and Livable Transport Planning. Monograph Victoria Transport Policy Institute. http://www.vtpi.org/wellmeas.pdf (last accessed 18/12/2013)
Mackie, P. & Worsley, T. (2013) International comparisons of transport appraisal practice: overview report. Report for the Department for Transport. https://www.gov.uk/government/publications/international-comparisons-of-transport-appraisal- practice (last accessed 18/12/2013)
Marsden, G. et al. (2005) Improved Indicators for Sustainable Transport and Planning. DISTILLATE Deliverable C1 – Sustainable Transport Indicators: Selection and Use. Leeds, York, 2005 http://www.distillate.ac.uk/outputs/reports.php (last accessed 18/12/2013)
11
Marsden, G. et al. (2006a) CityMobil - Towards advanced transport for the urban environment: Evaluation framework. Deliverable 5.1.1 http://www.citymobil-project.eu/site/en/documenten_deliverables.php (last accessed 18/12/2013)
Marsden, G. et al. (2006b) Selecting indicators for strategic performance management. in Management and Public Policy 2006. Transportation Research Board Natl. Research Council, Washington, pp. 21-29, 85th Annual Meeting of the Transportation-Research-Board, Washington, 22-26 January.
Mouter, N. et al. (2013), Attitudes towards the role of Cost–Benefit Analysis in the decision-making process for spatial-infrastructure projects: A Dutch case study, Transportation Research Part A: Policy and Practice, Volume 58, December 2013, Pages 1-14
Nelson, D., Sharkow, D. (1995) Least-Cost Planning: A Tool for Metropolitan Decision-Making, Transportation Research Record, #1499, National Academy of Sciences/Engineering, Washington, D.C., 1995
Nijkamp, P. & van Delft , A (1977) Multi-Criteria Analysis and Regional Decision-Making. Studies in Applied Regional Science. Springer Verlag
Odgaard, T. et al. (2005) Current practice in project appraisal in Europe – Analysis of country reports. HEATCO Deliverable 1 http://heatco.ier.uni-stuttgart.de/hd1final.pdf (last accessed 18/12/2013)
Tavistock Institute & AECOM (2010) Guidance for transport impact evaluations: choosing an evaluation approach to achieve better attribution. Report for the Department of Transport. http://webarchive.nationalarchives.gov.uk/20110202223908/http://www.dft.gov.uk/pgr/evalua tion/evaluationguidance/transportimpact/ (last accessed 18/12/2013)
van der Voet, M. et al. (2013) Sustainable Urban Transport Planning. The QUEST management tool and the ADVANCE Audit Scheme – a review of two practical tools to improve mobility planning. Discussion paper. http://www.quest-project.eu/files/7/d-8.1-quest-advance-discussion-paper.pdf (last accessed 18/12/2013)
Vickerman, R. (2007) Cost-benefit analysis and large-scale infrastructure projects: state of the art and challenges, Environment and Planning B: Planning and Design, 34, p. 598-610
Wefering, F. et al. (2013) Guidelines. Developing and Implementing a Sustainable Urban Mobility Plan. December 2013. http://mobilityplans.eu/docs/file/guidelines_developing-and-implementing-a- sump_final_december_2013.pdf (last accessed 18/12/2013)
German literature:
Landeshauptstadt München - Referat für Stadtplanung und Bauordnung, Mentz, Horst; Erfolgskontrollen und Monitoring der Verkehrsentwicklungsplanung München, Dresden, 2012/10/15 http://www.dresden.de/de/03/verkehr/verkehrsplanung/verkehrsentwicklungsplanung/vep/045_O effentliche_Veranstaltungen.php
12
Forschungsgesellschaft für Straßen- und Verkehrswesen e. V.; Hinweise zur Anwendung von Qualitätsmanagement in kommunalenVerkehrsplanungsprozessen, FGSV Verlag, Köln, 2007