defra evaluation of the catchment-based approach pilot stage · first quarterly review draft final...

Defra

Evaluation of the catchment-based approach – pilot stage

Quarterly Review Report May 2012

In Association with

YJRees

Consulting

Client: Defra Title: Evaluation of the catchment-based approach – pilot stage

(Quarterly Review) Project No: CC472 Date of Issue: May 2012 Status: Final Version No: 1 Produced By Authorised for Release By

……………………………….. …………………………………… David Corbelli Dr Kieran Conlan Principal Environmental Scientist Managing Director CONTACT DETAILS CASCADE CONSULTING

Enterprise House Manchester Science Park Lloyd St North Manchester M15 6SE Tel: 0161 227 9777 Fax: 0161 227 1777

Defra Evaluation of the Catchment-based Approach – Pilot Stage Quarterly Review Draft Final

Cascade Consulting

CONTENTS

1 Introduction..................................................................................... 1 2 Roles, Responsibilities and Skills ..................................................... 2 3 Timescales and Delivery ................................................................... 6 4 Engaging, sharing and support ...................................................... 14 5 Learning ........................................................................................ 23 6 Benefits .......................................................................................... 26 7 Costs and Value for Money ............................................................. 31 8 Feedback ........................................................................................ 33 Appendix A: Context to the Quarterly Review ........................................ 35 Appendix B: Timescales and Delivery .................................................... 38 Appendix C: Engaging, Sharing and support .......................................... 41

Defra Evaluation of the Catchment-based Approach – Pilot Stage First Quarterly Review Draft Final

Cascade Consulting 1

1 INTRODUCTION

1.1 THE QUARTERLY REVIEW

Defra are currently supporting a series of catchment-level partnerships to pilot a new

approach of improving the water environment through catchment-level engagement

and planning. As part of this pilot phase Defra is evaluating the 25 catchments to

learn all we can from those participating in this pilot phase and to share that

information with others who may be involved in a nationwide roll-out from 2013. As

part of the evaluation pilot hosts are participating in a quarterly review process with

the aim to:

‘review activities, to highlight potential challenges coming, up and

identify practices that it would be worth sharing, to help design the

learning events, and to look at changes for example, in stakeholder

groups/participants over time.’

This report is not the only output from the quarterly reporting, but it is intended to

summarise the high level messages that have been fed-back by the pilot hosts. Other

outputs include an action list to address support needs identified, and case-studies.

Further information on the wider context of the review is given in Appendix A.

1.2 OVERVIEW OF RESPONSES

The first quarterly review/survey was launched on 23/03/2012 following

consultation on content and design, and closed on 24/04/2012. The survey was set

up using an online survey tool and was sent to all pilot hosts by email. Responses

were received from 22 catchment pilots, of which three were partial and 19 complete.

All jointly hosted catchment pilots managed to submit their responses in a single

form for that pilot. In cases where the same individual from the pilot host had

submitted several copies of the review, these responses were combined.

This report is based on the responses from pilot hosts in the Quarterly Review up to

March 2012. We have not checked or queried the information received since the focus

is on learning and feeding back messages to pilots quickly. The report is an interim

evaluation, a snapshot of progress made and immediate lessons learned that will be

further developed next quarter and reviewed at the end the pilot programme.

1.3 REPORT STRUCTURE

The main body of this report is divided into sections, as per the Quarterly Review,

covering: Roles and Responsibilities, Timescales and Delivery, Engaging, Learning,

Benefits and Costs.



2 ROLES, RESPONSIBILITIES AND SKILLS

In this section we asked about the key roles within the pilot, what skills and training

were needed and which stakeholders were involved. It draws on Questions 4-16.

Most (20 out of 22) pilots completed all of this section.

2.1 ROLES

Typically the key roles in the catchment have been defined as the following:

Catchment co-ordinator, Project director/lead OR Pilot host, Project officer, Task

group lead or specialist (e.g. farming advisor), Facilitator/Working with Others

mentor, Project officer, Steering Group member, Chair of steering group.

The pattern of types of roles did not vary greatly across the pilots as most pilots

expressed similar types of roles. In terms of the number of roles identified 21 of the

pilots had 2 or more key roles identified, with 6 having 5 key roles identified. The

roles and responsibilities have not changed over this quarter for 17 of the 22 pilots

but have changed for 5 of the 22. Reasons for change include project officers being

appointed and also new groups being formed.

2.2 INVOLVING OTHERS

Interestingly, in terms of the numbers of different organisations involved in filling

those key roles, 14 of the pilots had only 1 or 2 organisations filling all the roles they

had identified, with the remaining 8 having 3 or 4 organisations filling all the roles.

This could give the impression of some pilots working in isolation but looking at Q13

Have you set up at catchment group or groups? this does not appear to be the case:

with the majority of catchments (19 out of 22) having some form of catchment group

all of which have between 4 – 70 groups represented. This is backed up by findings

from Q45 To what extent have you been able to set up a catchment group? Further,

response to Q14 How many stakeholders were actively involved in the pilots this

quarter? showed that 16 of the 22 pilots had 11 or more stakeholders actively

involved in this quarter suggesting a high level of engagement with stakeholders. In

terms of who those stakeholders are, findings from Q11 which asked respondents to

list those stakeholders actively involved in terms of whether there were Public,

Private, Community, Voluntary, show that there is good range across those four

sectors with a focus, not surprisingly at this stage, on environmental and water

related organisations. Clearly who is involved in the pilot will evolve over time and to

some extent that understanding is reflected in the answers to Q15 How confident at

this stage are you that the pilot is representative of the range of stakeholders within

the catchment?. 15 of the 22 responses said they were “Quite Confident” whilst 5 said

“Not Confident or Did not know” and only one said they were “Very Confident”.



Only 6 of the 22 pilots have written working agreements between the stakeholders in

the catchment group. This taken with the numbers of different organisations

involved in key roles (see above, Q4) and the amount of active involvement of others

suggests a model where one or two organisations carry out the co-ordination of the

pilot which then links into a wider group of stakeholders who may work on specific

issues or provide support for specific actions.

2.3 SKILLS

Pilots identified the key skills needed to be a pilot host as numerous as can be seen by

the “word cloud” in Figure 2.1 (the larger the word the more times it was repeated).



Figure 2.1: Q6. What are the key skills you have needed as pilot hosts?



Three key areas can be identified:

Individual characteristics: e.g. patience, imagination and influence

Skills relating to collaborative working: e.g. facilitation, ability to

communicate and a willingness to work collaboratively.

Skills relating to technical aspects: e.g. water quality, understanding of the EU

Water Framework Directive (WFD) , data analysis

Pilots identified a wide range of skills gaps. Table 2.1 summarises the top 6 needs

identified with examples of how they have been overcome.

Table 2.1: Q7. What skills gaps have you identified and how have these been

overcome?

Main gaps How these were overcome Facilitation expertise Support by Dialogue by Design team or bringing in facilitators

as required.

Project management Recruitment and training (InterAct).

Understanding the WFD Research and working with experts particularly those in the EA

GIS and mapping Using external teams or building capacity depending on

resources

Understanding wider issues (within the catchment)

Working with stakeholders

Communication Using communication teams within the Environment Agency

(EA) or provided experts (Dialogue By Design)



3 TIMESCALES AND DELIVERY

This section is designed to help answer the research questions:

How well were the pilots able to meet the principles/ objectives and

milestones of the catchment-based approach?

Where do pilots need support or can give support in the coming months?

In order to do this, pilot hosts were asked to assess progress and to highlight

challenges, support needs, tips and good practices that it would be useful to share

with others, against 10 key activities of a catchment-based approach (see Figure 3.1).

Figure 3.1: Q17, 26, 35, 45, 54, 64, 73, 84 and 93. To what extent have you been able

to carry out the following components this quarter? (22 responses)

*The options for the question on catchment walkovers were phrased differently, but the categories align with those

used in the other questions presented here.

3.1 OVERALL POSITION

The table below summarises the progress pilots had made, against 10 common

activities within a catchment based approach, by March 2012.

0% 50% 100%

Establishing outline costs and relative effectiveness of actions

Agreeing actions and gaining commitment

Agreeing catchment problems and priorities

Developing a shared vision

Agreeing specific objectives for the pilot

Setting up a catchment group

Developing a communication plan*

Planning stakeholder involvement

Stakeholder mapping

Baseline activity mapping

In full Mostly In part Not at all Don't know



Table 3.1: Q17, 26, 35, 45, 54, 64, 73, 84 and 93. To what extent have you been able to carry out the following components this quarter? (22 responses)

Number of pilots reporting this level

of progress†

Component Average

position*

Not at

all (1)

In

part

(2)

Mostly

(3)

In full

(4)

Don‟t

know

(0)

Baseline Activity Mapping 2.5 1 11 7 2 1

Stakeholder Mapping 2.9 0 5 14 3 0

Planning stakeholder involvement 2.5 1 13 4 4 0

Setting up a catchment group 3.1 2 3 6 9 0

Agreeing specific objectives for the pilot 2.3 3 9 8 1 0

Developing a shared vision 2.1 7 8 3 3 0

Agreeing catchment problems and priorities 2.3 3 11 5 2 0

Organising catchment walkovers 3.3 1 4 3 12 0

Agreeing actions and gaining commitment 1.8 7 11 1 1 1

Establishing outline costs and effectiveness 1.3 15 5 1 0 0

*The average position is calculated where a score of 1 is given for „not at all‟, a score of 2 for „In part‟, a score of 3 for ‘Mostly‟ and a score of 4 for ‘In full‟. Higher scores indicate more progression.

†The highlighted squares indicate the „majority position‟ at the end of Q1.

Note: Running Times for Pilots (from the baseline)

Although most of the EA hosted pilots began last summer, many of the non-EA

hosted pilots have been running less than 3 months (exceptions being the Bristol

Avon at 4 months, the Frome at 5 months and the Wey at 11 months). Overall, the

average time pilots have been running is 4.7 months (EA pilots average 7.7 months

and non-EA pilots 2.7 months) with the longest running pilot being the Irwell at 12

months.

Although it is still early days for many of the pilots, good progress is being made:

All pilots have progressed stakeholder mapping, at least in part.

All but one pilot has progressed with the baseline mapping.

All but one pilot is planning their stakeholder involvement.

All but 2 pilots have progressed setting up a catchment group.

All but 3 pilots have progressed agreeing specific objectives.

All but 3 pilots have progressed agreeing catchment problems and priorities.

As would be anticipated at this stage fewer pilots have progressed activities such as:



Developing a shared vision.

Agreeing actions and gaining commitment.

Establishing outline costs and effectiveness.

Table B.1 in Appendix B summarises pilot hosts‟ assessments of progress against key

activities in their own pilot, on a pilot by pilot basis. For pilots just starting activities,

or finding activities particularly challenging, this table can be used to identify those

who have completed activities and who may be able to share advice based on their

experiences.

3.2 KEY MESSAGES

The text below highlights some key messages from activities undertaken so far. A

very positive message is that there are pilots who are well advanced in each of the

activities identified and pilot hosts have provided a wide range of useful tips and

case-studies to help others who are about to tackle these activities in the coming

months. This information is given in a standard format in Appendix B. Note

currently this has been prepared for one activity as an example with an intention to

complete similar tables for all activities based on pilot host feedback.

Overall, pilot hosts are reporting that the activities of a catchment based approach

are very labour intensive. Recognising that they can't produce what is needed by

themselves, one of the key challenges hosts are reporting is that it is very difficult to

find sufficient time in peoples diaries for regular stakeholder meetings, particularly

as many of those involved are doing so on a voluntary basis making it impossible to

place too great demands on time. Linked to this are challenges of gaining resources

and funding to enable partners to be involved. Being part of the stakeholder group is

an additional work area for delegates, partners are either voluntary or for those in

post too busy. One pilot reported this is particularly noticeable for the Local

Authorities (LAs) which are under threat from funding cuts.

For many, the awarding of the contracts and the start date did not leave a lot of time

to recruit/obtain the necessary resources. Therefore the first few months‟ activities

have been a little condensed. Generally activities are running in parallel but pilots are

putting varying emphasis on the activities illustrating the wide range of ways

catchment based planning can be approached. For example, some pilots have focused

on stakeholder mapping, planning stakeholder engagement and setting a catchment

group, whilst others have been mapping baseline activities and clarifying objectives

before going out to seek stakeholder views.

There was considerable feedback on Baseline activity mapping (Q17-25) many

hosts indicating that it is „a time consuming exercise‟ that „will develop over the

projects lifetime‟. Baseline activity mapping is yielding useful information with hosts



reporting that there is „much good work already happening in the catchment to

forward WFD aims but isn't recognised as such, by us or the project authors… but has

much passionate support‟.

Many are summarising issues and ongoing work textually but have not yet

graphically mapped the information (in some cases as the data required is still

awaited from the EA and Defra). Experience of mapping reported in the Leam,

Cotswold, Welland and Lower Wear may provide useful insights for others.

All pilots have started to undertake their stakeholder mapping (Q26-34) and in

most cases this is an ongoing exercise with pilots expecting lists of potential partners

and networks to grow. Commonly stakeholders are being included as steering

groups, task groups and a wider network. This has been an easier task where partners

have been working together previously and hosts have been able to use existing

networks to start their analysis, but most are undertaking more detailed analysis

often with support from their facilitators.

Many report on the challenges of the complexity, especially in larger catchments and

are having to undertake targeted stakeholder mapping given the tight timescales.

Hosts found it essential to have local knowledge and to use this to identify any

additional stakeholders to involve but have found it difficult to gain commitment

from some key players and on the other hand to select groupings without offending

anyone. The Cotswold pilot notes that the Integrated Local Delivery (ILD) process

enables more partners to become linked in as an all inclusive partnership with a

communication structure such that organisations can become more active when a

particular issue is relevant to them, and less when it is not.

Planning Stakeholder Involvement (Q35-44) is generally not as developed as

stakeholder mapping, though most have partially progressed this. Most pilots have

started to develop a communications plan though none have them have yet

completed this. Typically pilots are reporting that their approaches are not

prescriptive but „have evolved in response to feedback from stakeholders‟, that „a plan

for involvement has developed organically‟, but is key to maintaining stakeholders

interest. Many pilot hosts are working with others to develop communications plans

making use of the specialist communication skills of mentors and facilitators, and in

some cases, working with other catchment pilots.

For setting up a catchment group (Q45-53) the model of a „catchment group‟ is

not considered appropriate for all pilots with some instead relying on existing groups.

In the Tidal Thames, the hosts are planning to ask key stakeholders whether they

would like a physical group that meets regularly, a 'virtual' group that liaises through

email/forum, or an advisory group that oversees and prioritises issues. Similarly, in

the Nene, there is no formal catchment group as such, instead, the pilot is utilising



existing pertinent project groups and partnership links, recognising that „generally it

is the same organisations (and sometimes individuals from these) that would be

involved and the stakeholders are keen not to duplicate effort, and spread staff time

too thinly, particularly if existing groups can bring the pilot into their scope‟. In the

Tyne, the hosts have signed up to the River Basin Planning (RBP) Liaison Panel

comprising 4 public and 3 private bodies, and will involve existing River Watch

groups. The Tame is another pilot using existing groups at the moment, but the aim

here is to work towards a catchment conference in September where it is anticipated

a number of sub-groups will be established to cover different parts of the catchment

as a single group for the whole area is not thought sufficient to the range of

circumstances and contrasts.

Again the process of setting up a catchment group is already bringing benefits with

pilot hosts commenting, “we were surprised at our initial meetings that some

stakeholders whose areas of interests overlapped weren't already talking to each

other. Our workshops have helped to address this problem. We often see people

exchanging business cards and email addresses over coffee breaks etc. at our events.

This shouldn't be overlooked as a benefit of the pilots though it is hard to quantify.”

For agreeing specific objectives for the pilot (Q54-62), most pilots reported

this was a key activity for them this quarter, but only one pilot noted this had been

completed in full. Agreeing objectives can be seen as an ongoing activity with 3 pilots

reporting changes to their objectives this quarter to increase specificity or introduce

new objectives to take on board comments from the catchment group or stakeholder

meetings.

The timing and degree of involvement of stakeholders in agreeing objectives has

varied. In some cases, Steering Group partners have drafted objectives for the pilot

for wider discussion, whilst other pilots have allocated this task to sub groups, and

others are drafting 'outcomes' organised into 'themes' using wider stakeholder group

workshops.

In terms of developing a shared vision (Q63-72), some were unclear what was

meant by a vision with one comment that stakeholders were put off by this and

wanted to get on with delivery. Others noted the importance of making stakeholders

aware of the „need to agree and be clear what we are aiming for and look long-term

strategic for everyone to buy-in‟. Pilot hosts noted that it is easy to agree on relatively

bland non-controversial statements for a shared vision created by the catchment

steering group, or a sub-group for example, but recognise that the vision will only be

fully developed when other key stakeholders have had an input and that this will

mean the vision will need to be reconsidered as issues are explored in more detail

which may give rise to disputes.



About half the pilots reported that agreeing catchment problems and

priorities (Q73-81) was a key activity for them this quarter but it is seen largely as

an ongoing activity with only 2 pilots reporting that it has fully progressed (the

Cotswold and the Frome and Piddle). The starting point for many has been

information from the EA on WFD failures in the catchment with pilots planning to

integrate this with stakeholders objectives to prioritise and problem solve in the

catchment. Interestingly, one pilot flagged „starting with EA evidence alone‟ as being

unhelpful and it may be wise to consider how best to present the current situation

when building in contributions from others. Pilots more progressed in this activity

noted that „people had very different impressions of the state of the river ...as it

currently is‟.

Most pilots have, or are planning to hold catchment walkovers (Q82-83). It is

clear from the comments received that catchment walkovers require considerable

resources and that pilots are planning to use them strategically, carrying out

walkovers in a targeted manner and in association with other techniques. With this in

mind, pilots consider that Q1 is generally too early to undertake catchment

walkovers, and are planning them for Q2. Key questions being considered are in

deciding on the use of catchment walkovers are: What other walkovers have taken

place already that we can make use of? What do we want to achieve from the

walkover (to identify problems, to get a feel for the catchment as a whole and locate

projects/ quick wins, to provide more detail on specific issues, to engage

stakeholders, or launch stakeholder initiatives such as monitoring or wardening)?

Many thought the EA should lead the walkover given their expertise in this field but

others raised the value of inputs from other groups such as anglers. Pilot hosts also

highlighted that walkovers are not suitable for all problems (e.g. not useful

identifying nitrate pollution in groundwater fed catchments or identifying sources of

diffuse pollution problems in the catchment with no identifiable point source with

solutions).

Agreeing actions and gaining commitment (Q84-93) is generally an activity

many pilots have not yet progressed, with only the Welland pilot reporting that it has

fully been able to agree actions and gain commitment as all Steering Group members

are fully supportive of the 'draft plan'. Many pilots indicate that some actions are

already agreed as most partners come to the process with ideas, but that they will

need to gain a better understanding as to how actions will be implemented under the

pilot, where the funding will come from and who will deliver them before they can be

initiated in earnest. This process takes time, and one pilot commented that whilst

partners seem very engaged and committed, the timescale of one year is not long

enough.

Most pilots have not started establishing outline costs and relative



effectiveness of actions (Q94-101), with only 2 pilots reporting this as a key

activity for them this quarter (the Wey and the Cotswold) and only 1 pilot (the

Welland) reporting that it has „mostly‟ established costs and effectiveness of actions.

For evaluation (Q102-Q105), 11 pilots (6 EA hosted pilots and 5 others) reported

that they are undertaking their own evaluation. They found it difficult to quantify the

time allocated to this but estimates, to cover the period of the pilots, ranged from 3 to

50 days and were typically around 10 days. One pilot stated time is „not quantified

but we view this is an important part of our engagement work‟. Others that they

would keep the evaluation time „minimal‟. In most cases the evaluation is being lead

internally by the co-ordinator, or the EA, though 2 pilots are being evaluated by

external organisations. Hosts report a range of benefits of the evaluation, but raise

concerns that evaluation should not take away too much time from the „doing‟.

3.3 SUPPORT NEEDS

There was considerable recognition of support given, particularly from pilot partners,

the EA and the independent facilitation. Pilots also highlighted the funding provided,

the data and GIS support from the EA (though in some cases it had not yet been

provided) and examples from other pilots as helpful.

Key areas where additional support was identified were:

Clearer guidance, for example, on how a vision differs from objectives, on the

specificity of objectives, on the purpose/objectives of the pilots (was it just about

implementing WFD or something wider?), to provide a communication plan

template, a timetable of events, more examples, for example of visions, clarifying

what is happening post 2012, and encouraging other organisations to engage in

ILD.

Information on list of EU Directives, Rural Development Programme for

England (RDPE) initiatives, Environmental Stewardship uptake, earlier on-line

collaboration tools, techniques for visioning, translating WFD evidence in a

meaningful way, funding deadlines for 2012/13, and EA/Defra funded projects.

Contact lists – e.g. a who's who in local authorities, a database of landowners in

the catchment, organograms of each agency.

More resources and funding so all partners could be involved, to hire venues etc,

for additional facilitation, for a secretariat.

More support: from the Forestry Commission; on legal issues; on how to

estimate costs; from WWO mentors.



Earlier training on collaborative working.

Note: Detailed support needs are being developed into an action list outlining how

best these needs could be addressed and by whom within the Defra family.

3.4 MILESTONES

Initially pilots were directed to progress their work according to a set timetable of

common milestones (set out in Defra‟s EoIs and for EA pilots in their Principles and

Evaluation Pack, see Appendix B). Defra and the EA have clarified the position on

this as follows:

Q: Will a pilot be judged to be failing if the milestones are not met and a final plan is

not produced?

A: By the end of the pilot phase we need to be able to see what actions have been

identified, what commitment to act has been gained, and when the improvements as

a result of those actions might be seen. This does not however mean that we expect

to see a detailed glossy plan that seeks to address all the problems in a catchment.

As set out in the initial guidance to hosts, we believe that it's key for pilots to work

collaboratively and seek collective agreement. The milestones were not set as a

measure of success or failure but rather as a guide to what might be realistically

achieved in the time available. The pace of progress is expected to vary significantly

between pilots according to the complexity and number of problems that they seek

to address, and by the extent and value of any collaborative working that was

already taking place prior to the pilot.

The feedback received from the pilot hosts this quarter shows that with one

exception, none of the pilots were able to fully progress the activities included under

the initial Milestones proposed for the first quarter (i.e. to undertake stakeholder and

baseline activity mapping, to set up stakeholder groups and gain feedback). The

feedback confirms earlier thoughts that the initial timescales proposed were not

feasible particularly for pilots who have had to appoint catchment co-ordinators,

identify stakeholders and arrange stakeholder meetings before being able to tackle

technical issues.



4 ENGAGING, SHARING AND SUPPORT

4.1 INTRODUCTION

In this section three key areas are presented and discussed:

Engagement activities: What engagement activities have pilots carried out

over the last quarter? This includes any engagement with the public, what

methods for engagement they are using and finding helpful (Q108 – 114; Q128-

129), together with the use of visualisation tools to engage (Q124-127)

Information and data sharing: How is information and data being shared?

This includes understanding the successes and challenges to sharing information

and data (Q115-123)

Support from Defra and its network: What is the role of external support?

This includes understanding the nature and usefulness of that external support

(Q130-132)

4.2 RESULTS

4.2.1 Engagement activities: types

The questions in this section were aimed at understanding the range and nature of

activities that has been undertaken by the pilots over the past quarter and Question

108 focussed on the meetings, events and activities that had taken place over the past

quarter. The table below provides an overview of the types of people who were

involved in these events.

Table 4.1: Types of people engaged in meetings, activities and events over the last

quarter (21 responses – multiple activities could be listed)

Who was engaged Number of activities

(approx)

Steering group members, project team members, specialists 65

Formal stakeholders (Local Authorities, regulators) 15

Community groups 2

Sector stakeholders (Water Companies, Wildlife Trusts; Catchment

Sensitive Farming groups)

30

All the above and wider stakeholders 3

With dialogue specialists 4

It is clear from this that the pilots are engaging with a wide range of groups in their

catchments, with the greatest amount of engagement activity is with the steering

groups and the working groups. Least activity is directed at wider stakeholders,



including members of the public.

Table 4.2: Types of meetings, activities, and events carried out this quarter (from

Q108)

Type of activity Carried out this quarter

Meetings,

workshops, events

Steering group meetings, Task group meetings, Sector group meetings

(e.g. habitats), Workshops, Launch events, One to one discussions,

Focus groups, Internal planning meetings, Stakeholder engagement

planning meetings, Presentations at meetings led by others, Public

engagement event

Visits Catchment visits, Catchment walkovers, Circuit riding,

Catchment actions Cattle fencing

Planning actions Community engagement planning, Draft plan development, Report

writing, Identifying and mapping where projects/work could be done,

Submitting proposals for funding, Developing pilot logo, Developing

websites, Writing e-newsletter, Stakeholder mapping, Data collection

Other Stall at park opening,

The table above gives an indication of the range of activities that are being carried out

from one to one meetings through to facilitated workshops, stalls at shows and cattle

fencing. In addition, it suggests that there are different ways in which pilots are

doing the catchment approach:

1. starting by forming a steering group, e.g. Welland Valley Partnershop

2. starting with an existing group e.g Wey Landscape Partnership

Steering group; Northumbria River Basin Disctrict Liaison panel

3. starting with data collection and additional stakeholder analysis e.g.

Tidal Thames, .

Clearly, not only are catchments engaging with core groups of stakeholders (what has

been termed the “catchment group” but is also called a “steering or project group”)

but they are also going out to meet others e.g. meeting with Cotswold Farming forum

as well as holding wider events to engage with different stakeholders e.g. stall at

opening of a park and members of the public e.g. public engagement event jointly

between Transition Durham and the Lower Wear Pilot.

4.2.2 Engagement activities: most and least useful

Questions 110 and Q111 drew out the methods that people felt were the most useful

and least useful for engaging with their “catchment group”1. to help reach a vision

1 It is recognised that not all catchments have a “catchment group” see question 45



and shared understanding of current problems and to identify action and

encourage commitment to action. Across both of these the most useful methods

were not were not very different as shown in Table 4.3.

Table 4.3: Q110. What engagement methods did you find most useful for engaging

with your catchment group?

Most useful type of engagement method Number of instances of engagement method being mentioned

Group meetings: facilitated, deliberative, small, 25 Emails 10 Conversations, discussions: one to one 12 Information provision: newsletters, objectives sheet 6

Group meetings were generally found to be the most useful method for engaging with

the catchment group. These varied in size and in their formality, some applied

specific techniques such as backcasting and carousels whilst others were just group

“discussions”. One on one conversations were identified repeatedly as a useful

approach. Emails were felt to be a useful way of encouraging comments on specific

aspects of work – such as draft materials and reports– as well as being effective at

accessing information. A number of catchments used presentations, newsletters or

posters to share information with stakeholders. Interestingly, those more deliberative

methods appear to be considered most useful in the reaching a vision and shared

understanding stage whereas whilst discussions were still considered most useful to

identifying actions some more focussed approaches appear here such as

“temperature checking” with stakeholders, “presentation to liaison panel”, and “task-

finish group set up”.

Across both actions the least useful methods were not very different and these are

detailed in the Table 4.4.

Table 4.4: Q110. What engagement methods did you find least useful for engaging

with your catchment group?

Type of engagement method Number of instances of engagement method being mentioned

Group meetings: un-facilitated discussion, facilitation exercise, first steering group meeting; full group workshops

9

Emails: sending out emails; emailing information 7 Conversations (one to one) 0 Information provision 2



A number of the respondents indicated that un-facilitated group discussions were not

very useful (rather than group discussions generally). The use of emails and websites

as a form of information provision was also indicated as not useful by respondents.

4.2.3 Engagement activities: engagement with the public and stakeholder

feedback

Q128 and 129 focussed on whether there had been opportunities to engage with the

public and whether stakeholders had opportunities to feedback to the pilot this

quarter. 14 of the 21 pilots that responded to Q128 indicated that there had been

opportunities to engage with the public in this quarter, whilst 7 said that there had

not been opportunities to engage with the public in this quarter. In terms of the 11

opportunities they listed, three categories emerged

Direct, public face to face opportunities: e.g. Launch Event for parish council

reps and local residents to attend (3)

Face to face via existing groups, representatives: e.g. Via exisiting

community groups, and Groundwork Northamptonshire (7)

On-line opportunities: e.g. website has gone live and contact details are there

for public to use if they want to know more. (1)

A number of the pilots ticked YES to Q 128 but from the comments it was clear that

whilst they had the intention to carry out engagement with the public they had not

taken any opportunities to do so in this quarter mainly because it was felt to be too

soon in their pilot.

Coming on to stakeholder feedback, 12 of the 20 responses to Q129 which focussed

on stakeholder feedback, said that there had been opportunities for their stakeholder

to give feedback in this quarter, and 8 said that there had not been opportunities this

quarter for their stakeholders to give feedback.

Feedback was gathered via event feedback forms, emails, verbally, steering group

meetings and through discussions at workshops. The feedback received (9 responses)

ranged from “Mainly positive” (2), to “mixed” (2), “some” (1), “give clear instructions

for workgroups” (1) through to “None” (2) and “None as feedback was direct to

evaluators” (1). In terms of what changes people have made on the basis of the

feedback they included changing language, instructions, accelerating the

development of task groups and informing the EOI for the Catchment Restoration

Fund. Feedback on the participant survey by Cascade was also requested.



4.2.4 Engagement activities: good practices

Question 112 asked for examples of good practice for a number of different activities

and these are summarised below with the verbatim comments in Appendix X.

Good practices for...organising meetings: respondents stressed the importance

of local venues and good catering, of planning, and of having focussed meetings using

third party facilitators.

Good practices for…communicating with stakeholders: respondents stressed the

importance of understanding stakeholders‟ organisations, views and priorities and of

finding common ground. They also suggested using existing networks and making

individual connections with stakeholders.

Good practices for…communicating with the public: respondents stressed use of

jargon free language together with respect for local knowledge, use of Parish Councils

and of innovative engagement methods e.g. world cafe2,, open space3 questions,

river mapping4

Good practices for…resolving conflict: respondents stressed that it was

important to seek help if you felt out of your depth. Third parties, chairs and

independent facilitators were suggested as solutions as well as being open and

honest.

4.2.5 Engagement activities: success in responding to views

Given the amount of engagement activity over the last quarter it is useful to reflect on

how successful people felt the pilot has been in responding to the views of all the

stakeholder involved. This was asked in Question 109. Of the 21 responses, 17 felt

the pilot had been very or quite successful in responding to the views of all the

stakeholders involved, and 4 did not know.

4.2.6 Engagement activities: use of visualisation tools

In response to Q124, (17 responses) 10 pilot hosts said that visualisation tools had

been used to engage stakeholders in this quarter and 7 said that they had not. Of

those who said that visualisation tools had been used Figure N shows what tools were

used and how successful they were considered to be:

2 http://www.theworldcafe.com/

3 http://www.openspaceworld.org/cgi/wiki.cgi?AboutOpenSpace

4 http://www.publicengagement.ac.uk/how/guides/participatory-mapping

http://www.theworldcafe.com/

http://www.openspaceworld.org/cgi/wiki.cgi?AboutOpenSpace

http://www.publicengagement.ac.uk/how/guides/participatory-mapping



Figure 4.1: Q125. (13 responses – respondents could give more than one response)

Other tools used were: GIS outputs within Power Point presentations; mapping of

WFD failure and a spreadsheet tool with links to waterbody maps. The specific

examples of visualisation tools used by pilots can be found in Appendix X.

4.2.7 Information sharing and data

Q115 – 122 focussed on information sharing and data. Looking at the responses to

Q115 there information is shared predominantly by email between members of the

catchment group with other methods being used to a lesser extent. Fig N shows that

there is room for improvement on sharing of information with respondents mostly

indicating that it is considered “Quite successful” with a minority suggesting they

“don‟t know”.

0% 50% 100%

Interactive web-based tools

Other

GIS-based tools

Maps

Very successful Quite successful Not successful Don‟t know



Figure 4.2: Q115. In what ways is information (e.g. meeting notes, dates of

meetings, general information) shared between members of the catchment group?

(20 responses – respondents could give more than one response)

Figure 4.3: Q117. How successful has the information sharing been in this quarter?

(18 responses)

In terms of data/evidence, Q116 asked if data/evidence (e.g. technical information

about the catchment had been shared between the stakeholders this quarter. Of the

21 responses, 16 indicated Yes and 5 indicated No. The types of data shared fell into

four categories:

Technical, specialist data e.g. findings from nutrient investigations, SSSI

condition assessment, data from Catchment Sensitive Farming (CSF) projects (7)

0% 50% 100%

viii. other

vii. social media

vi. media (e. g. paper, radio)

v. on an ad hoc basis

iv. regular newsletter

iii. circulation via email

ii. website with access only for catchment group members

i. website with public access

Percentage of responses per option

0% 50% 100%

Don‟t know

Not at all successful

Not successful

Quite successful

Very successful

Percentage of responses



General unspecified data e.g. key past reports (3)

Stakeholder activity data (1)

WFD status data (6)

The data comes mostly from the EA (7) followed by water companies (4) with specific

projects (CSF and Nature Improvement Areas (NIA)), steering group members,

wildlife trust, Natural England and the Forestry Commission providing data to just

one or two projects. The full list of data can be found in Appendix X

Figure 4.4: Q119. What were the major challenges of sharing evidence, data and

information? (18 responses)

Respondents provided good advice on how to overcome these challenges which are

summarised below.

How to overcome:

Data gaps (because the data doesn’t exist)… Working with stakeholders

was identified as a useful way to address missing data as stakeholders are

often able to indicate alternative data sources.

Data gaps (because the data isn’t being made available in time)…

Better planning and being persistent were indicated as the most effective ways

to address this challenge. Working with key stakeholders such as the EA was

also felt to be useful.

Data incompatibilities… There was no quick fix for this challenge other

than using „technical solutions’.

0% 50% 100%

Other (please specify below)

Communication and presentation of data to make it understandable

Balancing technical data and stakeholder information and views

Data gaps (because the data doesn‟t exist)

Getting consensus on what the data/information means

Data incompatibilities (data not in a format that can be used with other sources)

Data gaps (because the data isn‟t being made available on time)

Not a problem A minor problem

A problem that caused delays A problem that severely hampered progress

No view



Getting consensus on meaning of data… Working with specific

technical experts as to the meaning of the data was considered to be useful

but some respondents also suggested that it was too early to be developing

consensus.

Balancing technical and stakeholder information sources…

Agreeing what level or types of data were suitable for specific issues were felt

to be the best way to overcome this challenge. It was accepted that certain

groups were not set up for this at the moment but that “forums for shared

problem solving” could address this.

Communication and presentation of data… A number of respondents

indicated throughout this question that there were too much data and that

they were overwhelmed by the available information. This was particularly

relevant for the communication and presentation. Respondents suggested

that visualisations was useful but others felt that specific capacity was

required to do this and that therefore a learning event could be held on this

topic.

4.2.8 Support from Defra and its network

Question 130 -131 investigated support from Defra and its network. Out of the 21

responses 12 indicated that they had support from Defra and its network in the last

quarter, and 9 said that they had not. 28 pieces of support were mentioned in the

survey from 12 pilots. The types of support ranged from provision on information

across technical (e.g. WFD failures data) and process (e.g. collaborative working)

issues through to conversations and exchanges of views on specific issues. Of the 28

pieces of support 24 were considered useful or very useful and 1 was rated as not

useful. Of the remaining for one the request had not been responded to yet, for one

the information had not yet been used so it was not possible to say how useful it was

and for the final one the data had not yet been received so it was not possible to

assess its usefulness.



5 LEARNING

The learning questions focused on two aspects: what kind of learning pilots think

would help them to deal with the challenges ahead (Q106-107) and the learning

methods and tools they are using (Q133-138). The wealth of information provided in

the responses indicated that most pilots involve learning processes which are already

providing valued input.

5.1 CHALLENGES FOR THE NEXT QUARTER

Respondents predicted that they would face quite a range of challenges in the next

quarter and, probably as a result, identified a wide variety of topics they would like to

see covered in the next learning event. As no pre-determined categories were

provided, the responses were analysed using categories related to types of task (Table

5.1).

Table 5.1: Upcoming challenges by type of pilot activity

Categories of pilot activity Number of times mentioned as challenge

Number of requests for learning topics

Engaging partners, stakeholders and the public 17 6 Vision building 1 0 Catchment planning and prioritisation 10 5 Information collection, management and analysis 4 5 Understanding costs / Funding 5 3 Managing the process 13 5 Delivery 4 0 Other 2 3 Total responses 56 27

Challenges relating to working with others were often described in terms of

„managing expectations’, ‘potential conflict resolution’. Several pilots indicated that

the scale of the engagement activities was itself a challenge („handling the likely

volume of contacts‟, „sheer number of potential stakeholders in the catchment area‟).

Managing the process was another common theme: some responses mentioned

specific tasks or outcomes to be achieved while others identified challenges

associated with more general issues like „keeping up momentum’, lack of time and

difficulties of prioritisation.

5.2 FUTURE LEARNING EVENTS

As expected, the topics suggested for future learning events often mirrored the

challenges. They were also similar to the main training needs described by pilots in

Q10, which focused on stakeholder engagement (46%), process management (12%)



and information and data management (12%). Some respondents suggested pilot

learning events as a good way of addressing training needs, particularly in relation to

process issues. A few pilots had very specific information or learning needs, such as

„Heavily Modified Waterbodies mitigation measures‟ and „An outline on all the

strategic frameworks and EU directives that could be delivered within the

catchment‟.

Looking at pilots‟ experience of using five types of learning methods or tools, the

survey found that each had been used by some of the pilots, the most popular being

„advice and support from participants in the project‟. All of the methods were felt to

have been useful by at least 50% of participants. The people most likely to be

involved in learning were identified as the Pilot Coordinator and staff or members of

the pilot host organisation. Only 6 pilots said that members or staff from participant

organisations had been involved in learning.

Figure 5.1: Q133. What learning methods or tools have you used this quarter and how

successful have they been? (21 responses)

13 pilots identified important lessons that they had gained by using learning methods

and tools. Four mentioned „process planning‟ and others described specific lessons

such as „how to run the workshop in September 2011‟ and „Keeping the main group

relatively small and developing wider subgroups‟. However, most of the lessons were

more general; the comment, „Just gathering a broader understanding of the issues

involved and the aspirations of DEFRA and the EA‟ reflects the feedback from a

number of pilots.

0% 50% 100%

Other

Advice and support from people/ organisations outside the pilot project

Advice and support from participants in the pilot project

Other workshop/ meeting with pilots

Pilot learning event

Review/ learning meeting (hosts/ participants only)

Very useful Useful Not very useful Not at all useful No view Not used



Only 8 pilots provided information on how the learning events had helped them to

share good practice. Of these, more than half described discussions, meetings and

networking as ways of sharing. One mentioned regular liaison with another pilot

catchment through a Rivers Trusts coordination meeting.

For the next Quarterly Review, we suggest having only one question on training or

learning needs as these cover the same information.



6 BENEFITS

The benefits section of the Quarterly Review sought the views of pilot hosts on what

they expect the likely benefits of the pilot to be (Q139-150). 21 responses were

received of this section of the survey. Given that a number of the pilot catchments are

at a relatively early stage, most responses were associated with likely benefits rather

than actual ones.

6.1 PLANNING AND ENGAGEMENT

Pilot hosts identified that their pilot has led to, or is likely to lead to, a wide range of

benefits for planning and engagement (Figure 6.1). The largest benefit identified was

for improved stakeholder buy-in to catchment objectives and delivery. Other

significant benefits identified included an improved understanding and prioritisation

of problems within the catchment and agreement on actions that will have multiple

benefits and actions that will be delivered collaboratively by more than one party.

Figure 6.1: Q139. Planning and engagement. To what extent do you think that the

pilot has led to, or is likely to lead to, the following benefits/positive outcomes: (21

responses)

Other benefits identified from the survey include the additional partnership support

0% 50% 100%

j. Other benefits from enhanced planning and engagement?

i. Better integration with other planning processes?

h. Identification of additional funding, or funding opportunities?

g. Agreement to actions that will be delivered collaboratively by more than one party?

f. Agreement to actions that will have multiple benefits?

e. Agreement to actions to improve the catchment, that have not been agreed to previously?

d. Improved data/information availability and sharing?

c. Buy-in to further engagement / continuation beyond the initial phase of the pilot?

b. Improved stakeholder buy-in to catchment objectives / delivery?

a. Improved understanding and recognition of the problems/pressures in the catchment and which need

action first (priorities)?

Large benefit Medium benefit Small benefit No view Not likely



from a diverse range of stakeholders that have not previously worked together (e.g.

trusts, local councils, schools, angling clubs, etc.) and a co-ordinated partnership

application for catchment restoration fund support and alignment of the pilot

schemes with NIA.

6.2 ENVIRONMENTAL BENEFITS

Pilot hosts identified that their pilot has led to, or is likely to lead to, a range of

environmental benefits (Figure 6.2). The main benefits identified relate to

improvements in water quality and increased biodiversity, which were predominantly

large or medium benefits. Other largely beneficial outcomes include reduced flood

risk and increased confidence that the catchment will not deteriorate. Responses

indicate that the avoidance of carbon emissions and improved water availability were

the least likely environmental aspects to benefit from catchment pilot work (although

both aspects were still consider likely to see a benefit in around half of the catchment

pilots).

Figure 6.2: Q141. Environmental benefits. To what extent do you think that the

pilot has led to, or is likely to lead to, the following benefits/positive outcomes: (21

responses)

Other responses from the survey include the benefits of stakeholder engagement

upon strategic support plus the incorporation of projects to the wider green

infrastructure agenda. A number of responses noted the expectation of longer term

environmental benefits from the pilot schemes.

0% 50% 100%

g. Other environmental benefit?

f. Avoided carbon emissions?

e. Increased biodiversity?

d. Reduced flood risk?

c. Improved water availability?

b. Improved water quality?

a. Higher confidence that the catchment will not deteriorate?




6.3 SOCIAL BENEFITS

Pilot hosts identified that their pilot has led to, or is likely to lead to, a range of social

benefits (Figure 6.3). All responses were predominantly thought to lead to large and

medium benefits. The largest benefits identified were positive communication and

reputation, plus an improved awareness across catchment communities. Other

benefits identified from the survey include the impact of the pilots on community

engagement (e.g. volunteer groups, local farmers, etc.), public access to the water

environment and a feeling of public ownership within the catchment. Given that a

number of the pilot catchments are at a relatively early stage, most responses were

associated with the likely benefits rather than actual benefits of the pilots.

Figure 6.3: Q143. Social benefits. To what extent do you think that the pilot has led

to, or is likely to lead to, the following benefits/positive outcomes: (21 responses)

6.4 ECONOMIC BENEFITS

Pilot hosts identified that their pilot has led to, or is likely to lead to, a range of

economic benefits (Figure 6.4). Predominantly medium and small benefits identified

were efficiency savings and the reduced need for expenditure. Other benefits

identified from the survey include the encouragement of collaborative working,

access to funding for job creation and delivery from government funding. The

majority of other responses to the survey were that the economic benefits were

unlikely or unquantifiable, given that most of the pilots are still at an early stage. One

response noted that the specific economic benefits of the pilots were hard to identify

at present given that the scheme is being delivered concurrently with other activities

(e.g. CSF, WFD).

0% 50% 100%

d. Other social benefits?

c. Increased recreational use of the catchment?

b. Improved awareness across wide catchment communities?

a. Positive communication/ reputation?




Figure 6.4: Q145. Economic benefits. To what extent do you think that the pilot has

led to, or is likely to lead to, the following benefits/positive outcomes: (21 responses)

6.5 WIDER ENVIRONMENTAL BENEFITS

Around half (46%) of pilot hosts said that their pilot has explored the potential to

include and deliver wider environmental objectives and ecosystem benefits. Half of

these said that considering these wider benefits was very useful or useful, while a

third of respondents had no view (as yet) and none identified this exercise as not

useful. Additional comments include that some pilots have only just started and that

plans for undertaking ecosystem approach are in commission or under consideration,

with outcomes yet to be realised.

Figure 6.5: Q148. If the pilot has explored the potential to include and deliver wider

environmental objectives and ecosystem benefits, how useful has this been? (9

responses)

80% of pilot hosts said that the catchment approach did, or will, inform Regional or

River Basin District level planning. Examples from these pilot hosts include liaison

0% 50% 100%

d. Other economic benefits (e.g. ecosystem services)?

c. Positive impacts on employment, profitability, competitiveness, health?

b. Reduced need for expenditure because collaboration at the catchment level has

identified wider catchment based solutions?

a. Efficiency savings (e.g. as a result of improved targeting or sequencing of actions)?


0% 50% 100%

No view

Not at all useful

Partially useful

Useful

Very useful

Number of responses



with key delivery partners (e.g. EA, water companies) and the identification of local

issues. Several pilot hosts were hopeful that the catchment approach will inform

Regional or River Basin District Level planning in the future. Of the 80% of pilot

hosts who said that the catchment approach did, or will, inform Regional or River

Basin District level planning, around two thirds (62%) felt that this route is more

accessible to specific stakeholders. Additional comments include that the process

may improve accessibility to stakeholders not already representing an organisation

through an interface with DEFRA/EA. One issue highlighted by pilot hosts was that if

statutory plans do not reflect the aspirations of local stakeholders there may be a risk

of them losing faith in the process and of the lead organisations.



7 COSTS AND VALUE FOR MONEY

This section asked about the expected costs of the pilot to the host organisation (the

participant review asked about costs to others). Costs were broken down by:

Staff (hours or full time equivalent (FTE) posts dedicated to the pilot);

Other financial costs (such as travel and equipment); and

Less tangible costs (e.g. damage to relationships or environmental impacts).

There was generally a low response to this section, reflecting uncertainty in

estimating costs at a relatively early stage of the process. It is likely that more robust

and complete estimates will be provided as the process progresses.

In summary, the reported costs of the process to date are generally modest and

mainly relate to staff time, sub contracts and meetings. However, there is a

significant degree of difference in reported costs across the pilots even at this early

stage.

For EA-led pilots, the amount of staff time spent on the pilots in the last quarter is

shown in Figure 7.1.

Figure 7.1: Q152a. How many staff days (FTE) have been spent this quarter? (10

responses – EA-hosted pilots only)

The average amount of time spent in the EA-led pilots was just over 50 days. For

non-EA led pilots, responses were in the range 12 to 55 days (mean 34 days). Whilst

for some pilots, the process is “part of the day job”, others have allocated a specific

amount of time (e.g. a few days per month).

Other financial costs reported across the pilots were generally fairly trivial. A small

0 20 40 60 80 100

Staff days (FTE)

EA

pil

ot



majority of pilots (13) reported travel costs ranging from “minimal” to £1,000 (mean

£270). Other costs in this category were subcontracts (reported by 2 pilots, mean

£3,250) and consumables (including workshop venue hire, 11 responses, mean

£290).

Figure 7.1: Q153. Compared to the budget submitted to Defra, or the anticipated

costs, roughly would you say you are: (16 responses)

Most pilots claim that their initial cost estimates are turning out to be about right. At

this stage, just 2 pilots say they have overspent compared to their initial estimate, and

2 have under spent.

Figure 7.2: Q154. To what extent do you think that the pilot has led to, or is likely to

lead, to: (21 responses)

Only 3 catchments thinks that there could be any less tangible costs associated with

the pilot, the vast majority stating that the impacts will all be positive. However,

whilst it is probably too early to make any judgments in this area, one pilot sounded a

note of caution by observing that funding has only been provided for developing a

catchment plan, not for implementing this plan. There is therefore a danger of

“disenfranchising stakeholders who invest in developing this plan but see no actions

resulting from it”.

0 5 10 15

Don‟t know

Very over spent

Over spent

About right

Under spent

Number of responses

0% 50% 100%

c. Other?

b. Negative environmental impacts?

a. Damage to reputation/ relationships?

Not likely Small cost Medium cost Large cost No view



8 FEEDBACK

Nineteen pilot hosts provided feedback on the Quarterly Review form, and their

responses and comments are discussed below.

Figure 8.1: Q179. How long did this form take to complete? (18 responses)

About one third of pilot hosts completed the review in the 1-2 hours first envisaged at

the launch event and the majority too less than three hours, however a large number

(44%) of respondents took more than three hours to complete the review (Figure 8.1).

The length of the questionnaire was felt to limit the provision of more detailed

responses to individual questions in the time available. Several pilot hosts comments

that they felt the form was too long, such that the evaluation process was not

proportionate to the work done in catchment pilots to date.

Figure 8.2: Q180. How was the format? (19 responses)

Most respondees (70%) found the format of the form easy to use. Pilot hosts who

found the form difficult to use commented on the need for greater clarity and less

repetition, as well as making the form easier to navigate. Comments will be taken on

board where possible for the next Quarterly Review, including providing a blank copy

of the survey to pilot hosts to provide an overview of content and structure to help

0% 50% 100%

More than 3 hours

2-3 hours

1-2 hours

Less than 1 hour


0% 50% 100%

Very difficult to use

Difficult to use

Easy to use

Very easy to use




plan their responses. Some suggestions will not be possible given the current set-up

of the online form, including making it possible to spell-check text or upload

documents, or skip multiple pages at a time.

Figure 8.3: Q181. How well did the questions capture what is involved in running a

catchment based approach? (19 responses)

More than half of respondees (60%) felt that the questions in the Quarterly Review

form captured the process of running a catchment based approach well or very well.

Suggestions for improvement in terms of clarity and repetition will be taken on board

where possible for subsequent reviews. Most comments on the content of the form

centred on the difficulty of asking the same questions of such a diverse range of

catchment pilots at different stages of development. Although the survey has been

designed as far as possible to allow for this, it is recognised that hosts of pilots in

their early stages are not yet able to provide detailed responses to some questions.

Similarly, the hosts of more developed catchment pilots commented that some

questions were too prescriptive, for example in referring to approaches such as

stakeholder analysis when this may not be seen as necessary in a catchment with a

long history of engagement.

In response to this feedback on report structure and content, the questionnaire will

be shortened and the format and process improved to make it easier for pilot hosts to

provide relevant information in the next Quarterly Review to be issued in June.

0% 50% 100%

Not at all well

Not very well

Well

Very well




APPENDIX A: CONTEXT TO THE QUARTERLY

REVIEW

A.1 THE CATCHMENT-BASED APPROACH EVALUATION PROJECT

A series of catchment-level partnerships are being developed by Defra through a pilot

phase (May 2011 – December 2012) to test a new approach to improving the water

environment through catchment-level engagement and planning. Ten of these

partnerships are being hosted by the EA and a further 15 pilots are being hosted by

stakeholders such as the water industry, rivers trusts and wildlife trusts. The pilot

phase therefore provides the opportunity for the 25 catchment-level partnerships to

learn from one another, and develop effective approaches to stakeholder engagement

through adaptive management and continuous learning.

A key part of piloting the catchment-based approach will be evaluating the

experiences, to capture learning which can be shared by the pilots during the process

and with others in the roll-out from 2013. With this in mind, evaluation material will

be gathered from the pilot hosts, the pilot participants and, for five pilots, through

observation.

In this context, pilot hosts have been asked to provide the evaluation team with

information through a series of evaluation forms across the pilot period (Table A.1)

along with providing copies of materials produced as part of the pilot exercise.

Table A.1 Timetable for Pilot Host Evaluation Forms

Information Purpose

Timings

Period

covered

Forms

returned

Feedback

provided

Baseline

Characterisation

form

To understand key differences

between pilots, including specific

objectives set by the pilots

themselves and any self-evaluation

processes adopted.

Starting

position March 2nd March 16th

Quarterly Review

form

To review activities, to highlight

potential challenges coming up and

identify practices that it would be

worth sharing to help design the

learning events. To look at changes

for example, in who participates in

the pilot over time.

Jan–March

April–June

July–Sept

April 23rd

July 13th

Oct 12th

May 11th

July 27th

Oct 26th

Final Review To assess the feasibility of objectives Jan–Dec Jan 18th Feb 8th



Information Purpose

Timings

Period

covered

Forms

returned

Feedback

provided

form and milestones adopted, to identify

what worked well and what was

difficult, how were common

challenges overcome, to identify

practices that it would be worth

sharing in the handbook, to assess

costs and benefits across the range of

pilots (picking up different focuses

and links for comparative analysis).

2012

A.2 DESIGNING THE QUARTERLY REVIEW

The first Quarterly Review form was designed to learn as much as possible about how

to implement catchment-based approaches. At the Launch Event of the pilot scheme

in January 2012, feedback was sought from pilot hosts and other stakeholders with

regards to how the evaluation process could most effectively be carried out.

A consultation on the Quarterly Review was carried out in January/February 2012 to

seek pilot hosts‟ views on the evaluation process and the suitability of questions in

the Quarterly Review to pilot hosts‟ work. The responses from this consultation were

reviewed and incorporated into the design of the Quarterly Review where feasible,

with the proviso that the evaluation needs to be robust and based on consistent and

comparable material.

The EA was also consulted on the design of the Quarterly Review to ensure that this

one review will support their internal evaluation work as well as the wider Defra

catchment pilots‟ evaluation project.

An online survey tool, Survey Methods, was used to carry out the Quarterly Review.

This tool had also been used for the participant survey and was select to improve the

ease of use of the survey and improve collation of data for analysis.

A.3 IMPLEMENTING THE QUARTERLY REVIEW

The aim of this questionnaire was to learn as much as possible about how to

implement catchment-based approaches. The questionnaire covers all of the steps

envisaged for running a catchment-based approach over the 12 month pilot period,

with the intention that responses build as the pilot progresses.

It was made clear to pilot hosts that it is not a performance assessment. In cases

where pilots are being jointly hosted, it was recommended that co-hosts work



together to fill in the form, although separate versions would be accepted if this

wasn‟t possible.

A.4 THIS REPORT

When the evaluation framework was designed, the overall aim of the quarterly

reporting was to:

„review activities, to highlight potential challenges coming, up and identify

practices that it would be worth sharing, to help design the learning

events, and to look at changes for example, in stakeholder

groups/participants over time.‟

This report is not the only output from the quarterly reporting, but it is intended to

summarise the high level messages. In drafting the report we have had the following

audiences in mind:

1. Defra (and to an extent the EA) – who will want to know progress overall, and

whether pilots need additional support/direction?

2. Pilot Hosts – who will want to know where they sit within the pilots (who has

similar problems) and will want some solutions to solve challenges they are

facing

3. EA – who will want to know how their pilots are progressing

4. Learning Events team – who will want to know what could most usefully be

included in the learning events

We have tried to keep the text short and punchy, use figures to show range of

responses. We provide links to case-studies or other material that pilots might find

useful.

Our intention is to support learning within the pilots, not to give any impression that

we are creating a league table, after all pilots have been selected to representative of

the whole range of situations across catchments in the UK, so each situation is

different, but there are common challenges.

Please note: many of the questions are ranked based upon the pilot hosts

judgement of progress or how well things are going, we have not attempted to

standardise these. We are sending the report out quickly so that can be shared as we

are aware that many pilots are facing the challenges of collaborative planning at

present and we did not want to delay passing on messages and lessons learned. Many

pilots commented that it was too early to provide tips and share lessons learned,

however many others provided key messages that we hope others will find useful.



APPENDIX B: TIMESCALES AND DELIVERY

Table B.1 Pilot Hosts’ own assessment of progress against key activities in their pilots

Run-Time (m)

Base-line Activity Mapping

Stake-holder Mapping

Planning stake-holder involve-ment

Development of Comms plan

Setting up a catchment group

Agreeing specific objectives for the pilot

Developing a shared vision

Agreeing catchment problems and priorities

Walk-overs

Agreeing actions and gaining commit-ment

Establishing outline costs and relative effectiveness of actions

Evalua-tion

Adur and Ouse 10 In full In full Mostly Partly No response

Leam 4 In part Mostly In part Not very Mostly Mostly In part Mostly Yes In part Not at all Yes

Upper Tone 5 In part Mostly In part Partly In full Mostly In part In part Yes In part Not at all Yes

Lower Lee 8 In part Mostly In part Not yet In full Mostly In full Mostly No, but Not at all Not at all No

Lower Wear 8 In part In part In part Partly Mostly In part Mostly In part Yes In part Not at all Yes

Don and Rother 9 In part Mostly In full Partly In full Mostly In full In part Yes In part Not at all Yes

Ecclesbourne 7 Mostly In full In part Partly In full In part In part In part Yes In part Not at all No

Welland 8 Mostly Mostly Mostly Partly In full Mostly In full Mostly Yes In full Mostly Yes

Irwell 12 Mostly Mostly In part Partly In full Mostly Not at all In part Yes Not at all Not at all Yes

Tamar 1 Unsure Mostly In full Not very Mostly In full In part In part Unsure In part Not at all Yes

Tidal Thames 3 In part Mostly In part Partly No No Not at all In part No, but Not at all Not at all No Bristol Avon & North Somerset Streams 4 In part In part In part Not yet In full In part Mostly In part No, but In part In part No

Cotswold 3 In full In full In full Partly Mostly Mostly Mostly In full Yes Mostly In part Yes

Bradford Beck 1 In part In part In part Not very No No Not at all Not at all Yes Not at all Not at all No

Tyne 1 In part In part Not at all Partly In part In part Not at all Not at all No, but Not at all In part No

New Forest 2 In part Mostly Mostly Partly Unsure In part Not at all In part Not yet Don‟t know Not at all No

Nene 3 In part Mostly In part Not In part In part Not at all In part Yes In part Not at all Yes

Teme 1 Mostly Mostly In part Unsure In full In part In part In part Yes Not at all Not at all Yes

Wissey 1 Mostly Mostly In part Not very Mostly In part In part Mostly Yes Not at all Not at all Yes

Frome & Piddle 5 Mostly Mostly In part Not very In full Mostly In part In full Not yet In part In part No

Wey 11 Mostly In part Mostly Partly Mostly In part In part Mostly Not yet In part In part No

Tame 1 Not at all Mostly In full Partly In part No Not at all Not at all No In part Not at all No



Table B.2: Summary of key areas of performance: set out as objectives, principles, milestones/products

Ref Area EA pilots Non-EA pilots

1a Common objectives/principles developed by Defra/EA for pilots: CO1 Obj1: Ensure early identification and involvement of relevant partners

in dialogue. Yes Yes

CO2 Obj2: Develop a shared understanding of the current problems in the catchment.

Yes Yes

CO3 Obj3: Ensure co-ordinated activity that will deliver multiple benefits for the environment.

Yes Yes

CO4 Obj4: Gain commitment from partners to more catchment action and ambitious targets.

Yes Yes

CO5 Obj5: Embed learning and share current and early successes in implementation.

Yes Yes

CO6 Obj6: Assess the implications of adopting catchment working more widely in future.

Yes Yes

There may be other specific objectives developed by pilots for themselves 1b Other requirements specified for non-EA pilots (7 and 8 are principles for EA but 9 and 10 are not mentioned so would need to ask EA to include) CO7 Provide a single point of contact in each catchment5 Yes 3 Yes 4

CO8 Share data and information5 Yes 1 Yes 2 CO9 Form a catchment group and involve them in decision-making No Yes CO10 Organise catchment „walkovers‟ to identify pollution issues No Yes 2. Milestones/Products

P2 Baseline stakeholder and activity mapping

Identify current activity and issues, and potential partners and networks. Set up stakeholder group.

Sep 2011 Mar 2012

P3 Stakeholder feedback for each pilot

Ongoing evaluation of stakeholder feedback.

Quarterly reports

P4 Catchment appraisals and agreed catchment priorities for each pilot catchment

A brief description and shared understanding of the problems in each catchment, to inform the catchment „plan‟, and a shared vision.

Dec 2011 Jun 2012

P5 Catchment ‘plan’ for each pilot catchment

To show key problems, shared vision, action required, and commitment from partners. Description of key services provided by the catchment and relative value to users. Register of the outline costs and relative effectiveness of actions identified. Format to be defined locally.

Mar 2012 (draft), Dec 2012

Dec 2012

P6 National evaluation report

Including case studies, recommendations, lessons learned and resource assessment for future approach.

Quarterly reports

Dec 2012 (final)

Notes to Table:

Ref Objectives and Principles for EA pilots as stated in „Principles and Evaluation Pack v1‟

Non-EA pilot principles according to the declaration signed in the EoI‟s submitted by pilot hosts

1 Wording is: The EA will: Be generous in sharing our data; Use information and learning from others; Let others lead in determining what information and engagement is needed

2 Wording is: Share information and help develop agreed joint action plans to deliver the best solutions to achieve WFD objectives



Table B.3: Example Summary of Activity

Establishing outline costs and relative effectiveness of actions Progress to March

2012

Not started In part Mostly In Full

15 5 1

Challenges:

• Quantifying planned advisory visits and influencing change.

• Detailed costing of land-use intervention projects.

Support needs:

• Assistance with costing complex water issues (not given so far).

• Understanding what agency resource is available.

• Creating a baseline for funding NGO activity.

Key tips for others:

• An integrated framework that involves a combination of agencies, LAs and NGO activity can be

enabled if there is an understanding of what can be funded by government organisations and

what must be separately costed for delivery. Establishing what can be done 'in house' and what

can be done by supporting partners is key to the delivery framework.

• Delegate tasks to partner organisations who are happy to help as it will directly help them

deliver their own objectives.

• Invite all partners to work up annual budgets together to deliver a shared plan. This process

should be able to start at the commencement of financial year planning in late autumn/winter,

budgets populated together from all available funding streams through Jan - March ready for an

integrated funding framework from April 1st. Working up six months in advance of the

beginning of the financial year enables funding gaps to be identified that can become funding

bids, and end of year under-spend can be allocated and spent.

Case-studies:

1. Drayton Habitat restoration project ( Welland)

2. The Welland has a fully costed work programme. For habitats and diffuse pollution works,

effectiveness of actions will be evaluated through pre and post works monitoring.

3. Resource Protection Group leaflet. (Welland, see evidence folder.)

4. Working up business plans together in Gloucestershire on site specific projects, e.g. (outside

the Catchment) restoration of Leckhampton Hill SSSI which facilitated the biggest ever

capital grant under stewardship ( £690,000) that was end of year underspend and achieved

multi objective delivery working with partners and the local community. (Cotswold)



APPENDIX C: ENGAGING, SHARING AND SUPPORT

C.1 EXAMPLES OF GOOD PRACTICE (Q112)

Organising meetings

Venue and catering

Try to go central in catchment or near to main town/city for public transport links.

Use different venues for variety. Provide good refreshments!

Good quality venue and catering

Eco-themed or otherwise relevant venue

Planning

Get other stakeholders involved.

Speak to stakeholders about best meeting style

Identify availability of key participants first then extend invitation to wider group.

Meeting process

Keep the meetings short and sharp, with everyone bringing something to contribute

Have a third party facilitator

Plan, prepare, practice

Good meeting aims, use online resource scheduler tools to gauge availability

The more preparation the better and really useful to have a separate facilitator so that

could participate as a delegate

Communicating with stakeholders

Meet them where there are at

Understand their priorities and organisation.

Do your homework on what your stakeholders priorities are and find the common

ground

Use appropriate language for respective stakeholder typologies

Make sure they recognise a delivery benefit for their objectives

Don't bombard them early on; adopt a phased approach

Use existing networks

Making the maximum use of existing networks and partnerships

Be personal

Face to face and phone

Speak one to one where possible before meetings



Communicating with the public

Use existing networks

Use parish council contacts

Respect

Don't use jargon. Use language that they will identify with

Value and respect local knowledge

Use innovative methods

Planning a joint drop in session on 26th April with Transition Durham that trial

different techniques: ' world cafe5, open space6 questions, river mapping7

Resolving conflict

Bring in a third party to aid resolution

Seek help it you feel out of your depth.

Mediate through chair person

Use a skilled independant informed facilitator

Openness and honesty

C.2 EXAMPLES OF VIEWS ON DATA/EVIDENCE SHARING (Q116)

In response to Q116 (What types of data/evidence has been shared and from whom)

the following comments were made:

Findings from WW nutrient investigations presented at meeting. Maps of WFD

status and SSSI condition assessment provided at inception meeting and launch

event.

Wide range of pertinent data from the EA, and some form Anglian Water. Also

projects such as CSF and NIA.

Key past reports shared with steering group via email

Water company, Natural England, CSF data and evidence.

Water quality data from STW and EA

General background data on catchment from NE, WCC and Forestry

Commission"

EA EasiWFD database to Steering Group

5 http://www.theworldcafe.com/

6 http://www.openspaceworld.org/cgi/wiki.cgi?AboutOpenSpace

7 http://www.publicengagement.ac.uk/how/guides/participatory-mapping

http://www.theworldcafe.com/

http://www.openspaceworld.org/cgi/wiki.cgi?AboutOpenSpace

http://www.publicengagement.ac.uk/how/guides/participatory-mapping



EA Maps of WFD classification of water bodies

Greater Manchester Waste Disposal Authority (GMWDA) GIS map of their waste

sites

List of stakeholder activity in the Pilot area from Steering Group members"

Evidence shared between members of the Resource Protection sub group e.g.

water quality, metaldehyde, phosphate data between EA, Anglian Water and

Game & Wildlife Conservation Trust (G&WCT).

Though it is available - it has been requested that the draft catchment plan that

includes the data, is internally reviewed by the EA. It will then be presented by

the EA to the pilot steering group and available in early May. As pilot hosts there

has been access to this draft information for sometime

Contained within the Draft Wey CIP

Expand data workshops, Intention to send out updates, Spreadsheet tool

Principally from the EA

Visual maps of WFD element status were prepared and shared wit the group by

EA

WFD information relating to classification of water bodies

Further mapping information has been shared about land use, ownership, and for

council Green Infrastructure Strategy.

The Derbyshire Wildlife Trust have provided a verbal update at each

Collaborative Group Meeting on progress with baseline ecology scoping report.

Watershed Improvement Plans (WIPs) for some watercourses, WFD status data.

From presentations at initial project development group meeting. Also hand-outs

at the meeting prepared by EA/Pond Conservation.

In response to the request for examples of data/evidence or information or how it

was shared (Q116) the following comments were made:

Using figures in Excel, put into Power Point

"Water quality - chemical and biological, sedimentation surveys, metaldehyde

Habitat opportunity mapping

e mails, PDF, CDs etc

Within workshops, via presentations and via meeting minutes (email).

WFD failure of surface water bodies and ground water relating to good ecological

status and proposed actions by the EA



WFD classifications

Supplied by the EA and shared via email.

Summary of catchment WFD condition currently and issues.

C.3 EXAMPLES OF VIEWS ON VISUALISATION TOOLS (Q127)

In response to Q127 (Please provide examples to support you view on the success of

visualization tools in your catchment) the following comments were made:

Interactive web based tools are in development, primarily centred on web based

maps which demonstrate how existing projects sit within the catchment

Poster sized maps of the Teme: enabled all stakeholders to get a feel for the size

and amount of work being carried out in the catchment. Enabled overlap of work

to be visualised.

These maps stimulated high level of discussion as evidenced by the number of

post-it comments attached to them during the carousel process in our initial

stakeholder meeting

The draft catchment plan produced by the EA has easily recognisable and

distinctive maps showing GES failure by colour coding from Bad, Poor, Moderate

to Good. This makes the data easy to understand for all.

Common IT tools used (word excel and powerpoint) but range of IT skills and

familiarity of stakeholders we are trying to work with is an issue.

Need to build capacity of stakeholders to be more effective. Whose job is this -

resource?"

Understanding of where people worked and where the catchment and waterbody

boundaries are.

defra evaluation of the catchment-based approach pilot stage · first quarterly review draft final...

Documents