dŵr cymru welsh water pr19 customer engagement programme · 2019-07-25 · 3 assurance report for...

Research Assurance Report for the Chair and Members

of the Dŵr Cymru Customer Challenge Group

Dr. Dimitrios Xenias

__________________________________________________________________________________

Cardiff, April 2018

Dŵr Cymru Welsh Water PR19 customer engagement programme

吉

2

Assurance report for CCG

Contents

Executive summary ................................................................................................................................ 3

1. Background and scope ................................................................................................................... 4

2. Research questions ........................................................................................................................ 7

3. Research process ............................................................................................................................ 9

4. Discussion on research methodology .......................................................................................... 11

a. What is triangulation? How is it generally applied? ............................................................... 13

b. Individual research briefs and fit within the overall framework ............................................ 14

c. The application of the Willingness to Pay (WtP) research...................................................... 17

d. The DCWW internal (contact) data .......................................................................................... 20

e. The process for allocating weightings to different data sources ............................................ 21

f. Balance between informed and uninformed customers ........................................................ 22

g. Degree to which future customer needs are reflected in the evidence base ........................ 24

5. Missing elements - gaps ............................................................................................................... 25

a. Tensions .................................................................................................................................... 25

b. Longitudinal view ..................................................................................................................... 25

c. Customers’ views translation into action ................................................................................ 26

d. Lack of an established tracking mechanism ............................... Error! Bookmark not defined.

e. Company’s relationship with the CCG ..................................................................................... 27

6. Recommendations ....................................................................................................................... 28

7. Lessons learned from this exercise .............................................................................................. 30

8. Conclusions and next steps .......................................................................................................... 31

9. References .................................................................................................................................... 32

Acknowledgments

The author would like to thank the fellow members and Chair of the Consumer Challenge Group

for their views; the research companies, CCWater and Ofwat for contributing their views and

experience on this consultation process, and Dŵr Cymru Welsh Water for supporting this work.

3


Foreword from the Chair

The Customer Challenge Group (CCG) has a key role in the Price Review process in providing

independent challenge to Dŵr Cymru Welsh Water and independent assurance to Ofwat on

- The quality of the customer’s customer engagement; and

- The extent to which the results of this engagement are driving the company’s

decision making and being reflected in the company’s business plan

This report was commissioned by the CCG to contribute to its assessment of the first of

those roles.

The scale and intensity of the customer research undertaken by the company over the

period was such that as Chair I felt it important that we drew on the specific research

expertise CCG member, Dr Dimitrios Xenias, to take an overview of the research.

It aims to provide the CCG with assurance that the company has established an effective

evidence base from a range of customer data sources to allow the development of a

business plan that is based on customer priorities.

On behalf of the CCG I would like to thank Dimitrios for his work on this report which will

make an important contribution to our final independent report to Ofwat.

Peter Davies

Chair Dwr Cymru Welsh Water Customer Challenge Group

4


1. Executive Summary

1. The PR19 process to date presents a significant improvement compared to the PR14

experience but there are also opportunities for further improvement. It is clear that

DCWW has put much effort and resource into customer engagement since early

2016. The company has also broadened their research scope compared to PR14. This

expanded the breadth of available data and methods, and provided welcome

addition of qualitative information

2. This report, while highlighting areas for improvement and specific concerns,

concludes that the company has been successful in securing an understanding of the

views of its customer base from the Phase 1 research and triangulation process

3. This report only covers Phase 1 of the Willingness to Pay research, but it will be

important that the company provides reasoning on how it uses WtP to influence its

decisions and that the significant fluctuations are interpreted in the context of other

evidence known to the company. The report references the continued concerns over

the nature of WtP methodology despite the changes from the PR14 process. It will

be important that the company draws on the wide range of evidence gathered and

not depend on this one metric for key decisions in the business plan.

4. The report recognises that DCWW has committed significant effort and resource to

customer engagement activities. The report does though provide recommendations

for future improvements, outlined below:

DCWW and the CCG and other stakeholders would benefit from a summary

document of the whole customer engagement and research process.

Long term approach requires long term commitments: DCWW should prioritise the

development of its approach to permanent, representative, trackable consumer

panels.

Most research seems to be outsourced. DCWW might benefit if it retained this

knowledge and expertise e.g. through an enhanced in-house research capacity.

There needs to be a mechanism to track specific customer recommendations/

preferences and translate them into action e.g. ‘you said we did’ or similar

document.

Internal DCWW data, e.g. call centre contact data and written complaints, need

better indexing, breakdown and presentation. Some source data is presented in

5


PR19 Customer engagement – Phase 1 Companion to main report – summary of

evidence; however, it is not clear where the data came from, how it is categorised

and what do different categories entail.

There seems to be a plethora of relevant documents both from Ofwat and DCWW,

many of them undated. These need to be dated and mapped in one place, perhaps

in a document similar to that prepared by PwC to map the research process Phase 1.

It would also be useful to include links to important documents/reports in this

‘roadmap’ document.

In the end of the customer engagement and consultation period, produce an

updated summary of the process and main findings, and compare with initial plans

and findings.

DCWW website needs to be improved: If the customer is at the heart of the process,

it should also be made easy for them to participate and comment via the web site.

5. The CCG was engaged in the early stages of design and development of the customer

research but the report documents the challenges for the CCG in responding to the

scale of the research programme reducing opportunities for co-design. This process

can be improved in the future and needs to be part of the post PR19 review of the

working of the CCG.

6. The key test will now be the degree to which the company is able to assimilate the

customer evidence base it has gathered and apply to the business plan. CCG will

want to ensure there is a clarity of reference to the strength of the customer

evidence base in the defining of bills levels, measures of success and delivery

incentives

6


2. Background and scope

In December 2015 Ofwat launched their “Water 2020” consultation (Ofwat, 2015) which

kick started the process of the Price Review 2019 (PR19) consultation phase. In that

document, Ofwat set out their views on customer engagement, and their expectation of the

water companies to engage with customers deeper and over the longer term, as well as an

expectation of companies to come forward at the PR19 “with solutions developed in

conjunction with their customers” (Ofwat, 2015, p.5). In the same document, Ofwat stated

their intention to see the customer at the heart of the business planning process, building

upon the progress made in PR14.

In its customer engagement policy document (May 2016) Ofwat stated:

“For the purpose of PR19, CCGs have a very clear role. This is to provide independent

challenge to companies and independent assurance to us on:

the quality of a company’s customer engagement; and the extent to which the results of

. this engagement are driving decision making and are reflected in the company’s plan.”

. Source: Ofwat, (2016a) p.3.

Subsequently, in March 2016 Dŵr Cymru Welsh Water (DCWW) reconstituted the Customer

Challenge Group (CCG). It was chaired by Peter Davies, appointed in February 2016, and

comprised some members who had participated in the PR14 process, as well as new

members with policy, water and research expertise to help steer the DCWW process into

PR19.

Early in the summer of 2017 it was agreed that the author would provide a light touch,

independent commentary/narrative on the Welsh Water Customer Engagement

programme and evidence base. The rationale behind this was that the CCG will use this

commentary as part of an independent assurance of the DCWW customer research

programme, with specific focus on how DCWW addressed its customer engagement

expectations, how and whether customers’ views were used to address the DCWW strategic

objectives set out by Ofwat’s expectations, and how these were expected to be

incorporated into the company’s PR19 business plan.

This brief report attempts to address the first of Ofwat’s two points, i.e. to answer the

question of whether:

- customers were appropriately engaged,

- their evidence was properly collected, reported and presented; and

7


- whether the process has provided a good evidence base of customer views for the

company’s PR19 business plan submission.

The report is set in the context of the challenging nature of the CCG’s capacity to keep track

of an intensive period of customer research, to ensure that all sources of customer data are

being used in building the evidence base and that the correct messages are drawn out in the

formulation of the business plan.

Apart from reviewing the validity of the results of the DCWW customer research

programme, this report will also draw comparisons, where possible, with the PR14 process

as a starting point for the Ofwat expectations and DCWW plans. Over 1500 pages of

relevant documentation were reviewed, including policies, reports and results from the

PR19 research process and any available PR14 documents.

3. Research questions

While researching any topic, it is important to first ask why this topic is researched. This is

the most important question at the start of any research project, as it helps focus the

research effort, choice of methods, interpretation of results, and drawing of conclusions. It

is very tempting to embark on any number of research projects, using sophisticated

methods and state of the art approaches, but without asking why the research is done in

the first place, extracting results and reaching conclusions can become a futile exercise –

and yet this fundamental question is often neglected.

In our case in point, while DCWW had adopted its vision “to earn the trust of customers

every day” and driven a customer led success programme, the specific PR19 methodology

represented an externally imposed imperative with Ofwat’s expectations for a better

understanding of the customer base and an increased participation of the customers across

the business. Ofwat is not alone in this approach and they seem to follow an emerging trait,

mainly from the US, where other regulators have similar requests for their respective

industries (Hahn, 2017). Although it is not always clear what customer involvement may add

to some aspects of running the business, it is often assumed there is a great opportunity to

help customers move from passive recipients of water company services to active

participants in helping achieve a successful long-term future for water in England and Wales

(Ofwat, 2017b).

DCWW set out a 3 phase programme of research

8


Phase 1 - General attitudes – to shape the measures of success and anchor point for bill

levels

Phase 2 - Key priorities and trade-offs – to understand customer valuations and trade-offs

Phase 3 - Plan Acceptability – to test overall package with customers

Phase 1 included 12 different research topics with a further six in Phase 2, added to which

there was an extensive triangulation process with other sources of customer data.

This was a significant scale of response to the expectations set out by Ofwat, representing a

step change from PR14. However, although there was a clear framework and each piece of

work had a clear remit, the overall impression by reading DCWW’s research to date is that

the main emphasis was on what was being done, where, when and how, but not clearly why,

and so this question - linking all of the previous ones - was sometimes obscured by facts.

Articulating ‘why’ research overall was conducted, as well as individual pieces thereof,

would help DCWW focus on evaluating the quality of research findings, lessons learned and

actions needed; moreover, this would help internal and external stakeholders appreciate

and evaluate any research outcomes and ask ‘so what’ type of questions which in turn

would help guide the company’s next steps. This should permeate all relevant internal and

external documents, reports and results, perhaps adopting a format of ‘why’ this piece of

research was conducted, ‘why’ was it important and ‘so what’ - what do implications do the

findings have for the company.

It is, however, important to mention at this point, that DCWW went to considerable lengths

to employ novel methods in order to engage their customers, and collect valuable data for

the 2019 price review. The significant scale and range of research means that there is a

challenge to ensure that:

- each individual piece of evidence clearly contributes to the big picture as opposed to

sitting on its own

- there is a clear line of sight from the ‘why’ through to the learning and actions

- tensions between mixed messages from different sources are clearly identified

- the company can properly assimilate the scale and range of research findings and

evidence from other data sources in shaping the business plan

9


4. Research process

One of the areas of focus of this report is whether the chosen research process and

methodology were appropriate to answer the research questions necessary to inform the

PR19 process.

The brief for the public tender for research agencies was issued prior to the first meeting of

the reformed CCG in March 2016. The CCG was engaged much earlier in the research

process than in PR14, but the tender was formed prior to the reformed CCG, which meant

that there was a lost opportunity to contribute to the shaping of the initial research remit

and the strategic level of thinking around evidence needed to inform the business plan. As a

result, the basic research questions were not clearly articulated in the documentation.

However, the CCG workshop in May 2016 contributed to shaping the research framework

setting out key topics for the research, the range of methods, groups to target and the CCG

engagement plan.

It is important that the process of engagement and subsequent working with the research

agencies is reviewed. Speaking with the agencies as part of this review it is clear that while

there are areas for improvement there are also aspects of good practice in the relationship

between agency and company that have contributed to the effectiveness of the research.

It is also clear that the scale of the ambition of the research put significant stretch on the in-

house expertise in managing the relationships, the research programme and the

assimilation of the results.

DCWW advertised the research tender at the end of 2015. Once research companies were

selected, they were invited to develop, in collective discussions with DCWW, the research

approach for PR19. The fact there was more than one company on the framework contract

enabled greater scope for innovation. The company made it clear to the three companies

involved that they would have to share results with each other and also that they would

have to submit mini tenders for pieces of work designed to encourage innovation and to

keep costs low.

It is difficult to assess the degree to which this aim was successful, particularly given the

issue of competition across agencies working for different companies with confidentiality

limiting innovation and cross-company sharing.

The customer research process began in June 2016, when DCWW also appointed a research

manager, with several other senior DCWW members involved in the parallel streams of data

10


collection and management. This gave DCWW and the CCG about two years in total to

collaborate and scrutinise the process, a clear improvement on PR14. However, given the

breadth and volume of research and tight deadlines, the CCG found it both difficult to

comment on the detail of the research and to set each piece of work in the context of the

big picture of what each was intended to contribute.

It could be argued the process started with outdated assumptions and expectations, using a

PR14 baseline as opposed to the much heavier requirements resulting from the ambitious

step change implemented for PR19.

The overwhelming data volume was also appreciated by DCWW, which in January 2017

appointed Pricewaterhouse-Coopers (PwC), who, due to their extensive experience with

PR14, were well placed to advise and assist with research volume management and formally

document the framework process and the process of data triangulation. A workshop

involving PwC subsequently organised the ongoing DCWW approach to customer

engagement in three phases:

Phase 1 Getting a better understanding of the general attitudes of customers to shape

and inform the Measures of Success (MoS)

Phase 2 Understanding how customers prioritise trade-offs and the value they put on

them both in the short and long term

Phase 3 Test the overall acceptability of the Business Plan with customers

Although these phases were not conceived as clear research questions at the outset of the

process, they are useful to help map the process for the benefit of all stakeholders.

However, ideally such planning takes place, and is articulated, at the outset of the research

process.

This report is primarily concerned with the quality of data collected in Phase 1 and how this

will be used to inform the business plan for PR19 but this is also inseparably linked with the

process of engagement and data collection, and the customers’ role in the process as Ofwat

has clearly requested.

The nature of what is meant by ‘customer engagement’ is important to consider in the

context of the research process. Ofwat commissioned the “Tapped In” report which charted

the nature of the progression of customer engagement through to active involvement. The

expectation of customer participation was defined by Ofwat as

11


“The active involvement of customers in the design, production, delivery, consumption, disposal

and enjoyment of water, water services and the water environment in the home, at work and in

the community.” (Ofwat, 2017b, p.3).

A summary of key milestones for the customer research assurance review is provided

below:

Timeline

February 2016 Peter Davies appointed Chair of the CCG

March 2016 Customer Challenge Group (CCG) established

May 2016 Ofwat released ‘customer engagement expectation’

document. CCG 2nd meeting.

June 2016 Researcher kick-off meeting

January 2017 PwC appointed

May 2017 End of customer research Phase 1

Spring – summer 2017 Phase 2 data collection

September 2017 Summer consultation results presentation. CCG 7th meeting

November 2017 Phase 2 completion. Customer Research Assurance report

discussion (CCG). CCG 8th meeting CCG XXth meeting

January 2018 Customer Research Assurance report reviewed

April 2018 Customer Research Assurance report submission to CCG

September 2018 PR19 business plan submission to Ofwat

5. Discussion on research methodology

A main question for this report is whether the chosen research methods and triangulation

data were appropriate for the task and applied fairly – in other words, whether the research

data provide a fair representation of the customers’ preferences and opinions. If this is

12


established, then the next question is whether these data are appropriately reflected in and

guide DCWW’s proposed business plan. The latter question is, however, outside of the

scope of this brief report.

Public engagement is a prerequisite for data gathering, and the type and quality of

engagement dictates the type and quality of data collected. Hence, it is important to

consider both aspects when scrutinising DCWW’s PR19 engagement and research process.

For PR19, DCWW employed a range of research approaches which were implemented

across a range of households (HH), including vulnerable and hard-to-reach customers, and

non-household customers (NHH). These methods included:

a) Qualitative research such as:

Focus groups

Face to face interviews

Online qualitative interviews

Teledepths (in depth telephone interviews)

Breakfast events

Exploratory workshops on Willingness to Pay (WtP)

b) Quantitative research such as:

Online surveys

Telephone interviews

Surveys on Willingness to Pay (WtP)

Surveys following up customers who contacted DCWW (Rant and Rave)

Data was also provided by the Consumer Council for Water (CCWater) who conducted

independent research to both HH and NHH customers, as well as secondary data analysis.

The breadth of methods employed in this PR19 customer research, was substantial. The

customer samples covered were also broadly representative of a good spread of the

population and businesses.

However, as previously mentioned many CCG members felt that there generally was not

enough time allowed for meaningful CCG co-design of the research methodology. In

general, short turnaround times for paper approval before meetings, and limited available

expertise, did not leave much room for CCG to challenge or co-create research.

13


In the following sections specific aspects of the research approach are discussed in more

detail with the aim to improve the process for the benefit of both the company and the

CCG.

a. What is triangulation? How is it applied?

Triangulation in its essence is a rather simple principle, and not far from ‘common sense’. It

refers to the practice of viewing an object, or topic, from more than one viewpoints, thus

obtaining a better idea of what the object or topic looks like. The two viewpoints and the

object of observation are considered to form a triangle, hence the term. Consider the

following graph:

Farquhar and Michels (2016) describe a classical view of triangulation in social sciences as:

“using multiple and independent measures to improve the certainty of conclusions about a

hypothesis being investigated, by obtaining a fix on the phenomenon under investigation

from two known points”. In other words, triangulation intends to use multiple perspectives

to minimise bias and establish the validity of estimates.

It follows that the more abstract the topic under investigation, and the more complex the

data, the more important it is to establish clear rules on how to perform triangulation, what

data to consider or disregard. The main concept is that at least two viewpoints or datasets

should confirm a result or observation. In other words, that at least two datasets should

agree before we accept what these datasets tell us.

14


For the purposes of PR19, following input from PwC, DCWW established the following set of

rules on how to apply triangulation:

Triangulation principles: 1. More weight should be placed on data/information which is consistent with other

sources – e.g. where there is an established regulatory or industry consensus/view,

less weight should be placed on ‘outliers’ (unless there is strong evidence of local

nuances).

2. Most weight should be placed on data/information that is fit for purpose (collected

using a methodology which has been designed appropriately for eliciting customer

priorities/preferences) – e.g. where the methodology has enabled customers to

explicitly identify their priorities as opposed to where we have needed to infer

customer priorities.

3. More weight should be placed on data/information which is more robust and

reliable (statistically significant, consistent/repeatable/stable, and intuitive /

coherent). Data/information which is less reliable will still be considered, though only

for context or as corroborating evidence.

4. More weight should be placed on more recent data/information, except where

there is reason to suggest recent evidence is less reliable (or where a longer-time

series is required).

5. When considering comparative information, more weight should be placed on

data/information from closer comparators (companies with similar demographics,

issues/challenges, etc)

Source: Dŵr Cymru Welsh Water (2017a) p.5.

This set of principles is not dissimilar to the triangulation guidance issued by CCWater in

their July 2017 event in London. They have been applied by DCWW in the weighting of

datasets in this research. The basic point is to mutually assure datasets that are comparable

and to downplay findings from datasets that are disparate, or non-comparable. From such

evidence only limited conclusions can be drawn, and must be treated very cautiously for

important decisions.

b. Individual research briefs and their fit within the overall framework

15


Individual research briefs covered a broad remit, encompassing many aspects and

employing a wide range of methods and techniques. This was certainly an improvement

from PR14 as DCWW has broadened their scope, methods, and use of research companies

compared to PR14. This is commendable for the company, as was their drive for early

engagement with CCG and their customer base.

The full range of individual research pieces can be found in DCWW (2017b) and set out in

the appendix to this report. Of the five categories of data presented in DCWW (2017b), we

will focus on sections 2 (Primary qualitative research) and 3 (CCWater research) here.

Sections 1 (historical performance) and 4 (PR14 research) refer to past research, while

section 5 (continuous engagement) is covered in section (d) of this chapter.

A general observation from looking at this body of research is that the research briefs

reviewed so far were different and not directly comparable to each other. Comparability

versus breadth is a common trade-off in social and market research, and it depends on the

choice of different samples, methodologies and research questions. When opting for a

deeper understanding of an issue, there is always a risk that limited comparisons might be

drawn from across this research, such that only limited conclusions can be drawn from the

data. This applies to the present evidence base. From the evidence provided it is difficult to

view the company’s research programme and ensuing evidence base as a whole research

programme with strategically structured priorities and corresponding results. As a result,

important comparisons and judgement decisions need to be made on what datasets answer

which questions. This also makes triangulation more challenging.

On the other hand, implementation of widely comparable research across the piece would

have sacrificed depth of understanding: DCWW might ask several important questions on

e.g. a 5% or 10% sample of their entire customer base, and this would be very strong and

comparable evidence; however, such extensive survey would never be able to cover the

variety of topics that the present research did, and in such depth.

The range of independent research projects with different purposes and methodologies

generate a range of responses depending on the nature of the research. An example of

incomparable data, in seemingly two similar research briefs, is the customers’ top priorities:

‘Customer priorities’ research (spontaneous responses)

‘Summer consultation’ research (forced choice responses)

Priority 1 Cost was reported as the main priority for customers – reducing costs, keeping costs down and finding ways to support customers in reducing their bills.

Cleaner rivers and beaches

Priority 2 The majority of customers reported being unwilling to pay for improvements to the service,

Working with nature for cleaner water

16


particularly for investments where they will not experience the benefit

Priority 3 Customers do not prioritise improvements to core water and wastewater services, but requested improvements to the customer service offered by DCWW

Better water quality for all

The process of triangulating this data and reflecting the nature of differing results from the

range of research is critical in making judgements on customer priorities for the business

plan.

Two other pieces of research to highlight were the ‘Customer Service Priorities’ and the

“Have Your Say” research. The Customer Service Priorities research essentially focused on

‘what good looks like’ to customers. This piece of research was interesting in that it

compared quality of service across a range of services, and so its findings can be directly

translated into business plan actions and measures of success that lead customer service

improvement. The other interesting piece was the ‘Have You Say’ piece, where DCWW

attempted to correlate consumer preferences with other factors e.g. socio economic status

group. This was interesting research because it might be useful for future survey planning,

e.g. as a ‘horizon scanning’ exercise.

The coverage of non-household (NHH) customers featured in five of 22 research briefs, two

of which from CCWater. Given that market liberalisation in the NHH sector is not taking

place to the same extent in Wales it is important that the CCG is assured that the level of

research focused on this sector is sufficient to reflect its differing priorities.

My key observation in this section however is that since most of these research briefs look

at separate aspects, it is difficult to get a sense of continuity and cohesion. The research

framework and individual briefs appear to be geared for snapshots, rather than for

continuity. It is important to ensure that there is a strategic cohesive approach to research

planning and execution, which establishes main and secondary research objectives, and

appropriate milestones toward them.

In order to reflect Ofwat’s philosophy on customer engagement and participation it would

be expected that both company targets and customers’ perceptions, priorities, preferences

and views are treated as waypoints in a continuous process which spans beyond PR19. The

intensive research exercise undertaken for PR19 needs to be seen in that context as

opposed to a one off exercise related to requirements of the specific methodology.

It is clear that the company does track trust, satisfaction, value for money, performance

data and other measures beyond price reviews and such baseline data and longitudinal

tracking of key metrics are vital elements that contribute to the evidence base. The focus

17


has tended to be on the specific PR19 research but it is important that the CCG is assured

that there is a focus and transparency on ways in which the company can improve customer

data from business as usual processes; moreover, focusing commissioned research on

identifiable gaps/areas of concern where more in depth views are required.

For example, the company has recently established an online consumer panel which as the

capacity to adopt leading edge practice. Similar approaches would help to create a more

informed customer base that will know what their service is, what its value is, and what they

expect from the company. Such longitudinal tracking will provide valuable insights on

several aspects of the customer base and help the company better respond and prepare for

future strategies beyond Price Reviews. Also, tracking a bare minimum of indicators

longitudinally would solve some of the method comparability issues previously highlighted.

The CCWater ensemble of 11 research briefs (DCWW 2017a) particularly its annual Water

Matters Industry Household customer Tracking Survey and it biennial Non Household

Customer tracker ‘Testing the Waters’ were very useful for the assessment of the relative

position of DCWW in the sector – but lacks in-depth insights. CCWater research could not,

by necessity, afford the same depth as focus groups and qualitative research. Its strength

lies in its potential to help calibrate DCWW’s performance ahead of PR19 and to help

DCWW understand its place in the utility landscape.

The benchmarking of company performance must play a vital role as a driver for business

improvement. Customers need to be aware of how the company performance compares if

they are to make informed judgements on priorities. It is recognised that this is not a

straightforward process given the very differing nature of the water companies, their

geographical coverage and infrastructure, social disparities and so on. However it is critical

that such benchmarking is seen as priority both in informing priorities in the PR19 plan and

in the ongoing process of customer engagement.

Overall, the individual briefs appear to sit well within the engagement framework. However,

the framework itself might need updating as research realities unfold. This is natural to

research, where initial plans are rarely followed to the letter, and are usually modified as

research projects progress. For example, some intended triangulation points are not readily

available in a comparable format (see contact centre section) and further important findings

may emerge in Phase 2 research - which are out of the scope of this report.

c. The application of Willingness to Pay (WtP) research

Willingness to Pay research is of specific interest as it represents an approach that was

mandated by Ofwat.

18


Elements of the WtP work are being conducted as part of Phase 2 in DCWW’s research,

which is currently ongoing. Therefore, the report is only able to comment on the WtP

research that took place as part of the Phase 1 research programme.

The WtP methodology was prioritised early on in the PR19 process, coinciding with the early

days of the reformed CCG and at that point many of its members felt that the CCG was not

able to appropriately and meaningfully contribute to the research design process. The WtP

research preceded a CCWater report (CCWater, 2017) on Improving Willingness to Pay

which produced recommendations on WtP research. It would be valuable for the company

to provide a retrospective review of the WtP research work undertaken against the report

recommendations

The study of stated preferences in Willingness to Pay research is complex, technical, and

prone to biases, as explained below. One major requirement to properly conduct WtP

research and obtain meaningful results is to thoroughly educate participants on the method,

options and trade-offs involved; these aspects are resource and time consuming and were

explained in great detail by Steve Arnold of the Environment Agency at the CCG chairs’

meeting in Birmingham in January 2017.

One of the main issues with this research is that in practice research companies tend to only

administer a WtP preferences questionnaire without the earlier educating steps. The danger

then is that participants respond to the WtP statements on their face value and with little

understanding of the trade-offs involved. This is not how stated preference studies were

conceived and meant to be executed, and can distort its results and limit the value of

outcomes.

The research contractor did put considerable effort into improving and simplifying the WtP

methodology in PR19 compared to PR14. A simpler more accessible approach was

recognised as being important, building on the feedback from PR14. However, the CCG also

recognised the importance of comparative trend data, so would recommend that “an option

of a parallel exercise based on the PR14 model is considered” CCG, 2016, p.3). Following

discussion with the CCG it was agreed to pilot both the old and the new methodology

simultaneously, so that the results of both methods could be compared.

Ofwat recommends that, although important, stated preference WtP should be used as only

one of several measures and should be triangulated. This approach is critically important

and it is important that the company evidences a range of sources in making decisions as

opposed to relying on the results of the WtP process. There is a real danger in over reliance

of WtP to provide the “single right answer”.

It is also crucial when concerning unexpected results; for instance results such as a 34%

increase in package value in WtP for PR19 compared to PR14. In recent communications

19


between Accent (the contractor) and the CCWater, some alternative explanations for the

WtP results were offered by Accent, but they are not triangulated. Given that Accent has

also experimented with new versions of the WtP materials across the sector, a relevant

question is: what were their findings compared to PR14 results and what are their

conclusions with regard to the present differences between PR14 and PR19 for DCWW?

Moreover, there are many documented problems associated with stated preference

approaches (such as these employed for this round of WtP research) that may make these

approaches unreliable (Hahn, 2017). These include a number of issues not unfamiliar to

DWCC’s WtP evidence base for PR19: questions of a hypothetical nature; participants’

unfamiliarity with the decision environment or the goods or service proposed; the inquirer’s

power to exercise potential strategic design of the materials in order to influence whether a

service will be provided or not; the impact of how a question is framed for an otherwise

identical service (Ariely et al., 2003, Arrow et al., 1993; Diamond and Hausman, 1994); and

the impact of question framing on the way participants respond (Consumer Council for

Water, 2017).

Given the lack of technical expertise on WtP in the CCG, an external peer review of the WtP

results was required; just as the WtP methodology was initially peer reviewed. This is

common practice in general in the social sciences, especially when a new method or

experimental statistics are used.

DCWW and Accent did put in further effort to alleviate some of these concerns, and

commissioned an external peer review of the WtP results (Willis, 2017). Professor Willis

gave a positive review on the WtP research piece; however key questions remain

unanswered. For example, he only speculated on why the differences between PR14 and

PR19 research occurred; this implies the need for further points of triangulation before we

can trust these results, especially if the WtP results were to be used to inform business plans

or price changes. At the same time, the nearest possible set of results (although not directly

comparable) from the Summer Consultation (Blue Marble, 2017) concluded that there was

no consensus about preferred bill levels in the future, significant proportions of respondents

were for and against a £10 annual increase (corresponding to around 2% of the average bill);

whereas Accent (2017) found that 2/3 of customers feel their bill levels are ‘about right’,

and about a quarter feel that they overpay.

It is important that WtP research is further triangulated if its results were to play any role in

influencing price changes in PR19 and indeed if triangulation principles (see DCWW 2017a,

p.5) were to be followed strictly, WtP results may be disregarded from this exercise: the

present results are not in accordance with other datasets. Therefore, this piece of research,

meticulous as it may be perceived to be (Willis, 2017), only shows an isolated result which

includes (a) some unexplained results – which may or may not be due to the change in

20


methodology- and (b) some results that are not supported by other research -i.e. not

triangulated. Put simply, WtP should be considered as just one of many tools, and like any

tool it may be of high quality but still the wrong tool for the job. The present WtP results

cannot, without further support, be considered sufficient grounds for price changes in PR19.

This would be similar to deciding medical treatment on the results of a single discordant

laboratory test, while the patient’s global clinical picture and further examinations and tests

suggest otherwise. It is therefore important for DCWW to explain its reasoning on what

range of prices it is choosing on the basis of the research it conducted.

d. The DCWW internal (contact) data

An important point of triangulation of the present customer research on priorities was

expected to be the company’s internal data i.e. those coming from its customer base via the

call centre. The importance of this data lies in that (a) it is ongoing, and therefore able to

capture customer issues for a very long period and broader than the PR19 data gathering;

and (b) customer contacts are spontaneous, hence indicating the absolute top priorities that

matter to them, whether that be bills, leaks or floods. While these priorities would not be

exhaustive and certainly insufficient to formulate a business plan, they would be an

important indicator of top priorities.

The analysis which is applied in the triangulation is drawn from internal data sources. The

data is audited independently and so provides an appropriate evidence base. However it

does not make external comparison possible in that the topic, heading, classification and

codes shared by DCWW, do not allow them to be used to infer any conclusions or

comparisons with the rest of the current evidence base. For example, it is not clear exactly

how the contact/call volume presented in p.62-68 of the companion to the main report

(DCWW 2017b) relates to the Red, Amber, Green table presented in p.7 (DCWW 2017a) or

whether a low volume of calls indicates a non-problematic area or an extremely rare event.

It is clear that this is at least in part due to the way the call centre data is collected,

captured, classified and reported in a specific format internally; and this format probably

serves internal purposes, as long as all internal parties are aware of what is included under

each heading and assumptions are made. However, this format did not permit the author or

the CCG to use this dataset for the purposes of triangulation with other datasets collected in

the PR19 research, not least to substantiate claims made in other pieces of research such as

the Willingness to Pay research.

In the example shown below (from DCWW, 2017b, p.62-68), one can notice the same

figures but under different categories. This does not make intuitive sense to a third party,

21


although it probably does internally. Therefore, much more disaggregation, breakdown and

explanation of the meaning of the categories and contents of the call centre data is

necessary if this dataset is to be used as part of the evidence base for PR19.

It is recommended that the CCG has a focused session with the company on the data being

provided through the contact centres and its use in the customer evidence base.

Another category of internal data of potential interest and triangulation, is data from

DCWW’s use of mobile equipment that record customer feedback at the point of interaction

e.g. by repair crews on the property. This has been an important initiative and the CCG

needs to monitor progress of usage and the value of the feedback received. The CCG has

stressed the importance of employee engagement with customers in the field and this data

should provide valuable feedback and triangulation with the contact centre data and other

aspects of customer research.

e. The process for allocating weightings to different data sources

The idea behind allocating weights to different datasets according to several criteria is very

useful; DCWW and PwC went to considerable lengths to present a weighing scheme for the

evidence base. However, the method behind the resulting weights is not clear. This is not

22


entirely unexpected: the evidence base consists of several large datasets, and attempting a

relativistic multi-criteria ranking of all of these datasets would be a hard and time

consuming task – although if it was available it would help guide the process, and increase

transparency.

Instead, PwC and DCWW opted for a more manageable approach, involving heuristics (e.g.

the assumption that all datasets were of equal quality, and that any judgements made

would be made consistently) to arrive to the present allocation of weights outlined in

DCWW (2017a) Appendix 1. Knowing that there is no known ‘best practice for weighting the

different data sources, the present approach was based on general research experience,

PwC’s internal as well as sectoral knowledge, and DCWW’s custom circumstances.

Although it is difficult to provide exact judgement, the present process of allocation

weightings appears to be reasonable. One noticeable omission from Appendix 1 (DCWW

2017a) is the Phase 1 Willingness to Pay. It would be interesting to see how the company

estimated the weight of this particular method. More importantly, although Phase 1 may be

too early for this to be expected, in will be important to see how the WtP and other results

contribute to pricing and target setting through the business plan.

Nevertheless, the implementation of an even more systematised and sophisticated

weighing system across all of PR19 research is no doubt difficult, but the CCG should review

weightings applied and ensure transparency.

f. Balance between informed and uninformed customers

As mentioned earlier in this section, there are trade-offs to be made in this line of research,

one of which is the balance between informed and uninformed customers. The former

could be educated in panels or focus groups, while the latter may be viewed as the ‘person

in the street’. Both views are needed but serve different purposes, as each can answer

different questions.

It is important to have informed or even expert views in the early stages of designing

a policy creating a business plan, or proposing new measures to improve the customer

experience.

However, uninformed views are equally important because they can provide a

‘reality check’ for new policies or measures, inform their acceptability and are a source of

valuable feedback which helps improve the final product.

23


In this sense, a good balance between informed and uninformed customers should be

achieved in the PR19 evidence base for its results to be credible. An examination of the

research briefs for Phase 1 research, shows that the majority of research was conducted in

focus groups. This is a middle ground in the sense that although participants will not be as

uninformed as the person in the street, they will be a lot more educated on the particular

topic of the focus group, as the focus group progresses; some focus groups last two hours

while others even longer. By the end of the focus group, participants clearly cannot be

considered uniformed; although they would not be considered experts.

Therefore, in the present research the views of partly-informed participants appear to have

been captured; which is a good best way to investigate niche or difficult topics, such as

resilience or willingness to pay, and so on. These topics require time and information to be

understood. The truly uninformed sample research seems to be the ‘Return of value’ and

‘Have your say’ pieces, and almost all of the CCWater research since it is implemented

across large samples and often as one-off surveys.

There is no ideal way of balancing informed and uninformed samples in this line of research;

however, it is generally expected that the generic uninformed sample research will stem

from the focused, informed sample evidence. For instance, the return of value research took

place in the summer of 2016, while the ‘Customer Priorities’ piece followed. To follow the

‘informed sample development’ ‘uninformed sample testing’ principle, the ‘Return of

value’ brief should follow ‘Customer Priorities’ and test its key findings at a large,

uninformed sample; not the other way around. This reinforces the earlier point in respect to

strategic planning of all research activities, to provide a clearer thread that connects all the

elements of the research process.

There is also a key difference between self-selecting uninformed samples, the demographic

such as those engaged at certain events and other types of uniformed engagement, which

needs to be taken into account when considering the involvement of uninformed

customers. This particularly applies to the Water 2050 engagement exercise.

Finally, the company did make progress in establishing an online consumer panel and would

stress again the importance in establishing an informed sample, which over time would be a

reliable resource with which to gather deeper insights on important topics, receive early

feedback on proposed innovations, and track sentiment or attitudes over longer periods of

time. If this panel was representative it would reduce the need for repeated, one-off

research pieces and maintain a reasonably long term link between the company and its

customer base-and could be complemented by confirmatory, uninformed sample research

24


as needed. The CCG should review the progress and plans for the consumer panel against

established best practice

g. Degree to which future customer needs are reflected in the evidence base

Future customers’ needs should be a sizeable part of the evidence base in this research

programme because price reviews cover a five-year period and water industry business

plans are defined within a longer term programme; even more so in Wales due to our

unique expectations of the Well-being of Future Generations Act (2015). From examining

the individual research briefs, it appears that only Blue Marble’s ‘Customer Priorities’ had an

explicit future customer component. Blue Marble also facilitated sessions for a Youth Board,

drawing representatives from local secondary schools. Other research incorporated younger

customers as part of their sample. Research conducted online might also have a bias

towards younger customers, due to their use of social media.

In the present research carried out by Blue Marble, DCWW defines future customers as

‘between 18 and 29 years old, who are not yet bill payers’. Perhaps it would be useful to

narrow the age range of this sample to younger ages, e.g. 18-20 years or even younger, just

as the Climate Change Commission for Wales accepted secondary school

pupils/ambassadors from the Funky Dragon, to represent citizens of the future.

There does not seem to be evidence of use of the extensive DCWW education programme

in feeding into to the evidence base or in contributing to establishing youth panels. DCWW

also needs to build on relationships with key youth networks and representative structures

to establish an ongoing future generations’ focus.

It would be useful to clarify what the time horizon might be when we consider ‘future

customers’ in the evidence base, i.e. how many years from now. This is an important

clarification because some water infrastructure, which requires substantial investment,

spans several generations (just think of the Victorian sewers and dams still in use today) and

so the question becomes ‘when are changes and investment dividends expected?’ The

answers might vary if our horizon is 10 vs 50 vs 100 years; the definition of ‘future

customers’ should therefore vary accordingly. Lessons from the work in developing the WFG

Act also suggests that it is older customers, especially grandparents, have a clear concern

over services for future generations.

25


6. Open issues - gaps

This section briefly covers a number of gaps identified during this research assurance

exercise. It is important to clarify that although gaps are identified here, it is also clear that

that the research programme is still under deployment, therefore of the identified gaps

from Phase 1 might have already been covered by ongoing Phase 2 research.

a. Tensions

It was not clear from the available documentation whether any tensions have been

identified through the triangulation process in this evidence base. There could be several

types of tension identified by Ofwat, for example:

“For the purpose of PR19, we expect the CCG report (either in the main body or through an

annex), to include commentary on any concerns the CCG process has highlighted regarding

tensions between delivery of the proposed plan and compliance with statutory environmental

and drinking water quality obligations. This will be particularly relevant to our assessment of

business plan quality in the risk-based review.” (Ofwat 2016a, p.34)

Other examples of tensions that would be worth reporting are possible tensions between

household and non-household customers, or between vulnerable customers and other

groups.

On the potential tensions between environmental and water quality obligations, the CCG

has not seen evidence, and this may be because the research is not yet completed. On the

potential tensions among customer groups’ priorities, the CCG has again not seen evidence.

There are clearly tensions within the current evidence base in respect to different attitudes

to bill levels.

DCWW has responded that they are conducting, at the time of writing, segmentation

research investigating the needs, expectations, priorities, and perceptions of different

customer groups and how DCWW responds to them. This would certainly be a report the

CCG will need to consider before its final submission to Ofwat.

b. Longitudinal view

The regulator has made it clear on several occasions that their expectation of meaningful

public engagement in every aspect of the company’s business is to take a long view; that is,

beyond PR19. This implies that it would be to the benefit of the company to treat

26


customers’ perceptions, priorities, preferences and views not as snapshots, but as

milestones in a process which spans several years ahead.

There are already established mechanisms both internally and externally applied for this

longitudinal tracking and further will be introduced following the price review. It is

important that the specific research snapshots of PR19 research are viewed in this context

of long term tracking.

A specific point relates to the DCWW website where it is not easy for customers to engage

and contribute views. The site needs to be more enabled to encourage customer

engagement on an ongoing basis and not just for specifics such as Welsh Water 2050.

Greater prominence could have given to explaining and encouraging engagement around

the PR19 review exercise

The CCG should work with the company in agreeing the “basket” of metrics from a range of

sources that can be regularly reviewed, which go beyond the headline figures from external

surveys drawing on internal data via online panels, contact centres, employee feedback etc.

This would inform future requirements for commissioned research.

c. Customers’ views translation into action

In the CCGs report to Ofwat for the PR14 exercise, there was criticism on this point: “[…] we

feel that the Company could have made more extensive use of the data that it obtained from its

research and consultation exercise. We recommend that it reflects and builds on this as it further

develops its ongoing research and engagement programme with customers.” (CCG, 2013).

Although the timing and circumstances in PR19 are not the same as in 2013, it is obviously

not yet clear how the results of the customer engagement programme will be translated

into action in the business plan and measures of success.

It will be very important that there is a clear, transparent link between the business plan,

proposed bill levels, measures of success and associated targets and the customer evidence

base. Each element clearly needs to be referenced against the customer research. The

company needs to be aware of the danger of ‘cherry picking’ individual research findings to

justify decisions and to be transparent on judgements made where there are tensions in the

evidence base.

The PR19 process has engaged a large number of customers to differing degrees of depth it

is very important that there is some form of feedback loop to customers in respect of the

impact of their involvement and opportunity for continued engagement e.g. through on line

panels.

27


d. Company’s relationship with the CCG

The company’s view on their relationship with the CCG is described in the PR19 Customer

Engagement Framework (DCWW, 2107c). In that document, the company describes its view

on the expected influence from the CCG as follows: “Where there is a high degree of

judgement required, and the issue is more suitable for co-creation, the CCG will have a

greater role in making these judgements (through applying the principles above). Where the

issue is less suitable for co-creation or where there is lower degree of judgement involved,

the CCG will be informed of judgements” (p.15). The company also provided a map of the

types of issues ranked for importance and co-creation. (See next page)

This analysis is helpful in prioritising CCG opportunities for co-creation and areas to focus

customer participation as recommended in the Ofwat commissioned “Tapped In” report.

The progression to involving customers in key issues is an important priority for Ofwat and

the CCG. The Rhondda Fach project is a good example of this in practice.

However, it should not be the case that the CCG is restricted to meaningfully influence only

those issues within the red rectangle at the top right corner of the graph. The CCG is clear

that an understanding of customer priorities based on the evidence of customer

engagement is embedded as a business imperative across all elements of the business.

Customer trust is based on the whole operations of the business; e.g. how they treat

employees, who they purchase from, and so on, where co-creation may not be feasible but

where customers (and the CCG) have a real interest.

28


Ofwat has stressed the importance of the independent governance of the CCG. Importantly

the CCG has recently strengthened its function establishing an independent secretariat

through Cynnal Cymru – Sustain Wales and by using www.trello.com as a private online

mechanism for tracking the CCG’s inputs. Public minutes are now available on the Cynnal

Cymru website. The tracking of CCG contributions and company response can be improved

through these changes.

CCG should undertake a review with the company of its operations and future work in the

light of the PR19 experience.

e. Recommendations

Overall, DCWW appears to have been genuinely trying to engage with its customers on

various levels. Although the result of this effort is yet to be realised in impact on the

business plan, it must be recognised that DCWW has committed significant effort and

resource to approach the spirit of the regulator’s recommendations in much of their

customer engagement activities.

The further Stage 2 customer research and them into business plan preparations, for

example the translation of customer preferences into Measures of Success will require

further focus form the CCG and is beyond the scope of this report.

Of course there is always room for improvement and areas for further development in the

ways DCWW approached their customer engagement activities in PR19 and therefore, some

recommendations are provided below on the basis of this exercise, and with the aim of

helping DCWW learn and improve during PR19 and in the future.

1) DCWW and the CCG and other stakeholders would benefit from a summary

document of the whole customer engagement and research process, articulating:

- The main reasons behind each piece of research (i.e. “why bother” with it?)

29


- an outline of the methodology e.g. quantitative, HH or NHH, if

representative, informed/uninformed, sample size, only, F2F, telephone,

date or qualitative focus groups etc.

- The rationale and top level findings for each piece of research.

- Crucially, the company’s interpretation of the main findings, and how these

might be embedded into business plans.

- The consumers’ role in the process - how have they or will they steer the

process, going forward?

- A final section in each summary report which identified tensions within

research but also with other datasets and research.

Benefit: this will help the company internally and externally, demonstrating customer

engagement, data collection quality and transparency. Perhaps such document could be

regularly updated and circulated to stakeholders/CCG which have limited time/resources to

scrutinise the entirety of the research process.

2) Long term approach requires long term commitments: DCWW should prioritise the

development of its approach to permanent, representative, trackable consumer

panels.

Benefit: this will allow DCWW to track customer views on changes, policies etc through time-

Ofwat expects a long term approach.

3) Most research seems to be outsourced. DCWW might benefit if it retained this

knowledge and expertise e.g. through an enhanced in-house research capacity.

Benefit: DCWW could retain and improve methods for future reference, understand what

questions to ask and why, and how to design research in the long run and reduce costs (in

time and money), as well as better coordinating all its research strands.

4) There needs to be a mechanism to track specific customer recommendations/

preferences and translate them into action e.g. ‘you said we did’ or similar

document.

Benefit: This will show tracked actions/responses and actions taken. Feedback to customers

is critical, particularly those involved in more in depth exercises

5) Internal DCWW data, e.g. call centre contact data and written complaints, need

better indexing, breakdown and presentation. Some source data is presented in

PR19 Customer engagement – Phase 1 Companion to main report – summary of

evidence; however, it is not clear where the data came from, how it is categorised

and what do different categories entail.

30


Benefit: this will help the company internally and could provide the basis for an external

summary document demonstrating customer engagement and transparency.

6) There seems to be a plethora of relevant documents both from Ofwat and DCWW,

many of them undated. These need to be dated and mapped in one place, perhaps

in a document similar to that prepared by PwC to map the research process Phase

1. It would also be useful to include links to important documents/reports in this

‘roadmap’ document.

Benefit: This will increase transparency and communication of activity and objectives

internally (toward other DCWW colleagues) and externally (stakeholders, auditors,

regulator). The company would benefit from a strong coherent narrative of the customer

story

7) In the end of the customer engagement and consultation period, produce an

updated summary of the process and main findings, akin to point (1) and compare

with the initial plans and findings.

Benefit: This will allow a fair and rounded comparison between the start and end points of

research, how it will inform the business plan, and what lessons can be learned for future

research/data collection implementations.

8) DCWW website needs to be improved: for instance, where can one leave open

feedback? Why not on the first page? If the customer is at the heart of the process,

it should also be made easy for them to participate and comment even if this is

one-off-similar to Twitter; not just within specific fora or similar channels, which

discourage spontaneous feedback.

Benefit: This will reflect and project the company’s commitment to customer engagement in

general, its willingness to accept open feedback from the public.

f. Lessons learned from this exercise

This section highlights some of the learning and observations from the first phase of the

PR19 in the form of broad recommendations for the future benefit and continuous

improvement of both DCWW and CCG.

The PR19 customer engagement and research programme was the product of

several agencies and dozens of researchers who worked over two years to produce a

plethora of documentation and activities. It is no easy task for any single individual

to grasp the entirety of the process and the sheer volume of data and in this respect

there may be aspects that this report has missed or misunderstood. At the same

time, this highlights the importance of early, meaningful and extensive engagement

31


with the CCG and other stakeholders, and the need to allocate adequate time and

resources for the challenge process to be substantial and successful.

Research ownership appeared to rest with external consultancies, which might not

benefit DCWW in the long run. It is understood that in-house research expertise was

lacking and this is by no means unique to DCWW or the utilities’ sector. However,

the regulator’s imperative for deep and purposeful customer engagement and

insights only seems set to continue and increase through PR19 and beyond, and in

this respect DCWW might consider building on and expanding on in-house research

expertise which would provide continuity of customer engagement and

understanding.

There seems to be no single document reflecting the customer engagement process

and documenting relevant resources. This made the location of relevant documents

and sources difficult, for external users such as CCG members. This was partly

addressed by the PR19-customer engagement summary document, commissioned

by DCWW and produced by PwC in the spring of 2017 - which however does not

cover/map related documents and sources. Such document would help the CCG and

other stakeholders tremendously.

Many CCG members felt that there was not enough time for meaningful co-design of

the research methodology between CCG, DCWW and the researchers. The short

turnaround times involved, as well as the lack of available time, resource and

expertise within the CCG did not leave much room for CCG to challenge or co-create

research. This process can be improved in the future by allowing more time between

drafts/plans and CCG meetings and allocating more expert resource to the CCG.

g. Conclusions and next steps

The PR19 process to date presents a significant improvement compared to the PR14

experience but there are also opportunities for further improvement. It is clear that DCWW

has put much effort and resource into customer engagement since early 2016. The company

has also broadened their research scope compared to PR14. This expanded the breadth of

available data and methods, and provided welcome addition of qualitative information. It is

also very important to recognise that the company took initiatives for early engagement

with researchers, customers, and the CCG. However, given the volume and complexity of

designing and implementing a successful research programme, DCWW might benefit from

in-house research expertise for PR19 and beyond.

Although benchmarking is the regulator’s remit, the author feels that DCWW is likely in a

relatively favourable position in terms of their overall customer engagement activities.

32


This report only covers Phase 1 of the Willingness to Pay research, but it will be important

that the company provides reasoning on how it uses WtP to influence its decisions and that

the significant fluctuations are interpreted in the context of other evidence known to the

company. The report references the continued concerns over the nature of WtP

methodology despite the changes from the PR14 process. It will be important that the

company draws on the wide range of evidence gathered and not depend on this one metric

for key decisions in the business plan, its differences from the methodology followed during

PR14, and the absence of compelling points of triangulation for the different results of the

recent WtP exercise, the CCG does not feel confident that the existing WtP results should be

given considerable weight compared to the remaining empirical evidence gathered so far.

The CCG has now begun the process of working with the company to see how the collected

data will translate into practical results in the business plan. This is now the priority for the

CCG. This report, while highlighting areas for improvement and specific concerns, concludes

that the company has been successful in securing an understanding of the views of its

customer base from the Phase 1 research and triangulation process.

The author and CCG hope that the recommendations presented in this document will help

DCWW produce and submit a high quality, robust PR19 business plan.

References

Accent (2017). Water resources – quantitative research findings. Dŵr Cymru Welsh Water

internal document, August 2017.

Ariely, D., Loewenstein, G., and Prelec, D. (2003). ”Coherent Arbitrariness”: Stable Demand

Curves Without Stable Preferences." Quarterly Journal of Economics, 118(1): 73-106.

Arrow, K.J., Solow, R.S., Learner, E., Portney, P., Radner, R., and Schuman, H. (1993). Report

of the NOAA-Panel on Contingent Valuation. Federal Register, 58(10): 4601-4614.

Blue Marble (2017). 2017 Summer consultation report. Dŵr Cymru Welsh Water internal

document, 6th October 2017.

Consumer Council for Water (2017). Improving willingness-to-pay research in the water

sector. Final report (7/7/2017). Birmingham, Consumer Council for Water. Retrieved

10 March 2018 from https://www.ccwater.org.uk/wp-

content/uploads/2017/07/Improving-willingness-to-pay-research-in-the-water-

sector.pdf

Diamond, P.A., and Hausman, J.A. (1994). Contingent Valuation: Is Some Number Better

than No Number? Journal of Economic Perspectives, 8(4): 45-64.

https://www.ccwater.org.uk/wp-content/uploads/2017/07/Improving-willingness-to-pay-research-in-the-water-sector.pdf



33


Dŵr Cymru Welsh Water (2017a). PR19 customer engagement – phase 1: summary of

information sources and triangulation, final. Dŵr Cymru Welsh Water internal

document, 11 April 2017.

Dŵr Cymru Welsh Water (2017b). PR19 customer engagement – phase 1: companion to

main report - summary of evidence. Dŵr Cymru Welsh Water internal document.

Dŵr Cymru Welsh Water (2017c). PR19 customer engagement framework. Dŵr Cymru

Welsh Water internal document.

Farquhar J., Michels N. (2016) Triangulation Without Tears. In: Groza M., Ragland C. (eds)

Marketing Challenges in a Turbulent Business Environment. Developments in

Marketing Science: Proceedings of the Academy of Marketing Science. Springer,

Cham

Hahn, R., Metcalfe, R., and Rundhammer, F. (2017). Power to the people: a new trend in

regulation. (Working paper, June 2017).

Ofwat (2015). Water 2020: Regulatory framework for wholesale markets and then 2019

price review. Birmingham, Ofwat. Retrieved 10 July 2017 from

https://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ah

UKEwiY5vb6lfvYAhXiBcAKHcyPCrkQFggnMAA&url=https%3A%2F%2Fwww.ofwat.gov

.uk%2Fwp-content%2Fuploads%2F2015%2F12%2Fpap_con20150912water2020.pdf

&usg=AOvVaw1Dda7Uk36FnncdAnJus5h8

Ofwat (2016a). Ofwat’s customer engagement policy statement and expectations for PR19.

Birmingham, Ofwat. Retrieved 10 December 2017 from

https://www.ofwat.gov.uk/wp-

content/uploads/2015/12/pap_pos20160525w2020cust.pdf

Ofwat (2016b). Water 2020: our regulatory approach for water and wastewater services in

England and Wales. Birmingham, Ofwat. Retrieved 10 December 2017 from

https://www.ofwat.gov.uk/wp-

content/uploads/2015/12/pap_pos20150520w2020.pdf

Ofwat (2017a). Delivering water 2020: our final methodology for the 2019 price review.

Birmingham, Ofwat. Retrieved 10 January 2018 from https://www.ofwat.gov.uk/wp-

content/uploads/2017/12/Final-methodology-1.pdf

Ofwat (2017b). ‘Tapped-in’: from passive customer to active participant (22 March 2017).

Birmingham, Ofwat. Retrieved 10 November 2017

https://www.ofwat.gov.uk/publication/tapped-in-from-passive-customer-to-active-

participant/

Ofwat (2017c). Aide memoire information note for Customer Challenge Groups. (Summer

2017). Birmingham, Ofwat.

https://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ahUKEwiY5vb6lfvYAhXiBcAKHcyPCrkQFggnMAA&url=https%3A%2F%2Fwww.ofwat.gov.uk%2Fwp-content%2Fuploads%2F2015%2F12%2Fpap_con20150912water2020.pdf%20&usg=AOvVaw1Dda7Uk36FnncdAnJus5h8




https://www.ofwat.gov.uk/wp-content/uploads/2015/12/pap_pos20160525w2020cust.pdf

https://www.ofwat.gov.uk/wp-content/uploads/2015/12/pap_pos20160525w2020cust.pdf

https://www.ofwat.gov.uk/wp-content/uploads/2015/12/pap_pos20150520w2020.pdf

https://www.ofwat.gov.uk/wp-content/uploads/2015/12/pap_pos20150520w2020.pdf

https://www.ofwat.gov.uk/wp-content/uploads/2017/12/Final-methodology-1.pdf

https://www.ofwat.gov.uk/wp-content/uploads/2017/12/Final-methodology-1.pdf

https://www.ofwat.gov.uk/publication/tapped-in-from-passive-customer-to-active-participant/

https://www.ofwat.gov.uk/publication/tapped-in-from-passive-customer-to-active-participant/

34


The Customer Challenge Group for Dŵr Cymru Welsh Water (2013). Report on the

Company’s Business Plan for the 2014 Price Review. (02 December, 2013).

The Customer Challenge Group for Dŵr Cymru Welsh Water (2016). Minutes of meeting

held on 14th July 2016. Dŵr Cymru Welsh Water internal document, 14 July 2016.

Willis, K., (2017a). PR19 Willingness to pay research by Accent & PJM economics, January

2017 (review). Dŵr Cymru Welsh Water internal document, 22 January 2017.

Willis, K., (2017b). PR19 Willingness to pay research by Accent & PJM economics, August

2017 (review). Dŵr Cymru Welsh Water internal document, 22 August 2017.

dŵr cymru welsh water pr19 customer engagement programme · 2019-07-25 · 3 assurance report for...

Documents