evaluation of market research - democracy.leeds.gov.uk 5.pdf · it is normal practice in market...

Evaluation of Market Research Analysis of Leeds City Council Citizens’ Panel Survey

4th November 2013

Prepared for Woods Whur LLP

By Elke Neuteboom (Research Director) and Martin Wootton (Research Director)

This work has been conducted in accordance with ISO 20252, the international standard for market and social research

Table of Contents 1. Introduction .................................................................................................................................. 1

1.1 About RS Consulting ............................................................................................................ 1

1.2 About the authors ............................................................................................................... 1

2. Examination of the survey methodology ...................................................................................... 2

2.1 Appropriateness of the survey method .............................................................................. 2

2.2 Representativity of the sample ........................................................................................... 2

3. Examination of the questionnaire ................................................................................................ 4

4. Examination of the analysis and the conclusions drawn by the report ....................................... 6

5. Conclusions from our analysis ...................................................................................................... 7

6. Evaluation of “Lap Dancing Clubs in Leeds”: An Elliott Marketing and PR Market Research Report ........................................................................................................................................... 8

6.1 Appropriateness of the survey method .............................................................................. 8

6.2 Representativity of the sample ........................................................................................... 8

6.3 Evaluation of the questionnaire and the analysis of the data ............................................ 8

6.4 Conclusions .......................................................................................................................... 9

Appendix 1: Comparison of responses from demographic questions in the Citizens’ Panel survey data and the demographic distribution indicated by the 2011 Census. ............... 10

1

\\SLSFS01\WoodsWhur$\docs\B\BEA003\3\001348GENOT-CRG2013111312482835.docx

Date Last Edited: 13 November, 2013 Checked By: Date Checked:

1. Introduction

This document is an independent, impartial and professional evaluation of market research undertaken by Leeds City Council dated 25th June 2013, and has been commissioned by Woods Whur LLP.

Two sources of data have been analysed for this evaluation:

1. A report entitled “Report of Head of Licensing and Registration – Report to Licensing Committee” dated 25th June 2013 on the subject of “Sex Establishment Licensing Policy”, referencing findings from a survey of 1,847 Citizens Panel respondents undertaken in January 2013;

2. A file containing frequency tables, showing the questions asked and the frequency counts of the number and percentages of people giving each response.

Our evaluation comprises chapters 2-5 of this document.

Separately, we have also evaluated a report entitled “Lap Dancing Clubs in Leeds” prepared for Elliott Marketing and PR, conducted by the market research agency McCallum Layton. Our evaluation forms chapter 6.

1.1 About RS Consulting

RS Consulting is a market research consultancy based in London. RS Consulting specialises in undertaking and reviewing qualitative and quantitative market research on a range of topics, for a wide variety of clients, ranging from UK government departments to international technology brands. It has been trading since 1984 and employs 45 people. It is fully accredited to the UK Market Research Society and to ISO 20252, the international standard for market research services.

1.2 About the authors

Elke Neuteboom, Research Director, RS Consulting

Elke has 19 years’ experience in international market research. She joined RS Consulting in 2007 and specialises in quantitative research. Prior to RS Consulting, Elke was Head of Research for the UK division of a management consultancy specialising in customer and employee loyalty research and research within the leisure sector. She has authored a number of independent reports commissioned by commercial clients, which have been presented as evidence in hearings across Europe.

Martin Wootton, Research Director, RS Consulting

Martin has 15 years’ experience in market research. He joined RS Consulting in 2000, having previously worked in the marketing intelligence team of a French telecommunications company. Martin specialises in in-depth quantitative research, and has authored several independent research reports on behalf of clients in the technology and business services sector. Martin is a regular speaker at industry conferences and events, and is an examiner for the Market Research Society’s qualifications programme.

2



2. Examination of the survey methodology

The report states that 1,847 responses were analysed. The survey was sent out to members of Leeds City Council’s Citizens’ Panel via post with the option to complete it on-line. The survey was issued in January 2013, with 81% of respondents completing the survey on-line.

The purpose of the survey was to canvass a representative and robust sample of residents of Leeds, to gauge their thoughts and opinions on the numbers and localities of Sexual Entertainment Venues (SEVs) within Leeds.

2.1 Appropriateness of the survey method

The method used to survey opinion – a quantitative questionnaire - sent as a paper copy by post and also made available online - is appropriate to the objective of the research. However, there are limitations inherent in such methods which can introduce significant bias into the sampling. By their very nature, postal and on-line surveys are self-selecting; they offer little control over which specific target respondents choose to take part. In our professional experience, survey data collected on-line tends to over-represent people who are particularly engaged with the subject matter, both positively and negatively. It therefore tends to under-represent those with more neutral views.

2.2 Representativity of the sample

The Citizens’ Panel is defined in the report as “a group of [4,000] residents of Leeds who have volunteered to give up their time to completed between 4 and 6 surveys a year about a range of issues.” This suggests the panel is self-selecting to some extent, and therefore also likely to over-represent the views of those who feel strongly about the issues being discussed.

The sample size of 1,847 respondents is statistically robust, and represents 46% of the total panel. This is a notably lower response rate than normal – the report states that the “response rate is usually around 60%”, suggesting that this survey was less engaging than other surveys panellists typically receive.

We have compared the demographic profiles of the sample of 1,847 respondents with the 2011 Census data. Specifically, we have compared the distribution of age range, sex, ethnicity, family status and employment status with Census data available for Leeds. We have compared the distribution of disability and sexual orientation with UK-wide data from the Office for National Statistics, since this is not readily available for Leeds alone.

This comparison shows that there are major differences between the demographic composition of the Citizens’ Panel survey data and the demographic composition from the 2011 Census. Specifically, compared with the 2011 Census data, the survey:

Over-represents those aged between 45 and 75 and under-represents under-45s;

Over-represents White British respondents and under-represents other ethnicities;

Over-represents households with two adults and no children living at that address;

Over-represents those who are retired or in full-time employment and under-represents students, part-time workers and homemakers.

A table showing line-by-line comparison is shown in Appendix 1.

3



Data in the survey on the individual areas of Leeds where respondents are resident appears to be broadly evenly distributed, though we note that 59% of responses contain no indication of the area of Leeds in which the respondent resides. This means we cannot conclude whether the survey data is representative of Leeds as a whole.

4



3. Examination of the questionnaire

The actual questionnaire used has not been supplied for us. We are therefore using the data tabulations supplied as a proxy. The data tabulations set out the questions and response options asked, but do not indicate the sequence, visual layout or exclusions used.

We have examined the number, type and wording of the questions, and the frequency counts showing the number of responses to each question option. Overall, there appear to be 51 questions, all with fixed response options. While a relatively long survey, it is not untypical of the length of consumer market research surveys.

For each question, at least 100 of these respondents do not appear to give valid responses – it is assumed that these are respondents preferring not to answer, or who might be selecting a “don’t know” or “not applicable” option, which is not mentioned in the data tabulations. Note that section B14 of the UK Market Research Society Code of Conduct1 specifically states that “Members must take reasonable steps to ensure…that Respondents are able to provide information in a way that reflects the view they want to express, including don’t know/prefer not to say where appropriate.” It is unclear from the data tabulations what mechanisms were in place to allow respondents to choose not to express a view.

The same Section B14 also explicitly states that reasonable steps must be taken to ensure “that Respondents are not led towards a particular point of view” and “that responses are capable of being interpreted in an unambiguous way.” Many of the agreement scale questions, where respondents are asked to indicate the extent to which they agree or disagree with a statement, might be seen as somewhat leading, since they are phrased negatively.

Example:

It would not be acceptable to locate a lap dancing club near to a Family leisure facilities [sic] such as cinemas, theatres and concert halls

Strongly agree

Agree

Neither agree nor disagree

Disagree

Strongly disagree

Accepted best practice is to avoid the use of negative phrasing, since this can both (a) be interpreted as leading the respondent, and (b) lead to double negatives and therefore confusion among some respondents as to how to express their view. In other words, to convey the view that “it is acceptable to have a lap dancing club...” the respondent is required to select a negative response, to cancel out the negative sentiment in the phrase being tested.

1 https://www.mrs.org.uk/pdf/Code%20of%20Conduct%20(2012%20rebrand).pdf

https://www.mrs.org.uk/pdf/Code%20of%20Conduct%20(2012%20rebrand).pdf

5



The repetitive nature of the questioning – there are no less than 35 statements in the form “it would not be acceptable to…”, each with the same choice of responses as in our example above – seriously risks introducing respondent fatigue. Respondent fatigue is the term used to describe the drop in data quality that occurs when respondents lose concentration or motivation during a survey. It is symptomatic of lengthy surveys, especially those with repetitive sequences of questions or complex wording or subject matter. Respondent fatigue is characterised by an increasing number of ‘don’t know’, ‘prefer not to say’ or blank responses as the survey progresses, and an increase in the time taken to answer each question (in the case of on-line surveys).

There is no evidence that the survey designer has randomized the sequence of statements being tested – something that is commonly considered best practice in questionnaire design to avoid respondent fatigue. Nor does it appear that the scale has been varied, another common method of ensuring respondents remain engaged. However, on the basis of the tabulations alone, there is no evidence that respondent fatigue has definitely occurred.

6



4. Examination of the analysis and the conclusions drawn by the report

The question-by-question analysis presented in the report appears to be correctly conducted and accurately represents the data tabulations.

The report does not give details of any data processing applied to the data. It is normal practice in market research analysis – and particularly where on-line surveys have been used – to check each respondent’s answers to ensure they are valid and sensible. In a typical on-line survey, at least 10% of all responses are identified as being inappropriate, and they are typically discarded or replaced. Responses are most commonly deemed inappropriate if they meet one or more of the following criteria:

Contain plainly nonsensical open-ended responses (e.g. “qwertyuiop”);

Are measured to have been completed so quickly that it is not feasible that the respondent could have read and understood the question properly;

Contain contradictory data – i.e. a respondent whose responses do not appear logical (i.e. someone who marks their age as “under 18”, yet states their employment status as “retired”);

Contain ‘flat-lining’ data, i.e. a respondent who consistently gives the same answer to every scale question, when such patterns would not be logical, for instance where scale data suggests the respondent has extreme views but responses to open questions openly contradict this.

Therefore, it is not known whether data checking procedures have been applied to the data.

7



5. Conclusions from our analysis

While the analysis of the data appears to have been conducted fairly and reasonably, we have nonetheless identified a number of areas that might lead to the data being inadvertently mis-interpreted.

Two areas are cause for concern:

The data is not representative of the demographic distribution of the population of Leeds;

Wording of agreement scale questions has the potential to introduce double negatives and therefore has the potential to misrepresent respondents’ true views.

Three other areas merit further examination:

No evidence of measures in place to prevent respondent fatigue, despite the length and repetitive nature of the questionnaire;

No evidence of data cleaning and checking procedures;

For the majority of questions, 10% of responses or more are missing from the analysis; presumably these represent blanks or respondents choosing not to answer from the fixed list of responses.

The conclusions drawn in the report’s analysis are a reasonable assessment of the data at face value. However, the data is clearly not representative of the Leeds population as a whole. Furthermore, the design and quantity of the agreement scale questions is such that some respondents risk mis-interpreting the questions. There is no evidence that the analysis takes into account any data weighting or bias correction mechanisms, and therefore cannot be said to accurately reflect the views of all Leeds residents.

8



6. Evaluation of “Lap Dancing Clubs in Leeds”: An Elliott Marketing and PR Market Research Report

The report describes the findings from a survey of 100 people conducted between 3rd and 8th May 2012, using street intercepts in and around The Headrow, Leeds.

6.1 Appropriateness of the survey method

The survey intercepted adults aged 21 and over who are either residents of Leeds city centre, or visit the city centre at least once every two months. Intercepts were carried out on one street only, in close proximity to the Wildcats lap dancing venue. While the survey was not designed to be carried out in multiple locations, due to the relatively small sample size and the need to run for several days, it would have been preferable to intercept respondents at multiple city centre locations, in order to increase the representativity of the sample.

The street intercept method itself is commonly used in consumer market research and is well suited to sampling a wide variety of people within a very specific location. The survey has been run over several days to prevent introducing bias by surveying only on weekdays or weekends.

A sample size of 100 is relatively small but large enough for findings at the overall level to be reported with a good degree of statistical stability.

6.2 Representativity of the sample

The report outlines the age and gender mix in the survey sample and this can be compared directly to the 2011 Census data for Leeds. This is shown in the table below:

This comparison shows that the survey sample is broadly representative between male and female respondents. In terms of age ranges, it appropriately represents those aged between 30 and 59. However, it does over-represent under-30s and under-represent those aged 60 and over.

It is important to note, however, that this sample also includes visitors to Leeds, so is unlikely to be exclusively a survey of Leeds residents.

6.3 Evaluation of the questionnaire and the analysis of the data

The questionnaire comprises 15 questions, including 5 open-ended questions to capture verbatim responses from respondents, and thus gauge the strength of emotion associated with responses.

There are two scale questions, framed simply and neutrally, with an appropriate balance of positive and negative response options.

The questionnaire is relatively short in length with little repetition. Show cards have also been used to help illustrate the question responses to respondents. This means there is little likelihood of respondent fatigue.

Sample profile Census 2011 (see notes) Notes

Gender Female 54% 51% Source: Leeds - The Big Picture. A summary of the results of the 2011 Census

Male 46% 49% http://www.scribd.com/doc/121792123/Leeds-Census-Big-Picture

Age Range Under 30 35% 24.9% Lowest ages: Census (16), Survey (21)

30-59 49% 50.7% Source: Leeds - The Big Picture. A summary of the results of the 2011 Census

60 and over 16% 24.4% http://www.scribd.com/doc/121792123/Leeds-Census-Big-Picture

Demographic Profile

9



6.4 Conclusions

Although the data set is smaller, and is not fully representative of the population of Leeds (and bearing in mind its goal was to survey both residents and visitors), it is a much clearer, shorter and unambiguous survey in contrast to the Citizens’ Panel research.

If possible (depending on sub-group sizes), data weighting techniques should be applied in order to adjust the data to make it more representative of the Leeds population, and the results restated taking into account the weighted data.

10



Appendix 1: Comparison of responses from demographic questions in the Citizens’ Panel survey data and the demographic distribution indicated by the 2011 Census.

Citizens Panel Results Census 2011 (see notes) Notes

Gender Female 49.9% 51% Source: Leeds - The Big Picture. A summary of the results of the 2011 Census

Male 50.1% 49% http://www.scribd.com/doc/121792123/Leeds-Census-Big-Picture

Age Range Under 45 30.9% 53.9% Lowest ages: Census (16), Citizens Panel (18)

45 - 59 32.1% 21.7% Source: Leeds - The Big Picture. A summary of the results of the 2011 Census

60 - 74 31.3% 15.7% http://www.scribd.com/doc/121792123/Leeds-Census-Big-Picture

75 and over 5.6% 8.7%

Ethnicity White British 89.8% 81.1% Source: Leeds - The Big Picture. A summary of the results of the 2011 Census

White Irish 1.6% 0.9% http://www.scribd.com/doc/121792123/Leeds-Census-Big-Picture

White Other 1.9% 2.9%

Asian Indian 0.9% 2.1%

Asian Bangladeshi 0.1% 0.6%

Asian Pakistani 0.3% 3%

Asian Chinese 0.2% 0.8%

Asian Other 0.2% 1.2% Includes groups (Asian Kashmiri) and (Asian Other) from Citizens Panel

Black African 0.8% 2%

Black Caribbean 0.4% 0.9%

Black Other 0.1% 0.6%

Arab 0.1% 0.5%

Mixed White and Asian 0.5% 0.7%

Mixed White and Black African 0.1% 0.3%

Mixed White and Black Caribbean 0.1% 1.2%

Mixed Other 0.4% 0.5%

Any Other Ethnic Group 1% 0.6%

Disability No 83.1% 82%

Yes 16.9% 18%

Sexual Orientation Bisexual 1% 0.4%

Gay/Lesbian 1.2% 1.1%

Heterosexual 89.6% 93.9%

Family Status 1 Adult, 0 Children 26.4% 31.6% Source: Leeds - The Big Picture. A summary of the results of the 2011 Census

1 Adult, 1+ Children 3.7% 9.8% http://www.scribd.com/doc/121792123/Leeds-Census-Big-Picture

2 Adults, 0 Children 40.5% 11.8%

2 Adults, 1+ Children 20.5% 21.5%

Other 8.9% 25.2%

Employment Status Employed Full-time 46.7% 31.6% Source: Leeds - The Big Picture. A summary of the results of the 2011 Census

Employed Part-time 9% 20.1% http://www.scribd.com/doc/121792123/Leeds-Census-Big-Picture

Looking after home/ family 2.5% 6.4%

Other 1.9% 2.3%

Retired 27.6% 14.1%

Self Employed 5.9% 4.1%

Student/ Full-time Education 0.8% 14.1%

Unable to Work 2.6% 3.8%

Unemployed and Seeking Work 2.9% 3.5%

Demographic Profile

Source: ONS - UK Census data from http://www.ons.gov.uk/ons/rel/census/2011-

census/key-statistics-and-quick-statistics-for-local-authorities-in-the-united-

kingdom---part-1/stb-key-statistics-for-local-authorities-in-the-uk.html

Source: ONS - UK Census data from http://www.ons.gov.uk/ons/rel/integrated-

household-survey/integrated-household-survey/april-2011-to-march-2012/stb-

integrated-household-survey-april-2011-to-march-2012.html#tab-Sexual-identity

Includes groups (All Pensioners, Family), (Cohabiting Couple Households) and (Other

Household Types) from Census

evaluation of market research - democracy.leeds.gov.uk 5.pdf · it is normal practice in market...

Documents