surveys for communicators
DESCRIPTION
A one day workshop on surveys for communicators. Increasingly communicators need the ability to evaluate their activities and know what their audiences think and desire. Being able to design and set-up online surveys is a key tool for communicators for soliciting feedback and interacting with audiences. These slides from the workshop will take participants from the design to the analysis stage. Workshop originally conducted on 14 June 2013 for the Geneva Communicators Network.TRANSCRIPT
Surveys for communicators
Glenn O’Neil
(these slides used for originally for
workshop held on 14 June 2013 for
Geneva Communicators Network)
3
1. Introduction
2. Planning
3. Creation
4. Management
5. Analysis
Schedule
4
Why a survey?
surveys are used to collect information from or about people … …to describe, compare or explain their knowledge, feelings, values and behaviours
But they are not the only research method!
1. Introduction
5
Other research methods:
– Desk studies
– Interviews
– Focus groups
– Content analysis
– Case studies
– Records/statistics
1. Introduction
6
Recommended: combination of qualitative
and quantitative methods
1. Introduction
7
Advantages of surveys:
– Inexpensive
– Obtains current information
– Can obtain a great deal of information at one
time
– Provide quantitative data
1. Introduction
8
Disadvantages of surveys:
– People don't always tell the truth
– People make mistakes
– Representative samples are difficult to obtain
– People refuse to participate
– Writing questions is tricky
1. Introduction
9
1. Introduction
In communications - surveys are used for:
– Assessing opinions/attitudes/behaviours of
audiences
– Measuring reputation of organisations
– Assessing staff satisfaction
– Evaluating communication products,
campaigns, projects or programmes
What other uses?
10
All surveys follow a similar timeline – or
should!
2. Planning
11
2. Planning - timeline
Review
background
information
Survey
design
& objectives
Prepare
survey
& final report
outline
Draft questions
& other
components
Place
questions
in appropriate
order
Test
survey
Revise
survey
following test
Launch &
Promote
survey
Monitor
responses
& send
reminders
as needed
Collect
responses
Analyse data
Draft report
Present report
Step 1: Planning Step 2: Creation Step 3: Management Step 4: Analysis
12
Setting objectives (defining the project): – What is the time period – when are the results
required?
– What is the audience of interest (and are there sub-
groups)?
– What information is needed?
– How will the information be collected?
– What form should the results take? How are the
results going to be used?
– How accurate should the results be?
– Is this project going to be repeated?
2. Planning
13
Moving from concepts to questions: – Operationalisation – translating abstract concepts into
something that can be observed and measured:
Concepts -> Variables ->Questions
3. Creation
14
Example: Moving from concepts to questions:
3. Creation
Assess success
of
campaign
Reach
Understanding
Interaction
Adhesion
Concept -> Variables -> Questions
Do you recall having seen campaign xy?
Which of the following is the best
description of what you recall of
the campaign?
Etc.
Etc.
15
3. Creation
– Survey outline can be a guide in creating
structure of final report (think about it at this
stage!)
– This helps to avoid “survey creep”
16
3. Creation
What are the variables we are often trying to
measure in communications?
– Awareness / Knowledge
– Understanding
– Opinion / attitude
– Behavior
17
3. Creation
Writing questions
– Some questions are easy to create (such as
gender, length of employment, etc.)
– But questions that ask for information or
measure attitudes or behaviour are more
difficult to create
18
3. Creation
Writing questions
- Different question types are suited to different
types of information sought, for example:
- Likert scale is suited to measuring attitudes
- A yes/no question is suited to measuring if a
precise action has been taken
19
3. Creation Types of questions
- Open questions
- Closed questions:
- Likert scale
- Rank order scale
- Multiple choice (singular response)
- Multiple choice (plural response)
- Binary (e.g. yes/no)
- Numerical response
20
3. Creation Types of questions: Open questions
- On all surveys its good to have at least one
open-ended question
- Open-ended questions are often useful for
probing the “why” and the “how”
- Limit the number of open-ended questions
you include – as analysis of results takes
time!
21
3. Creation Are there areas of weakness in your communications
programme that are hindering your performance?
Please list problems that have arisen in the management of your programme:
Please provide details of projects in your organisation:
Do you have comments or feedback?
22
3. Creation
In your view, what are the two greatest weaknesses of your communications programme?
In the past 12 months, what have been the main challenges in the management of your programme
Please provide short descriptions (1 paragraph maximum) of major projects currently underway in your organisation:
This questionnaire has been about campaign management. Do you have any further comments on this subject?
23
3. Creation
Questions: Likert scale
Most commonly used scale to measure
attitudes and opinions: “The most important aspect of internal communications is accuracy”
o Strongly agree
o Agree
o Neither agree nor disagree
o Disagree
o Strongly disagree
24
3. Creation Questions: Likert scale – Create points on a scale that cover all dimensions
– Consider using scales already developed and used in
other questionnaires
– Scales should have maximum seven points
– Labels (“good”, “satisfied”) need to be as clear as
possible
– The use of a middle (neutral) point is debatable
– Begin with negative points (left) and move to positive
points (right)
25
3. Creation How would you rate your agency’s website?
oPoor oGood oVery good oExcellent
How often do you use your mobile phone? oNever oHardly ever oSometimes, oAlways
The current mobility policy is suitable for me oStrongly disagree oDisagree oNeutral oAgree oStrongly agree
26
3. Creation
How would you rate your agency’s website? oTerrible oPoor oGood oExcellent
In an average working day, how often do you use your
mobile phone? o0 times ounder 5 times o6-10 times o11-15 times o16-20 times, omore than 20 times
The current mobility policy is suitable for me
oStrongly disagree oDisagree oAgree oStrongly agree No opinion
27
3. Creation
Likert scale & attitudes
– Questions often go beyond asking about an
attitude as a whole
– But ask about concrete instances of an
attitude in a series of questions
Example: job satisfaction (see over)
28
3. Creation “Are you satisfied with your job?”
o Yes o No o Don’t know
“Please indicate your level of satisfaction with the
following aspects of your job”:
Very
dissatisfied
Dissatisfied Neither
satisfied nor
dissatisfied
Satisfied Very satisfied
current salary o o o o o
current benefits
o o o o o
relations with
colleagues o o o o o
relations with
supervisors o o o o o
29
3. Creation
Rank order scale questions
– These questions ask respondents to rank or
state their priority on items
– Useful for learning of what is of value for
respondents and what are their priorities
30
3. Creation Which of the following features would you like for your
organisation’s website? e-newsletter
[] recruitment tool
[] flash modules
[] video modules
[] page print function
Out of the following features, please select the top three
you would like for your organisation’s website: (select number 1 for this highest and number 3 for the lowest)
[0]e-newsletter
[0] recruitment tool
[0] flash modules
[0] video modules
[0] page print function
31
3. Creation
Multiple choice (singular) questions
– These questions ask respondents to choose
one item on a list
– Are used to check knowledge, collect
information & demographic data, assess
priorities
– In online questionnaires, long lists are
typically put in a pull-down menu (e.g. “what is
your country of residence?”)
32
3. Creation What is your current work?
o Administrative
o Management
o Staff
o Technical
Which of the following best describes you current work? o Senior management (director level)
o Professional staff
o Support staff
o Other (please specify __________)
33
3. Creation
Multiple choice (plural)questions
– These questions ask respondents to choose
multiple items on a list
– Useful in providing a prepared category list for
respondents
– Are used to check knowledge, collect
information & demographic data
34
3. Creation What is your prior major field of working experience?
o Political
o Diplomatic
o Military
o Development
o Humanitarian
What is your prior major field of working experience?
[] Political
[] Diplomatic
[] Military
[] Development
[] Humanitarian
[] Other, (please specify_________)
35
3. Creation
Binary (yes/no)questions
– These questions ask respondents to choose
one of two responses
– Should have only one thought / action in
mind. e.g. « Have you graduated from
university? »
– Often misused where Likert scale should be
used (as yes/no may not represent all
dimensions of a response)
36
3. Creation Should the UN cut its communications and fundraising
budget?
o Yes
o No
Should the UN substantially reduce its communications
budget?
o Yes
o No
(consider adding “don’t know”)
37
3. Creation
Numerical questions
– These questions ask numerical data from
respondents
– Often used to collect budget and demographic
data
– Must be carefully thought out as they often
are sensitive issues
38
3. Creation How may children do you have?
O1-3 O3-6 O6-10
What is your age?
[18] – pull down list with numbers 0-100
What was your annual income last year? $_________
39
3. Creation
How may children do you have? O None O1 O2 O3 O4 O5 O6 O7 O8 O9 O10 O More than 10
What is your age?
ounder 18 o19- 30 o31-40 o41-50 o51-60 o61-70 oover 70
(or use “what year were you born in?)
In which category does your annual income of last year
fit best? oBelow $40,000 oBetween $40,001 & $50,000 oBetween $50,001 & $60,000
oBetween $70,001 & $80,000 oAbove $80,000
40
3. Creation Formulating questions – Hints – Only ask for one piece of information per question
– Avoid overly complex words/sentences
– Avoid use of undefined terms, e.g. “year”, “technology”, “management”
– Be consistent in the use of words and terms
– Ensure questions are answerable
– Avoid biased questions, e.g. “Do you still have confidence in your top management?”
41
3. Creation Formulating questions – Hints – Avoid unreasonable recall period
e.g. “Please provide budget details for the past five
years”
– Make questions as specific and concrete as possible
– Don’t ask questions which require information or a
level of skill which respondents may not have
e.g. “What is the percentage increase in your IT
budget from 2007 to 2008?”.
42
3. Creation Formulating questions – Hints – Avoid response options not matching the question
– Avoid response options that are not overlapping or
mix “levels”
– Avoid response options that are not exhaustive
43
3. Creation Survey components Aside from the questions other components must be
prepared:
– Instructions
– Transition texts (if required)
– Thank you text
– Invitation text
44
3. Creation Survey components - instructions – Instructions appear on the first page of the survey
– Instructions duplicate some information from the
invitation text as invitation letter/email may get
separated from the survey
– Instructions should specify if any special preparation
or documents are required before starting the survey
45
3. Creation Survey components - instructions Example (short form):
This questionnaire is part of a review of the communications unit
being conducted by the Evaluation Department. Your feedback
on this issue would be highly appreciated. This questionnaire will
take some 10 minutes to complete. All feedback provided is
anonymous and will be treated confidentially. Please complete
the survey by 1 October 2013. If you have any comments or
questions, please contact Ms Marie Rose ([email protected])
46
3. Creation Survey components – transition text – Transition texts are used in longer questionnaires where
some explanation is required for respondents when
moving from subject to subject:
Example:
The next ten questions will ask if you agree or disagree with
different UN Human Resources policies
47
3. Creation Survey components – thank you text – Thank you text appears at the end of the questionnaire.
In online surveys, it appears on a final page.
Example:
Thank you for your feedback and comments on the ABC
management structure. If you have any further comments or
questions on this project, please contact Ms Marie Rose
48
3. Creation Survey components – invitation text – Invitation texts are important as they play a key role in
determining if respondents complete the survey
– Invitation texts need to include the following information:
• Explain why you want an input
• Tell respondents who you are
• Tell people how long it will take
• Reassure respondents about their privacy
• Keep it short
For online surveys:
• Take care with “from”, “to” and “subject” fields
• Make sure the link works
49
3. Creation
Survey components – invitation texts Example (online survey): From: [email protected]
Subject: Communication Unit view - Seeking your feedback
Dear colleague,
The Evaluation Department is currently undertaking a review of the Communication Unit.
We would very much appreciate your feedback on this subject by completing a brief online
survey. This survey will take some 10 minutes to complete. All replies are anonymous and will be
treated confidentially. Please complete the survey by 1 October 2013.
To complete the survey, please click here >>
If this link does not work, please copy and paste the following link into your internet window:
http://www.surveymonkey.com/survey123
Thank you in advance; your feedback is very valuable to us.
Kind regards,
Marie Rose
Research Officer
Evaluation Department
email: [email protected]
tel: ++ 41 22 917 000
****
50
3. Creation Putting the survey together – Start with easy to answer questions
– Broad to the narrow
– Non-sensitive to the sensitive
– Demographic questions always last
– Group subjects together
– Questions should be in a logical order
– Ensure routing (branching) exists to skip not relevant
questions
– Avoid “response set”
51
3. Creation Putting the survey together Which question should come first?
1) How important or not is the intranet to your work?
2) What are the important tools you use daily for your
work?
52
3. Creation Putting the survey together Avoiding “response set”
Staffing policies are fair and applied equally
oStrongly disagree oDisagree oAgree oStrongly agree
People are treated differently under Staffing policies
oStrongly disagree oDisagree oAgree oStrongly agree
53
4. Management Getting ready to send the survey… – Once you have created the questions and placed
them in an appropriate order you are now ready to
send it out…
– But wait! Test it first!
– You may not have access to the intended
respondents but test the survey at least with
colleagues
– For online surveys, important to test if the survey
functions as planned
54
4. Management Testing survey - Issues – How long did it take to complete it?
– Are the questions understandable?
– Did the survey flow comfortably?
– Do the questions obtain the information needed?
– Are new questions needed?
– Are their questions that can be removed?
– Is the “don’t know” / “no opinion” being used a lot?
– Should we try different methods of data collection?
55
4. Management Print verses online surveys Print (respondents receive a word document, and
respond by completing it and posting/emailing back)
• Respondents can answer questions in order they like
• Respondents can see length of survey
• Typically has low response rates
Online (respondents receive a link by email)
• In most cases, respondents must complete survey in one
sitting
• Response rates can be monitored in real time
• Issues of data confidentiality
56
4. Management Print verses online survey – The choice depends also on the context
– For small groups that can be physically handed the
survey consider print
– For large groups dispersed globally consider online
Other options: telephone (CATI) surveys,
face-to-face surveys, mailed surveys
57
4. Management Launching & promoting surveys Once your survey has been tested and you have
made the appropriate adjustments, you are now
ready to launch it. Have you done the following:
• Reviewed the timeline of the project?
• Identified the correct target audience & arranged access to
them?
• Ensured that all relevant stakeholders are briefed on the
project?
• Planned some time for follow-up and monitoring?
• For an online survey, emptied the database of test results?
58
4. Management Launching & promoting surveys
The question:
How can we get the necessary number of people to
complete the survey in order to make the project valid
and credible?
59
4. Management Launching & promoting surveys Some advice to increase responses:
• Ensure your survey is short, logical and answerable
• Explain incentive for completing survey
(e.g. access to results)
• Ensure that you have the “buy-in” from key stakeholders
• Ensure that your invitation covers all points mentioned above
• Give people sufficient time to respond (minimum 3 weeks)
• Spend some time encouraging key relays with audiences
• Plan to send out at least one reminder – but no more than
three
60
4. Management Launching & promoting surveys During the period that respondents are
completing the survey:
• Check regularly the number of responses received
• Ensure that all sub-audiences are represented
• Send out reminders based on response rate
• Be available to respond to any queries
• Correct any spelling / numbering errors but do not
fundamentally change the survey once launched
61
4. Management A word about response rates & sampling – In communications, we often are surveying precise
groups (e.g. staff, stakeholders, clients)
– With these groups (often small in number) the
maximum amount of people are desired to complete a
survey
– Although not 100% applicable, sampling theory can
offer some guidance
But how many responses will make a survey
credible?
62
4. Management A word about response rates & sampling – When dealing with groups under 100 try and get as
many responses as possible
– Divide your audiences into sub-audiences and ensure
results are representative of all sub-audiences
– Small response rate will be more credible if
complemented by other research (e.g. interviews)
63
5. Analysis Basics of analysing survey data – What do the survey results show?
– Looking for totals, averages, trends, patterns,
differences, relationships and changes
– Different type of analysis required for quantitative
(closed questions) and qualitative (open questions)
data
– Both qualitative and quantitative results provide
evidence to support conclusions and recommendations
64
5. Analysis Basics of analysing survey data Two main types of analysis methods for quantitative data:
– Descriptive statistics: most commonly used, including
counts, proportions (%), measures of central tendencies
(mean/mode/median) and variation (range/standard
deviation), cross tabulation (comparison between
different audiences)
– Correlations & regressions: used to show
relationships and predict outcomes and future
relationships
In communication evaluation, descriptive
statistics are usually sufficient
65
5. Analysis Basics of analysing survey data Descriptive statistics can be displayed in different ways
Comparison
Ranking/priorities
Proportions
66
5. Analysis Basics of analysing survey data Descriptive statistics can be displayed in different ways
Trends / changes
Relationships
67
5. Analysis Basics of analysing survey data Tallies/frequency counts: how many people fit into a
category – often displayed as percentage of a total:
Shows only
percentage
Better to show
percentage and
actual number
68
5. Analysis Basics of analysing survey data Cross tabulation: Basic data split to show which
demographic groups make up categories:
Interested
to see who
said “No”!
Graph shows
from which
categories are
the “No”
69
5. Analysis Basics of analysing survey data Cross tabulation
70
5. Analysis Basics of analysing survey data Putting responses together: Certain question responses
(but not all) can be put together to provide a “total”
score, e.g. “staff satisfaction”
Responses
to these
question
can be
added up
Very
dissatisfied
Dissatisfied Neither
satisfied nor
dissatisfied
Satisfied Very
satisfied
current salary 4 1 2 2 1
current benefits
3 4 1 1 1
relations with
colleagues
3 2 3 1 1
relations with
supervisors
1 1 2 3 3
Overall
satisfaction
score
(11) 27.5 % (8) 20 % (8) 20% (7) 17.5% (6) 15% Summated
total scale
generated
71
5. Analysis Basics of analysing questionnaire data Hints on evaluating quantitative data
– Simple counts, proportions and comparisons are sufficient “evidence”
for most communication evaluation
– Numbers and percentages must be reported in context, e.g:
-90% satisfaction is impressive but less so when it’s 9 out of 10 people.
-1,000 people say “yes” is impressive but less so when we know total
population is 10,000.
– Be cautious in making conclusions based on small samples
– Qualitative data stronger when combined with quantitative data
– More sophisticated statistical analysis can be made to check
correlation, relationships and statistical significance
(chi square , t-test)
72
5. Analysis Basics of analysing survey data Evaluating qualitative data
– For survey, qualitative data is collected through open-ended questions
– Often such data is consulted but not analysed
– Data should be analysed using simplified content analysis
73
5. Analysis Basics of analysing survey data Steps for evaluating qualitative data
– 1. Read through the responses.
– 2. Create response categories.
– 3. Label each comment with one or several categories.
– 4. Look at what you have.
– 5. Think what are the responses about?
– 6. Identifying the patterns and trends.
74
5. Analysis Reporting and using results
– Now you have made some analysis of the data
you are ready to write a report
– How can you turn the data into findings and
ultimately recommendations?
75
5. Analysis Reporting and using results
Hints on going from data to a report
– Return to the original research question asked – how did
you split this up into variables (or topics) and then into
questions?
– What does the quantitative and qualitative data show you
about each topic? What are the conclusions you can
reach?
– You need to show a series of “evidence” that support your
conclusions – showing solid “proof” is rarely possible!
76
5. Analysis Reporting and using results
Best practices for report writing
– Quantitative data is best represented in graphs – but for
data with small number of categories (e.g. yes/no) it can be
entered into the text directly
– Direct quotes from interviews and open questions can be
powerful testimonies in a report
– Try and avoid just reporting survey results “as is” – you
need to interpret it for readers
77
6. Conclusion – Some say that survey writing is more of an art than a
science
– Always look towards existing, well-established surveys for
examples of question types and scales used
– For further information and resources, consult my blog
(search for “surveys”):
http://intelligentmeasurement.wordpress.com
78
Contact details
glennoneil
www.owlre.com
@glenn_oneil
www.intelligentmeasurement.wordpress.com
Contacts: