ACC Survey Research Overview

Key considerations for practical surveys

“A high response rate does not compensate for a faulty questionnaire or weak data analysis”*

What is the purpose of your survey – state clear objective.

WHAT DO YOU WANT KNOW and WHY?

What is the best way for me to collect this data based on the purpose,

timeline, population, budget and my expertise?

Determine what actions will result from the data collection.

Ask - do we need all the data we want to collect right now? Don’t ask

questions that do not inform an action.

Does your questionnaire help you determine next steps or how to

improve?

4

*Pubic Perspective, May/June 2003

Your Questions

• What’s a good response rate?• How do I get people to respond?• What are the do and don’t dos of surveys?• How do I know what questions to include?• Can I just write my own questions?• Where can I find surveys that others are

doing?• Can I compare x and y?• What do I do with the data?

5

Common Surveys – Modes -Types of Data

• Satisfaction Survey• Program evaluation• Course evaluation• Needs assessment• Interest/perception• Sponsor

• Qualitative• Quantitative• Mixed-Method• Mixed-Mode

6

Paper Web Phone In-Person

Do this Don’t do this Articulate your purpose Have an action plan for the results Share the results and actions you

take Describe your population Use research ambassadors Test your survey Plan your surveys annually Protect anonymity, report groups

over 5 or 10 Check what ACC or industry

surveys have the same information

X Create long surveys (over ten minutes)

X Ask questions just because…X Avoid open-ended questionsX Ask questions with jargon,

acronyms or terms that aren’t clear

X Design a questionnaire without peer review (even if its just one person)

X Promise to share results and not share them

X Make generalizations with small samples

7

Three important questions for members• Value• Net promotion (would you recommend)• Intent to renew

– Yes – What benefit most important/useful, etc.

– No- Reason for not continuing• Engagement

– Last attended event– Last conference– Participate in webinar, feedback, survey,

voting etc.8

Exercise

Take 5 minutes and as a group, write down the number one thing you want to know from members in your chapter. Be ready to share the following:1. The research question (what you want to

know)2. The rationale for wanting/needing to know3. What the question would look like (write it)4. What you would do with the results5. How you might analyze the data

9

Common Survey Topics for Associations

Satisfaction• With services• With programs• With support• With cost/value

Expectations• Assess at sign-up• Track progress

meeting expectations and retention

Needs• What are

members most pressing needs

• How are current needs being addressed

COURSE/EVENT• Measure aspects not

the whole event• Consider a pre-

survey to leverage expectations with the instructor’s plans

10

Determine Mode

Web•Cost effective•Survey large numbers•Automatic skip patterns and data input•Analysis capability•Confidentiality

Phone•Personal contact•More in-depth•Time intensive/costly•Opportunity to test concepts•Not anonymous

Paper•Immediate feedback•Requires data input•Mail and return mail costs•Confidentiality

In-person

•Provides connectivity and observation•Limited participants but more in-depth•Not anonymous

11

Budget, Time, Resources!

Determine Mode• Paper survey or mobile onsite, e-mail web• Give time to complete, collect if in class • Consider pretest/posttest and learning follow-up to

help assess impact

Course evaluations

• Can be cost effective, especially for large population

• Requires good communication to boost response• Best practice is to Include open-ended opportunity

for comments on improving

Satisfaction

•In-person, phone, web, paper•May use open-ended questions to get unknown/unanticipated responses

Memberassessment

12

Designing questions

13

1. Map questions to your purpose2. Have an action item for each question 3. Use easy-to-understand terms- not jargon or acronyms4. Put definitions first on complex questions, not after the

question5. Use words on scales (i.e. use Strongly Agree, Somewhat Agree

etc., rather than 1 to 5)6. Think about how the order of questions influence each other 7. Use forced-answer and required questions sparingly8. Be mindful of confidentiality, anonymity and social desirability

when asking questions9. Scale from negative to positive10. Ask yourself, will the audience know about this? Do I Have an

option for them to select “Don’t Know”, “Not Sure”, “Prefer not to answer”, etc.

Common Pitfalls in Question DesignThe double-

barreled question

• How would you rate the conference and training today?

• Poor,• Fair,• Good• Excellent

Avoid double negatives

• I would not attend a training on cybersecurity

• Yes• No

The loaded question

• How honest are you at work?

• Not very• Somewhat• Mostly• Always

The ambiguous question

• How would you rate the training?

• Poor• Fair• Good• Excellent

14

Ask about the conference and training separately

Be specific to better isolate what you’re measuring, i.e. trainer, content, etc.

Consider asking about comfort speaking openly

Remove ‘not’ or ask about level of interest

Resources for questionnairesCourse Evaluation Topics1. Extent content meets needs of

attendees2. Leader assessment3. Effective teaching approach4. Facilities5. Schedule6. Learning aids effective in improving

communication and maintaining interest

7. Coordination satisfactory8. Opportunities to improve program

TemplatesACC, AAPOR, CASRO and other professional groups, Survey Monkey, QuestionPro, Survey Gizmo have templates for various surveys: http://www.questionpro.com/survey-templates/association-member-survey/ 15

 

Strongly Disagree

Disagree

Neither

Agree

Strongly Agree

Not applicable

The material covered in the program was relevant to my job            The materials was presented in an interesting way            The instructor was an effective communicator            The instructor was well prepared            The audiovisual/technical aids were effective            The materials will be helpful to me            The facilities were suitable            The schedule was suitable            There was  good balance between presence and group involvement            I feel the workshop will help me do my job better            What would improve the program?

Response 10% - 15% percent industry average

16

Web-surveys, 5 minutes or less is optimal, over 10 minutes is NOT.– Estimate roughly 11 to 12

seconds per question, 3-5 questions/min

– Roughly 15 questions– 1 to 2 open-ended

questions/min Paper surveys should be no more

than 10 questions, 15 if grid style or with

Phone interviews under 20 minutes can be completed without an appointment Schedule interviews of 30+

minutes

Boost response• Use ambassadors to build buy-in, let members know the survey

is coming and why (census example)• Consider incentives (monetary or in-kind, i.e. free event)• Clearly state the benefit of participating• Let members know how long the survey will take• Mobile optimize• Stay away from long grids on web surveys• Use multiple question pages but limit scrolling to avoid drop-

offs• Use four touch-points

17

Explain what you want, when the

survey will arrive, and reason to participate

Previte

Invite, provide link, assure

confidentiality, share benefits of participation

Invite

Call to action, create

personal link to the results,

remind of incentives

Remind and

thank

Final opportunity to be heard, why

you need participation

Remind and

thank

Incentives• Knowing the data will drive change, improvements or

address concerns is a big incentive, communicate this in pre-vite and invitation

• Share the report or write up results in a newsletter• Use monetary incentives if you have budget

– Studies conflict but small amounts are generally sufficient (little statistical difference between $5 and $20)

– Prepaid are most effective, promised incentives have limited impact.

– Charity donations– Web store (Amazon), local store or common chain

(Starbucks)• Check local laws if you want to have a drawing or

contest18

Generalizability and Coverage• Confidence Interval (how

confident you can be the data falls within the margin of error)

• Margin of error (+/- point or percentage)

• Estimate your sample neededhttp://americanresearchgroup.com/sams.html

http://www.raosoft.com/samplesize.html

http://

www.surveysystem.com/sscalc.htm

19

Confidentiality versus Anonymity: Be specific

Anonymity

• Researchers do not know the identity of the respondent and cannot connect data to individuals. No identifying data is recorded.

• Participants may feel more comfortable answering• Response may be higher• Researchers cannot track individual responses• Reminders go to the entire group as a thank you/reminder to

complete if one had not done so• Use general link to the survey

Confidentiality

• Researchers will not disclose personally identifiable information or share personal data at the individual level. Name, email and other contact variables are often known.

• Respondents must trust personal information is protected• Respondents may not feel comfortable sharing opinions on

sensitive topics• Ability to send respondent unique link makes outreach and

tracking response easier• Can preload demographics such as gender or birthdate

20

Web Surveys and Tools

• http://www.surveygizmo.com/

• http://freeonlinesurveys.com/login-form/

• www.surveymonkey.com

• www.Questionpro.com

Most online survey platforms have templates and free basic access.

21

Analysis

22

• Most online software programs have an analysis capability/reporting built into the platform

• Sample resource to help you work with data: http://aea365.org/blog/ • Only perform analysis that you understand- it doesn’t have to be a complex

analysis to be a good analysis• Distribution is important

• Mode is the answer that occurs most frequently• Mean or Average is the sum of responses divided by the number• Normal distribution is a bell curve

Example: Overall course rating = (40% very dissatisfied, 40% very satisfied) you would never use an average with this bi-modal distribution.

Qualitative Research: Map responses to your purpose, look for themes.Create theme words and code each response by theme to identify patterns.

Frequencies and Crosstabs Say A LotHave you created surveys at any time in the past or in your current role?

Yes No

Strongly disagree

0.0% 0.0%

Disagree 23.1% 0.0%

Neither agree nor disagree

38.5% 100%

Agree 23.1% 0.0%

Strongly agree

15.4% 0.0%

Not Applicable

0.0% 0.0%

100% 100%

23

I k

no

w w

he

re t

o l

oo

k f

or

info

rma

tio

n

wh

en

I n

ee

d h

elp

cre

ati

ng

a s

urv

ey.

Have you created surveys at any time in the past or in your current role?

Yes 92.9%

No 7.1%

1. Please describe your knowledge of survey research on a scale of 1 to 5, with 1 being no knowledge and 5 being an expert level of knowledge.

1 2 3 4 501234567

1

4

6

1 1

Value Percent Count

1 7.7% 1

2 30.8% 4

3 46.2% 6

4 7.7% 1

5 7.7% 1

Total 13

2. Have you created surveys at any time in the past or in your current role?

94

6

Yes No

Value Percent Count

Yes 93.8% 15

No 6.3% 1

Total 16

Please rate your level of agreement or disagreement with the following statement: I know where to look for information when I need help creating a survey.

25

44

19

13 Strongly disagree Disagree Neither agree nor disagree

Agree

Strongly agree Not Applicable

Value Percent Count

Strongly disagree 0.0% 0

Disagree 25.0% 4

Neither agree nor disagree 43.8% 7

Agree 18.8% 3

Strongly agree 12.5% 2

Not Applicable 0.0% 0

Total 16

ACC Resources and Strategy

• Membership• ACC Research

Planned for FY2015• Master calendar• Question Database• Survey resource portal for chapters

27