tell me what to ask designing surveys: practice makes pretty good research & evaluation...

Tell me what to ask

Designing Surveys: Practice makes pretty

goodResearch & Evaluation

Strategic Planning & ImplementationKamehameha Schools

Presented by: Brandon Ledward, Wendy Kekahio, Summer Keli’ipio, Nolan Malone and Kathy

Tibbetts10 December 2007

Activity 1Please stand if you’ve ever:

(No shame! Get prizes, you know.)

Agenda• Activity 1• How do we collect data?• Activity 2• Before you get started• Activity 3• Creating a survey• Item Types• Activity 4

How do we collect data?Quantitative• Surveys/

evaluations• Forms• Roll books/

participation counts

• Assessments/tests

• Collaborators

Qualitative• Observations• Interviews• Artifacts• Open-ended

survey questions

Do you really need/want a survey?

Activity 2• Envelope contains 5 scenarios• What method of data collection

would you use? Why?– Interview– Focus Group– Survey– Observation

There’s no wrong answer

Before you get started

What is a survey? • Method of gathering information

– Sample vs. Census

• Every individual is asked the same question in more or less the same way

• Create a composite profile of a population

• Types of surveys:– Mail– Telephone– In-person (“paper & pencil” or “show of

hands”)– Internet

Before you get started

How do I create a good survey?

The quality of a survey is largely determined by its purpose and the way it is conducted.

Scheuren, Fritz. “What is a Survey?” 2004.

Before you get started

How do I begin?• Focus on the purpose of the survey

– Where does the purpose come from?• See Workshop #1: Logic Models Are Our Friends

• Know how you will use the information

• Allocate resources accordingly– Do you need to give makana?– What time of day is best?– Can the survey take a long time?

Activity 3• With someone at your table

discuss one program outcome you each wish to measure– Why do you need to measure this?– How are you going to use the

information?– What are some challenges?

Be prepared to share with the group

Creating a surveyStep 1: Define the purpose

• Why do you need a survey?• What are you going to do with the

information? (i.e. reporting)• Have you verified the purpose with

leadership and staff?

Creating a surveyStep 2: Plan the administration

of the survey• What method? (mail, in-person,

internet, telephone, etc.)• How many people will take the

survey?• How long will the survey take?• When will the survey be

administered?• What logistical issues should be

considered? (age group, materials, makana)

Step 3: Write questions & then pilot those questions

Step 4: Review & revise your questions and fully implement your survey

Creating a survey

easier s

aid than

done!

ChecklistRevisions Approval START!

Survey Basics: Item TypesTwo item types:• Open-ended• Close-ended

– Ratings– Multiple choice(s)

Survey Basics: Item TypesExamples: Open-ended questions

1. What was the single best part of your experience today?

____________________________________________________________

2a. Did participating in this program impact your future career goals?

o Yeso Noo Not sure

2b. Please explain:__________________________________________________________________________________________

Survey Basics: Item TypesExamples: Close-ended questions

Rating scales

1. Rate each activity:Poor Fair Neutral Good Excellent N/A

Beach LunchStoriesSinging

2. I learned educational strategies I can use in my classroom.

1 2 3 4 5 6 7 8 9 10

Strongly Disagree

Strongly Agree

Survey Basics: Item Types• Examples: Close-ended questions

Multiple Choice(s)

1. What language is primarily spoken in

your home? (Choose one only).

○ English○ Pidgin○ Hawaiian○ Some other language

(specify):__________

Activity 4: Writing your questions

Go back to the program outcome that you wanted to measure:

Write down 3-5 questions to measure your program outcome.

With someone at your table, discuss how you would turn one of these questions into a survey item.

Be prepared to share with the group

Survey Basics: Reporting• Begin at the first step

– If you cannot find a reason to report a piece of information don’t waste a question on your survey

• Know beforehand how info will fit into your report

• Be brief in reporting all findings

Mahalo a nui loaAdditional questions?

Brandon Ledward Summer Keli‘ipio Wendy Kekahio541-5325 534-3964 523-6373brledwar@ksbe.edu sukeliip@ksbe.edu

wekekahi@ksbe.edu

Nolan Malone Kathy Tibbetts 534-3849 541-5321

nomalone@ksbe.edu katibbet@ksbe.edu