tell me what to ask designing surveys: practice makes pretty good research & evaluation...
TRANSCRIPT
Tell me what to ask
Designing Surveys: Practice makes pretty
goodResearch & Evaluation
Strategic Planning & ImplementationKamehameha Schools
Presented by: Brandon Ledward, Wendy Kekahio, Summer Keli’ipio, Nolan Malone and Kathy
Tibbetts10 December 2007
Activity 1Please stand if you’ve ever:
(No shame! Get prizes, you know.)
Agenda• Activity 1• How do we collect data?• Activity 2• Before you get started• Activity 3• Creating a survey• Item Types• Activity 4
How do we collect data?Quantitative• Surveys/
evaluations• Forms• Roll books/
participation counts
• Assessments/tests
• Collaborators
Qualitative• Observations• Interviews• Artifacts• Open-ended
survey questions
Do you really need/want a survey?
Activity 2• Envelope contains 5 scenarios• What method of data collection
would you use? Why?– Interview– Focus Group– Survey– Observation
There’s no wrong answer
Before you get started
What is a survey? • Method of gathering information
– Sample vs. Census
• Every individual is asked the same question in more or less the same way
• Create a composite profile of a population
• Types of surveys:– Mail– Telephone– In-person (“paper & pencil” or “show of
hands”)– Internet
Before you get started
How do I create a good survey?
The quality of a survey is largely determined by its purpose and the way it is conducted.
Scheuren, Fritz. “What is a Survey?” 2004.
Before you get started
How do I begin?• Focus on the purpose of the survey
– Where does the purpose come from?• See Workshop #1: Logic Models Are Our Friends
• Know how you will use the information
• Allocate resources accordingly– Do you need to give makana?– What time of day is best?– Can the survey take a long time?
Activity 3• With someone at your table
discuss one program outcome you each wish to measure– Why do you need to measure this?– How are you going to use the
information?– What are some challenges?
Be prepared to share with the group
Creating a surveyStep 1: Define the purpose
• Why do you need a survey?• What are you going to do with the
information? (i.e. reporting)• Have you verified the purpose with
leadership and staff?
Creating a surveyStep 2: Plan the administration
of the survey• What method? (mail, in-person,
internet, telephone, etc.)• How many people will take the
survey?• How long will the survey take?• When will the survey be
administered?• What logistical issues should be
considered? (age group, materials, makana)
Step 3: Write questions & then pilot those questions
Step 4: Review & revise your questions and fully implement your survey
Creating a survey
easier s
aid than
done!
ChecklistRevisions Approval START!
Survey Basics: Item TypesTwo item types:• Open-ended• Close-ended
– Ratings– Multiple choice(s)
Survey Basics: Item TypesExamples: Open-ended questions
1. What was the single best part of your experience today?
____________________________________________________________
2a. Did participating in this program impact your future career goals?
o Yeso Noo Not sure
2b. Please explain:__________________________________________________________________________________________
Survey Basics: Item TypesExamples: Close-ended questions
Rating scales
1. Rate each activity:Poor Fair Neutral Good Excellent N/A
Beach LunchStoriesSinging
2. I learned educational strategies I can use in my classroom.
1 2 3 4 5 6 7 8 9 10
Strongly Disagree
Strongly Agree
Survey Basics: Item Types• Examples: Close-ended questions
Multiple Choice(s)
1. What language is primarily spoken in
your home? (Choose one only).
○ English○ Pidgin○ Hawaiian○ Some other language
(specify):__________
Activity 4: Writing your questions
Go back to the program outcome that you wanted to measure:
Write down 3-5 questions to measure your program outcome.
With someone at your table, discuss how you would turn one of these questions into a survey item.
Be prepared to share with the group
Survey Basics: Reporting• Begin at the first step
– If you cannot find a reason to report a piece of information don’t waste a question on your survey
• Know beforehand how info will fit into your report
• Be brief in reporting all findings
Mahalo a nui loaAdditional questions?
Brandon Ledward Summer Keli‘ipio Wendy Kekahio541-5325 534-3964 [email protected] [email protected]
Nolan Malone Kathy Tibbetts 534-3849 541-5321