data collection & technology educ 894 week 6. plan for today next week ▫no class,...
TRANSCRIPT
Data Collection & Technology
EDUC 894 Week 6
Plan for Today• Next Week
▫ No class, consultations available ▫ E-mail me by next Monday to set up a time▫ Reading handouts
• Instrument Focus: WebSurveys▫ What NOT to do: An Example
How many problems can you find?▫ What To Do: Principles
Designing a Valid & Reliable Survey▫ The Wario Survey▫ Working with WebSurvey data
• Instrument Focus: Eye-tracking & Brainwaves▫ Field trip to the Engrammetron
-----------------------------Break-----------------------------• Group Work
▫ Honing down your instruments
Survey Design
•(Most) surveys are conducted from an objectivist perspective
•Writing a survey is all about making decisions:1. Recognizing when a choice is being made2. Making choices that will create a reliable
instrument that will help you make valid inferences to answer your research questions
3. It’s (usually) not about making the “best” decision in some abstract sense, but making a choice that will be most useful for your purposes
Limitations of Surveys
•Self-report data▫Notoriously unreliable▫Indicates only what we think, not reality
•Data at one point in time▫Limited Sample
•Retrospective questions rely on memory•The Devil is in the Details
▫Question form, wording, and answer choices can inadvertently dictate responses
•Sampling ▫Who responds isn’t random
Image Source: http://www.nynpa.com/images/NYNPA0081.jpg
Survey Decisions
•Topics Covered•Number & Focus of Items per Topic•Item Sequence & Presentation•Item Type / Format•Item Wording
•Always think about what you will do with the answers you get – i.e. how will you analyze the data?
What NOT to do
•Take the survey posted here (link sent via email)
https://my.sfu.ca/cgi-bin/WebObjects/WebSurvey.woa/wa/survey?17437110
•How many problems in each category can you find? ▫Topics Covered▫Number & Focus of Items per Topic▫Item Sequence & Presentation▫Item Type / Format▫Item Wording
Survey Decisions Topics Covered• Remember the idea of content-related validity?
▫Goal:Determine the most popular animals▫Assumed “like” indicates popularity
What pet you had as a child may or may not relate Are the plants in you garden a better indicator?
Feelings about gardening don’t give relevant info Caveat: Questions to lay a foundation
• Define your terms ▫Are you concerned with popularity of animals (in
general) or only pets? Can ask a similar question about plants / garden
plants▫What about ability self-esteem (in general) or
self-esteem as related to academic work? How would this affect your survey design?
Survey Decisions Topics Covered
Self Esteem
Social
Academic
Athletic
Survey
Survey Decisions Number & Focus of Items per Topic•Number: Do you have enough questions
on each topic to give a reliable measure?▫Three questions is usually the absolute
minimum given for a scale•Focus: Do questions give you info you can
actually triangulate?▫How will you compare the names of plants
reported to be in their garden with Likert scales given for specific plants?
▫What will you do with the question about liking “trees without flowers”?
Survey DecisionsItem Sequence & Presentation• Question sets and introductions• Broad to narrow progression
• How satisfied are you with service in the food court?• Do servers in the food court smile when taking your
order?• Will you group or distribute similar content items?
• Putting similar items together will help learners think about what you want them to
• But, if you have multiple items try and get convergence on a construct, putting them near each other will lead to a false reliability
• Pagination• No one likes to scroll forever• No one likes unnecessary clicking
• The “required item” dilemma
Survey DecisionsItem Type / Format• Open versus Closed Ended Questions
▫It all depends on what you are looking for, but from an objectivist perspective:
“Use close ended questions where at all possible, if you can limit the possible answers it can really speed up the data crunching when you
are finished”
▫You may need open-ended questions when you don’t know the range of possible answers But then you will have to go through and code
these answers for themes▫Thus, don’t ask an open-question with a limited
number of possible answers or where answers range predictably “Do you like to garden?” “How much do you like dogs?”
Survey DecisionsOpen Ended Question Guidelines•Be specific
▫What did you think of class? BAD▫What 3 things did you like best about class? GOOD▫ What 3 things did you like best about the class content?
BETTER•Leave people enough space to write
▫A philosophy of life, love and happiness is not 64 characters or less!
•Don’t ask too many open-ended questions▫Your participants will tire and it will show in their
responses G. W. Bush example
Survey DecisionsClosed Ended Question GuidelinesQuestion Types 1
▫Yes/No▫Lists
• Single choice (“best answer”, radio button)
• Multi (“all that apply”, checkbox)• Can do a series of these in a grid, but
dangers of confusion and auto-pilotListen Speak Read Write
English □ □ □ □
Spanish □ □ □ □
French □ □ □ □
Italian □ □ □ □
Survey DecisionsClosed Ended Question Guidelines
Question Types 2▫ Rating Scales
• Ordinal • E.g. Rank the following from best to worst
• Likert • E.g. Levels of agreement
• Likert-style • E.g. Levels of approval, satisfaction etc.
• Frequency • E.g. Often, sometimes, almost never
• Bipolar • E.g. Outgoing to introverted
Survey DecisionsClosed Ended Question Guidelines
•Focus on Rating Scales▫ Graphic vs. Finite Choice
▫ Think about the following:• Does your scale goes to the extremes?• Does your scale have enough choices?
• Participants are less likely to choose the anchors (n-2)
• Does your scale imply interval spacing?• Will your scale have a “neutral” option?
Mark an X on the line to show your level of agreement
Disagree Agree
Indicate your level of agreement by circling your choice
Strongly Disagree Disagree Neutral Agree Strongly Agree
Survey DecisionsItem Wording 1•Clear & Concise
▫What do you do when you get a headache? UNCLEAR
▫What headache medicine do you use? CLEARER
•Concrete▫Did you enjoy the book? ABSTRACT
▫Did you recommend the book to a friend? CONCRETE
•One idea per question▫Should BC continue its urban density and needle
exchange programs? CONFOUNDED
•Be careful when asking personal questions▫Use categories to avoid emotional responses
Survey DecisionsItem Wording 2•Avoid leading questions
▫Do you think Barak Obama will continue to be a successful candidate?
▫Do you think Barak Obama will continue to be plagued by challenges about his level of experience?
•The challenge of reversed questions▫Can be used to break up question sequence or to get
at something in a different way, but big reliability issues I do not like venus fly-traps EXPLICIT
Self-Esteem Survey: Sometimes I doubt myself IMPLICIT
Survey DecisionsThe Big Picture
•Every time you have to make a decision, always think about 3 things1. How will your decision affect
participant responses?2. How will your decision affect your
analysis?3. What does your decision imply about
the underlying construct?
The Wario Survey
Working with WebSurvey Data
Field trip to the Engrammetron
Break
Group Work