outline of today’s discussion 1.developing questionnaires 2.in-class exercise: developing...
Post on 17-Jan-2018
231 Views
Preview:
DESCRIPTION
TRANSCRIPT
Outline of Today’s Discussion1. Developing Questionnaires
2. In-Class Exercise: Developing Questionnaires
Part 1
Developing Questionnaires
Developing Questionnaires1. Questionnaire - A written set of items that are
asked of every respondent in the study.
2. The items can be closed-ended or open-ended.
3. The questions may be self-administered, or interviewer-administered.
Developing Questionnaires1. Caution is needed to achieve accuracy and
precision when defining variables.
2. Ex: “Race” and “Ethnicity”.
“Hispanic” – A Spanish-speaking ethnic group.
“Latino” - people originating from North or South America, outside of the U.S. or Canada.
A person having European Spanish heritage is Hispanic and Caucasian (not Latino).
Developing Questionnaires
Rensis Likert (1903-1981)
B.A. University of Michigan 1926Ph.D. Columbia University 1932
Doctoral thesis addressed the now-famous Likert ScaleFounder: U of M’s Institute for Social Research
Director: 1946-1970
Developing Questionnaires1. Central Tendency Bias - Respondents may
avoid the extreme response categories.
2. To ensure the participants demonstrate “polarity” (a positive or negative tendency) remove the neutral response option:
Strongly Agree Agree Disagree Strongly Disagree
Issues for Likert-Like Scales
An even number of response categories can force “polarity”.On the other hand, if you want your participants to have the
option to express neutrality, use an odd number of response categories.
Developing Questionnaires1. Social Desirability Bias - Respondents may
portray themselves or their group in a favorable way.
2. Anecdote: Krueger & Clement on experimentally manipulated in-group out-group differences.
- Social Desirability- Out-group homogeneity effect
Issues for Likert-Like Scales
Developing Questionnaires1. Test-Retest Method - The same questionnaire is
administered to the same, large sample at two separate times.
2. Reliability is indexed by the extent to which the relative rankings of respondents remains stable.
3. Note: the overall scores on the second administration of the test may be higher or lower than the first. The critical factor is the relative rankings.
4. What statistic might we use for this?
Reliability
Developing Questionnaires1. An alternative would be the “split-half method”…
2. Let’s assume you want to measure depression, and you have a 100-item questionnaire on the topic.
3. The 100 items can be “split in half” (first half versus second half, or odd versus even items) and the two halves could be separately scored for each respondent.
4. If the test were reliable, people who score high on one half should score high on the other half.
Reliability
Developing Questionnaires1. Establishing validity is, typically, more complex than
establishing reliability.
2. How did Binet establish the validity of his IQ tests?
3. Binet’s efforts exemplified convergent validity - independent measurements of a given construct are correlated. (They independently “converge” on whatever it is they are measuring.)
4. Subtle Point: We can use correlations to inform us about validity, not just reliability!
Validity
Developing Questionnaires1. Ideally, we would also like to establish discriminant
validity - The ability of a questionnaire designed to measure one concept to yield answers that are NOT correlated with those of another questionnaire that measures a similar (but different) concept.
2. Discriminant Validity Example: “Life satisfaction”, versus “Positive Affect”. (See PubMed Abstract)
http://www.ncbi.nlm.nih.gov/pubmed/8831165
3. Potential Pop Quiz Question: Generate your own example of discriminant validity.
Validity
Developing QuestionnairesTips for good questionnaire items
-The vocabulary should be simple
-The item should be clear & specific
-No leading, loaded, or double-barreled items
-The item should be brief (<20 words)
Developing QuestionnairesTips for good questionnaire items
-Present all conditional information prior to the key ideaGood: If money were not an issue, what would you study in college?
Not As Good: What would you study in college, if money were not an issue?
-Detect response bias with oppositely worded itemsI often feel anxious: 1 2 3 4 5 I rarely feel anxious: 1 2 3 4 5
-Test for readability…
Developing Questionnaires
Pre-testing is critical!(say it with me)
Developing Questionnaires1. The Order of Questions Matters: Ex. “Do you
think it should be possible for a pregnant woman to obtain a legal abortion if she is married and does not more children?”
60.7% agreed when this question preceded another.48.1% agreed when this question followed another.
Schuman, Presser and Ludwig (1981).
2. What are “filter questions”, and what is their value?
Order Matters!
Developing Questionnaires1. Reported versus Actual Behavior…not as
congruent as we would like!
2. People self-report that they would help someone in need, regardless of by-standers. Behaviorally, the “by-stander” effect replicates reliably. (Latante & Darley, 1970)
3. This suggests a multi-method approach to a phenomenon may be best –surveys would be one of several methods.
About Interpretation
Developing QuestionnairesInterpreting Correlations:
Why Correlation Does Not Imply Causation
1. Spurious correlations – “Third Variables”
2. The direction of causation is ambiguous!
About Interpretation
Developing QuestionnairesAbout Interpretation
• At best, correlational analyses can only identify POTENTIAL causes.
• Path Analysis – a statistical technique that can help clarify the interpretation of correlations.
– Mediator: A variable that is used to “explain” the correlation between two variables.
– Moderator: A variable that affects the strength and/or direction of the correlation between two variables.
(e.g., interaction effects)
Developing QuestionnairesPath Analysis
Path A Path B
Path C
Mediator
D.V.Or
Criterion
I.V.Or
Predictor
Developing QuestionnairesPath Analysis
Path A Path B
Path C
S.E.S
College GPAAcademic
PerformanceSAT Scores
Developing QuestionnairesPath Analysis
Path A Path B
Path C
Chaos
PsychologicalDistress
Poverty
Developing QuestionnairesAbout Interpretation
• Assume that Poverty (predictor) correlates significantly with Psychological Distress (criterion).
• Is that correlation mediated (explained) by “Chaos”?
• Yes, if the data pass this “Mediational Test”:– Predictor & mediator are significantly correlated.– Criterion & mediator are significantly correlated.– The predictor & criterion correlation is significantly
reduced after controlling for the mediator.» We’ll describe the role of so-called “moderator” variables later,
if you take the stat course (PSYC 370).
Part 2
In-class ExerciseOn
Developing Surveys
AcknowledgmentsImages used in this educational presentation were obtained from Wikimedia Commons, in accordance with regulations regarding copyright, use, and dissemination.
http://commons.wikimedia.org/wiki/Main_Page
top related