lecture 5 data coding and experimental research methods
Post on 21-Dec-2015
218 views
TRANSCRIPT
Lecture 5Lecture 5
Data Coding and Experimental Data Coding and Experimental Research MethodsResearch Methods
22
OverviewOverview
Research Problems: my commentsResearch Problems: my comments
Working with DataWorking with Data Error-CheckingError-Checking Basic Data PreparationBasic Data Preparation
MethodsMethods Introduction to Experimental MethodsIntroduction to Experimental Methods
33
Problems and JustificationsProblems and Justifications
Research Question/Problem: What is the effect Research Question/Problem: What is the effect of obtaining a MIMS degree versus a CS MA of obtaining a MIMS degree versus a CS MA degree for getting a job in the tech industry?degree for getting a job in the tech industry?
Different types of degrees teach different skill Different types of degrees teach different skill sets. Jobs in the tech industry will look for these sets. Jobs in the tech industry will look for these skill sets, which should lead to different skill sets, which should lead to different outcomes for CS students versus MIMS outcomes for CS students versus MIMS students.students. Not a justification by itself– this is part of the Not a justification by itself– this is part of the context context
of the argumentof the argument. By itself, this just sets up an . By itself, this just sets up an argument that will lead to specific hypotheses.argument that will lead to specific hypotheses.
44
Problems and JustificationsProblems and Justifications(Round 2)(Round 2)
Research Question/Problem: What is the effect Research Question/Problem: What is the effect of obtaining a MIMS degree versus a CS MA of obtaining a MIMS degree versus a CS MA degree for getting a job in the tech industry?degree for getting a job in the tech industry?
If we can understand how different degrees (CS If we can understand how different degrees (CS versus MIMS) give different job opportunities, it versus MIMS) give different job opportunities, it has implications for future enrollment in various has implications for future enrollment in various academic programs. Furthermore, this research academic programs. Furthermore, this research could help us to begin the process of could help us to begin the process of differentiating between various fields of study.differentiating between various fields of study. Nope, still not a justification. These are Nope, still not a justification. These are implicationsimplications
of the research.of the research.
55
Problems and Justifications Problems and Justifications (Round 3)(Round 3)
Research Question/Problem: What is the effect of obtaining a MIMS Research Question/Problem: What is the effect of obtaining a MIMS degree versus a CS MA degree for getting a job in the tech degree versus a CS MA degree for getting a job in the tech industry?industry?
Knowing which graduate degree to obtain is a significant problem Knowing which graduate degree to obtain is a significant problem for those who want the best jobs in the tech industry. Once a for those who want the best jobs in the tech industry. Once a student joins a program, he or she makes a large commitment of student joins a program, he or she makes a large commitment of time and money. Degree-granting programs often entice potential time and money. Degree-granting programs often entice potential students to enroll based on the potential job prospects that they will students to enroll based on the potential job prospects that they will have once they receive a degree. Yet, it is not always clear to have once they receive a degree. Yet, it is not always clear to prospective students what skill sets are prospective students what skill sets are offeredoffered by different by different programs, and which skill sets are being programs, and which skill sets are being sought sought by the best by the best employers in the tech industry. This research will increase our employers in the tech industry. This research will increase our understanding of the relationship between degree type (CS versus understanding of the relationship between degree type (CS versus MIMS) and the related skill sets that ultimately lead to technology MIMS) and the related skill sets that ultimately lead to technology job outcomes. job outcomes.
66
Data PreparationData Preparation
Basic Data SetupBasic Data Setup Unique ID for each caseUnique ID for each case Numeric responses whenever possibleNumeric responses whenever possible
Includes categorical, scale, count, etcIncludes categorical, scale, count, etc Variable namesVariable names Variable labelVariable label Value labelValue label Missing CodesMissing Codes
RecodingRecoding SimpleSimple Intermediate (equations, computations)Intermediate (equations, computations)
77
Error Checking ExamplesError Checking Examples
Checking Original Variables for ErrorsChecking Original Variables for Errors FrequenciesFrequencies DescriptivesDescriptives
Checking and Setting “Missing” codesChecking and Setting “Missing” codes
Recoding and Creating New Variables from Recoding and Creating New Variables from Existing VariablesExisting Variables FrequenciesFrequencies Cross-TabulationsCross-Tabulations
88
Two Example Datasets:Two Example Datasets:
Class Data SetClass Data Set
Subset of 1993 General Social Survey Subset of 1993 General Social Survey
Introduction to Experimental Introduction to Experimental Design and MethodsDesign and Methods
1010
Pro’s and Con’s of ExperimentsPro’s and Con’s of Experiments
Pro’sPro’s Gives researcher tight control over independent factorsGives researcher tight control over independent factors Allows researcher to test key relationships with as few Allows researcher to test key relationships with as few
confounding factors as possibleconfounding factors as possible Allows for direct causal testingAllows for direct causal testing
Con’sCon’s Usually a smaller N than surveysUsually a smaller N than surveys Sometimes give up large amounts of external validity in favor of Sometimes give up large amounts of external validity in favor of
construct validity and direct causal analysisconstruct validity and direct causal analysis Require a large amount of planning, training, and time– Require a large amount of planning, training, and time–
sometimes to test relationship between only 2 factors!sometimes to test relationship between only 2 factors!
1111
Active versus Attribute Independent Active versus Attribute Independent VariablesVariables
ActiveActive independent variable(s): independent variable(s): The I.V. is given to the participants, usually for some The I.V. is given to the participants, usually for some
specified time period. It is often manipulated and specified time period. It is often manipulated and controlled by the investigator.controlled by the investigator.
Attribute Attribute independent variable(s):independent variable(s): A predictor, but a defining characteristic of individuals. A predictor, but a defining characteristic of individuals.
Cannot be manipulated. Cannot be manipulated.
1212
True ExperimentsTrue Experiments
True experimentsTrue experiments protect against both time and group threats to protect against both time and group threats to internal validity by internal validity by randomly assigning subjectsrandomly assigning subjects to treatment and to treatment and control groups. The treatment (independent variable) is active.control groups. The treatment (independent variable) is active.
If we cannot randomly assign subjects to different groups, then it is If we cannot randomly assign subjects to different groups, then it is a a quasi-experimentquasi-experiment. . The independent variable is active.The independent variable is active.
If we cannot randomly assign subjects to groups because the If we cannot randomly assign subjects to groups because the groups groups containcontain the attribute of interest, and if we give all groups the the attribute of interest, and if we give all groups the same treatment, then it is an same treatment, then it is an associational non-experimentassociational non-experiment.. The The independent variable is not active.independent variable is not active.
1313
Randomization in Sample and Randomization in Sample and AssignmentAssignment
Random Sample Random Sample System for choosing participants from a System for choosing participants from a
populationpopulation Generally, the larger the sampling population Generally, the larger the sampling population
the better your generalizability becomes.the better your generalizability becomes.
Random AssignmentRandom Assignment Method for assigning participants randomly to Method for assigning participants randomly to
experiment conditionsexperiment conditions
1414
Two Essential Criteria in True Two Essential Criteria in True Randomized Experimental DesignRandomized Experimental Design
(1) Independent Variables must be (1) Independent Variables must be manipulated (usually by experimenter, manipulated (usually by experimenter, sometimes by context)sometimes by context)
(2) Participants must be assigned (2) Participants must be assigned randomly to various conditions or groupsrandomly to various conditions or groups
1515
Pre-test, experimental manipulation Pre-test, experimental manipulation and post-testingand post-testing
Pre-test:Pre-test: allows us to check group equivalence allows us to check group equivalence before the intervention X is introduced.before the intervention X is introduced.
Experimental manipulation:Experimental manipulation: An independent An independent variable (X) that the experimenter manipulates.variable (X) that the experimenter manipulates.
Post-test:Post-test: allows us to check group equivalence allows us to check group equivalence after intervention X has been introduced. after intervention X has been introduced.
1616
Common types of true experimentsCommon types of true experiments
R
O
O
X O
O
R
X
O
O
R
O
O
X O
O
(1)
(2)
X O (3)
O (4)
Pretest-Posttest Control
Post-only Control
Solomon 4-group
1717
Example: Pen StudyExample: Pen Study
Question: Do individuals in Japan and the Question: Do individuals in Japan and the US make differential choices about US make differential choices about ‘unique’ versus ‘less unique’ items when ‘unique’ versus ‘less unique’ items when given a choice?given a choice?
1818
Pen StudyPen Study
Independent VariableIndependent Variable Cultural difference: Japanese students compared to Cultural difference: Japanese students compared to
US studentsUS students
AssignmentAssignment Subjects were not randomly assigned because they Subjects were not randomly assigned because they
already fell into one of the two groups.already fell into one of the two groups.
Dependent Variable:Dependent Variable: Pen layout (3 of one type, 1 of another)Pen layout (3 of one type, 1 of another) Would they choose the ‘common’ pen or the ‘unique’ Would they choose the ‘common’ pen or the ‘unique’
one?one?
1919
Example: Trust-Building StudyExample: Trust-Building Study
Question: Do increased risk-taking Question: Do increased risk-taking behaviors over time increase interpersonal behaviors over time increase interpersonal trust?trust?
2020
Trust-Building Study Trust-Building Study
Independent VariableIndependent Variable Experiment Condition (3 conditions): Experiment Condition (3 conditions):
Fixed partner on every trial, cannot control amount to entrust to partnerFixed partner on every trial, cannot control amount to entrust to partnerFixed partner on every trial, can control amount to entrust to partnerFixed partner on every trial, can control amount to entrust to partnerRandom partner on every trial, can control amount to entrust to partnerRandom partner on every trial, can control amount to entrust to partner
AssignmentAssignment Random assignment of participants to one of the 3 conditions.Random assignment of participants to one of the 3 conditions. Same experiment conducted in Japan and US, and comparisons made Same experiment conducted in Japan and US, and comparisons made
between the two studies.between the two studies.
Dependent VariableDependent Variable Cooperation rate (i.e., whether they returned the coins to the partner or Cooperation rate (i.e., whether they returned the coins to the partner or
not)not)
Validity: RevisitedValidity: Revisited
2222
Experiment Construct ValidityExperiment Construct Validity
How do we know that our independent variable How do we know that our independent variable is reflecting the is reflecting the intended causal constructintended causal construct and and nothing else?nothing else?
ContaminationContamination Demand CharacteristicsDemand Characteristics
AnythingAnything in the experiment that could guide subjects to in the experiment that could guide subjects to expected outcomeexpected outcome
Experimenter ExpectancyExperimenter ExpectancyResearcher behaviorResearcher behavior that guides subjects to expected that guides subjects to expected outcome (self-fulfilling prophecy)outcome (self-fulfilling prophecy)
2323
Solving Expectancy EffectsSolving Expectancy Effects
Naïve experimenterNaïve experimenter Those conducting study are not aware of theory or hypotheses Those conducting study are not aware of theory or hypotheses
in the experimentin the experiment
Blind Blind Researcher is unaware of the experiment condition that he/she Researcher is unaware of the experiment condition that he/she
is administeringis administering
StandardizationStandardization Experimenter follows a script, and only the scriptExperimenter follows a script, and only the script
““Canned” ExperimenterCanned” Experimenter Audio/Video/Print material gives instructionsAudio/Video/Print material gives instructions
2424
Other Issues with Demand Other Issues with Demand CharacteristicsCharacteristics
Evaluation Apprehension:Evaluation Apprehension: Subjects Subjects knowknow that they are being evaluated and this that they are being evaluated and this
changes their behaviorchanges their behavior
SolutionsSolutions Double-blind experimentsDouble-blind experiments Experiments in natural setting (i.e., subjects do not Experiments in natural setting (i.e., subjects do not
know they are in an experiment)know they are in an experiment) Cover storiesCover stories Hidden measurementsHidden measurements ““Faithful Subject”Faithful Subject”
2525
Generalizability (external validity) in Generalizability (external validity) in ExperimentsExperiments
Threats to external validity always involve Threats to external validity always involve an interaction of the treatment group with an interaction of the treatment group with some other factor.some other factor.
Threats usually fall into 3 types:Threats usually fall into 3 types: SettingSetting PopulationPopulation HistoryHistory
2626
Three threats to generalizability in Three threats to generalizability in experimentsexperiments
SettingSetting Physical and social context of the experimentPhysical and social context of the experiment
PopulationPopulation Is there something specific about the sample that Is there something specific about the sample that
interacts with the treatment?interacts with the treatment?
HistoryHistory Is there something about the time that interacts with Is there something about the time that interacts with
the treatment?the treatment?
2727
Why Generalizability is not always Why Generalizability is not always a problema problem
Experiments often are trying to isolate specific Experiments often are trying to isolate specific causes and effects in controlled settings. Thus, causes and effects in controlled settings. Thus, they may not even be claiming to be they may not even be claiming to be generalizable to specific settings.generalizable to specific settings.
Experimental findings can provide theoretical Experimental findings can provide theoretical basis for real-world tests. basis for real-world tests.
It is often a balancing act for research: true It is often a balancing act for research: true causation versus large-scale associational and causation versus large-scale associational and comparative testing.comparative testing.
2828
Considerations before using Considerations before using experimentsexperiments
Cost and EffortCost and Effort Is the effort worth it to test the concepts you are Is the effort worth it to test the concepts you are
interested in?interested in?
Manipulation and ControlManipulation and Control Will you actually be able to manipulate the key Will you actually be able to manipulate the key
concept(s)?concept(s)?
Importance of GeneralizabilityImportance of Generalizability Are you testing theory, or trying to establish a real-Are you testing theory, or trying to establish a real-
world test? world test?