Slide 1

Copyright Allyn & Bacon (2010) Controls to Reduce Threats to Validity Graziano and Raulin Research Methods: Chapter 9 This multimedia product and its contents are protected under copyright law. The following are prohibited by law: (1) Any public performance or display, including transmission of any image over a network; (2) Preparation of any derivative work, including the extraction, in whole or in part, of any images; (3) Any rental, lease, or lending of the program. Slide 2 Copyright Allyn & Bacon (2010) Threats to Validity Covered in Chapter 8 Covered in Chapter 8 Validity can be threatened in many ways Validity can be threatened in many ways Presence of confounding variables Unrepresentative samples Inappropriate statistical tests or violations of statistics assumptions Subject and experimenter effects All these threats can be controlled All these threats can be controlled Slide 3 Copyright Allyn & Bacon (2010) Control Procedures General control procedures (applicable to virtually all research) General control procedures (applicable to virtually all research) Control over subject and experimenter effects Control over subject and experimenter effects Control through the selection and assignment of participants Control through the selection and assignment of participants Control through experimental design Control through experimental design Slide 4 Copyright Allyn & Bacon (2010) General Control Procedures Preparation of the setting Preparation of the setting Free of distractions that might interfere A natural setting increases external validity Response Measurement Response Measurement Use reliable and valid measures Replication Replication Demonstrates that findings are consistent and robust Slide 5 Copyright Allyn & Bacon (2010) Types of Replication Exact Replication Exact Replication Repeating a study using identical procedures to the original Systematic Replication Systematic Replication Using a theoretical or procedural change Conceptual Replication Conceptual Replication Varying the operational definitions of the variables to get new research hypotheses Slide 6 Copyright Allyn & Bacon (2010) Subject and Experimenter Effects Blind procedures Blind procedures Best control for expectancy effects Single-blind: The experimenter does not know what condition the participant is in Double-blind: Neither the experimenter nor the participant knows what condition the participant is in Slide 7 Copyright Allyn & Bacon (2010) Subject and Experimenter Effects Automation Automation Reduces contact between participants and the experimenter Gives the experimenter less opportunity to affect participants Using objective measures Using objective measures Objective measure require less judgment Provides less opportunity for subtle experimenter biases to affect the data Slide 8 Copyright Allyn & Bacon (2010) Subject and Experimenter Effects Multiple observers Multiple observers Reduces bias because it challenges observers to be as precise and objective as possible Can measure amount of observer agreement (percent agreement or Kappa) Using deception Using deception Hides purpose of the study from participants Balanced placebo design is a good example Slide 9 Copyright Allyn & Bacon (2010) Balanced Placebo Design Separates the pharmacological effects from the expectancy effects of alcohol Separates the pharmacological effects from the expectancy effects of alcohol A two-factor design A two-factor design Factor 1 is whether the person drinks alcohol Factor 2 is whether the person thinks he or she is drinking alcohol Slide 10 Copyright Allyn & Bacon (2010) Balanced Placebo Design This design crosses the consumption of alcohol with the belief that alcohol is being consumed People Led to Believe Drinking Alcohol Not Drinking Alcohol Actual Situation Drinking Alcohol Not Drinking Alcohol Slide 11 Copyright Allyn & Bacon (2010) Participant Selection Can generalize only if your sample is representative Can generalize only if your sample is representative Populations and samples Populations and samples General population: all potential participants Target population: those participants you are interested in Accessible population: portion of target population that is available to the researcher Sample: drawn from the accessible population Slide 12 Copyright Allyn & Bacon (2010) Populations and Sampling This figure shows the relationship between the various populations This figure shows the relationship between the various populations General Population Target Population Accessible Population Sample Slide 13 Copyright Allyn & Bacon (2010) Sampling Procedures Random sampling Random sampling Every participant has an equal chance of being sampled Stratified random sampling Stratified random sampling Random sampling within strata (subgroups) Ad hoc samples Ad hoc samples Random sample from accessible population Must generalize cautiously Should describe sample to help define limits of generalization Slide 14 Copyright Allyn & Bacon (2010) Participant Assignment Assignment Procedures Assignment Procedures Free random assignment Random assignment of participants to groups Random assignment of participants to groups Randomize within blocks Randomly assign in blocks of one participant per condition Randomly assign in blocks of one participant per condition Matched random assignment Random assignment of participants in matched sets to groups Random assignment of participants in matched sets to groups Other matching procedures e.g., match groups on key characteristics e.g., match groups on key characteristics Slide 15 Copyright Allyn & Bacon (2010) Matched Random Assignment Match on relevant variables Match on relevant variables Variables likely to affect the dependent measure Variables that show the largest variability in the population Procedures Procedures Match in sets on the relevant variable Set size is the number of groups in the study Set size is the number of groups in the study Randomly assign participants from the set, one to each group Keep track of matching data for the statistical analysis Slide 16 Copyright Allyn & Bacon (2010) Experimental Design Main focus of Chapters 10 through 13 Main focus of Chapters 10 through 13 Experimental design maximizes validity Experimental design maximizes validity Need to also include the other control procedures covered in this chapter Key elements of experiments Key elements of experiments One or more control groups Random assignment of participants to groups Slide 17 Ethical Principles Balanced Placebo Design raises several ethical issues Balanced Placebo Design raises several ethical issues Alcohol is a controlled substance Intoxication poses risks Some individuals are at especially high risk (e.g., people with certain medical conditions) Must address these issues Must address these issues Copyright Allyn & Bacon (2010) Slide 18 Screening Participants Must assure that participants are legally old enough to consume alcohol Must assure that participants are legally old enough to consume alcohol Must screen out people who abuse alcohol and those with no experience with alcohol Must screen out people who abuse alcohol and those with no experience with alcohol Must exclude those with medical problems that might be exacerbated by alcohol Must exclude those with medical problems that might be exacerbated by alcohol Must inform potential participants that alcohol may be consumed, so those with moral objects can decline Must inform potential participants that alcohol may be consumed, so those with moral objects can decline Copyright Allyn & Bacon (2010) Slide 19 Deception Participants know that, depending on the condition, they may or may not be consuming alcohol Participants know that, depending on the condition, they may or may not be consuming alcohol They agree to this in the informed consent What they dont know is that they may be deceived about the condition to which they are assigned Debriefing is required to clear up misconceptions Debriefing is required to clear up misconceptions Copyright Allyn & Bacon (2010) Slide 20 Protecting Participants Must have medical safeguards available in the event of an adverse reaction Must have medical safeguards available in the event of an adverse reaction Must assure that the participant does not drive intoxicated Must assure that the participant does not drive intoxicated Usually by keeping them in the lab until the blood alcohol level has dropped Copyright Allyn & Bacon (2010) Slide 21 Summary Most threats to validity can be minimized with proper use of control procedures Most threats to validity can be minimized with proper use of control procedures Broad classes of control procedures Broad classes of control procedures General control procedures Control over subject and experimenter effects Control through participant selection and assignment Control through specific experimental design