Experimental DesignExperimental Design

Brian MenneckeCollege of Business

Iowa State University

Brian MenneckeCollege of Business

Iowa State University

The Source…The Source…

The source of much of this information comes from Campbell & Stanley…– Campbell, D. T. and J.C. Stanley (1963).

Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally College Publishing Company.

The source of much of this information comes from Campbell & Stanley…– Campbell, D. T. and J.C. Stanley (1963).

Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally College Publishing Company.

The Goal of ResearchThe Goal of Research

When one conducts science, the goal is to seek out the truth. – Question: How does one identify truth?

Experimentation is one mechanism for identifying causation, which is a step toward understanding how one set of factors influence another set of factors

When one conducts science, the goal is to seek out the truth. – Question: How does one identify truth?

Experimentation is one mechanism for identifying causation, which is a step toward understanding how one set of factors influence another set of factors

Causation and PositivismCausation and Positivism

Positivism is a research perspective that has as its premise that inferences about cause can be made.

David Hume espoused the conditions by which inference could be made; these include…– Contiguity between the cause and effect– Temporal precedence– Constant conjunction (i.e., when the effect is seen,

the cause is always present)

Positivism is a research perspective that has as its premise that inferences about cause can be made.

David Hume espoused the conditions by which inference could be made; these include…– Contiguity between the cause and effect– Temporal precedence– Constant conjunction (i.e., when the effect is seen,

the cause is always present)

But how do we know something is true?But how do we know something is true?

Some propositions are not true; how do we know when something is true or not?

One approach is to test for validity.– Validity is a term used to describe whether

the conclusions one draws about a proposition are true or false

Some propositions are not true; how do we know when something is true or not?

One approach is to test for validity.– Validity is a term used to describe whether

the conclusions one draws about a proposition are true or false

Types of validityTypes of validity

Internal Validity: How sure are we that the cause leads to the expected results? In other words, is it appropriate for us to infer that the relationship between variables is causal

External Validity: How sure are we that we can generalize the finding of causation to other populations, settings, or variables?

Construct Validity: How sure are we that the variables we are using actually measure the concept (i.e., the construct) that we are seeking to measure?

Statistical Conclusion Validity: Do the statistical tests that we perform accurately measure the relationships between the variables under study?

Internal Validity: How sure are we that the cause leads to the expected results? In other words, is it appropriate for us to infer that the relationship between variables is causal

External Validity: How sure are we that we can generalize the finding of causation to other populations, settings, or variables?

Construct Validity: How sure are we that the variables we are using actually measure the concept (i.e., the construct) that we are seeking to measure?

Statistical Conclusion Validity: Do the statistical tests that we perform accurately measure the relationships between the variables under study?

Threats to Internal Validity (Campbell & Stanley)Threats to Internal Validity (Campbell & Stanley)

History: events that occur between the first and second measurement that are unrelated to the experiment but that could affect the results.

Maturation: Changes in the participants that occur as a function of the passage of time and not specific to the experiment.

Testing: The effects of taking a first test on the scores of a second test. Instrumentation: Changes in the measurement instrument or changes

or the observers make changes in the obtained measurements. Statistical regression (toward to mean): Groups having extreme scores

on the pretest (or selected on the basis of extreme scores) will tend to have scores closer to the mean on the posttest.

Selection: Biases resulting in differentials selection of respondents for the comparison groups.

Experimental mortality: Differential loss of respondents from the comparison groups.

Selection-maturation interaction, other interaction effects:

History: events that occur between the first and second measurement that are unrelated to the experiment but that could affect the results.

Maturation: Changes in the participants that occur as a function of the passage of time and not specific to the experiment.

Testing: The effects of taking a first test on the scores of a second test. Instrumentation: Changes in the measurement instrument or changes

or the observers make changes in the obtained measurements. Statistical regression (toward to mean): Groups having extreme scores

on the pretest (or selected on the basis of extreme scores) will tend to have scores closer to the mean on the posttest.

Selection: Biases resulting in differentials selection of respondents for the comparison groups.

Experimental mortality: Differential loss of respondents from the comparison groups.

Selection-maturation interaction, other interaction effects:

Threats to External ValidityThreats to External Validity

Reactive or interaction effects of testing: The pretest itself might be a learning experience such that by taking the pretest students gain information that will affect posttest results

Interaction of selection and the experimental variable: Different groups may respond differently to the experimental variable.

Reactive effects of experimental arrangements: Subjects respond differently because they know they are in an experiment (i.e., the Hawthorne effect)

Multiple treatment interference: Multiple treatments applied to the same respondents; the effects of prior treatments cannot be erased.

Reactive or interaction effects of testing: The pretest itself might be a learning experience such that by taking the pretest students gain information that will affect posttest results

Interaction of selection and the experimental variable: Different groups may respond differently to the experimental variable.

Reactive effects of experimental arrangements: Subjects respond differently because they know they are in an experiment (i.e., the Hawthorne effect)

Multiple treatment interference: Multiple treatments applied to the same respondents; the effects of prior treatments cannot be erased.

What is the basis for asking questions about causation? What is the basis for asking questions about causation?

The source for all questions pertaining to research experimentation is theory– Why is theory important?

Theory should always drive research because it defines expectations about the relationships that exist between variables.

The source for all questions pertaining to research experimentation is theory– Why is theory important?

Theory should always drive research because it defines expectations about the relationships that exist between variables.

Before we get started…Before we get started… Some definitions:

– Construct: An idea or concept that you are attempting to measure• Latent Construct: A construct that cannot be measured

directly (e.g., group cohesion)– Independent Variable: Variables that are presumed

to be the cause of an effect being studied; independent variables are manipulated to examine their impact on results

– Dependent Variables: Variables that are observed to understand the result of causation.

– Hypothesis: A statement of a possible explanation for causation. An hypothesis is tested by drawing conclusions from an experimental examination of the variables that are expected to be related

Some definitions:– Construct: An idea or concept that you are

attempting to measure• Latent Construct: A construct that cannot be measured

directly (e.g., group cohesion)– Independent Variable: Variables that are presumed

to be the cause of an effect being studied; independent variables are manipulated to examine their impact on results

– Dependent Variables: Variables that are observed to understand the result of causation.

– Hypothesis: A statement of a possible explanation for causation. An hypothesis is tested by drawing conclusions from an experimental examination of the variables that are expected to be related

Types of Experimental DesignsTypes of Experimental Designs

Pre-experimental designs: One group designs and designs that compare pre-existing groups

Quasi-experimental designs: Experiments that have treatments, outcome measures, and experimental conditions but that do not use random selection and assignment to treatment conditions.

True experimental designs: Experiments that have treatments, outcome measures, and experimental conditions and use random selection and assignment to treatment conditions. This is the strongest set of designs in terms of internal and external validity.

Pre-experimental designs: One group designs and designs that compare pre-existing groups

Quasi-experimental designs: Experiments that have treatments, outcome measures, and experimental conditions but that do not use random selection and assignment to treatment conditions.

True experimental designs: Experiments that have treatments, outcome measures, and experimental conditions and use random selection and assignment to treatment conditions. This is the strongest set of designs in terms of internal and external validity.

Pre-Experimental DesignsPre-Experimental Designs

Design 1: One-Shot Case Study: A single group is studied once after some intervention/treatment that is presumed to cause change. – For example, a training program is

implemented and participants are given a posttest at the conclusion of the training.

X O

Design 1: One-Shot Case Study: A single group is studied once after some intervention/treatment that is presumed to cause change. – For example, a training program is

implemented and participants are given a posttest at the conclusion of the training.

X O

Pre-Experimental DesignsPre-Experimental Designs

Design 2: One-Group Pretest-Posttest Design: One group, not randomly selected nor randomly assigned, is given a pretest, followed by a treatment/intervention, and finally a posttest. There is no comparison group. Generally done with intact groups.– For example, a classroom teacher gives her

students a pretest then implements an instructional strategy followed by a posttest.

O1 X O2

Design 2: One-Group Pretest-Posttest Design: One group, not randomly selected nor randomly assigned, is given a pretest, followed by a treatment/intervention, and finally a posttest. There is no comparison group. Generally done with intact groups.– For example, a classroom teacher gives her

students a pretest then implements an instructional strategy followed by a posttest.

O1 X O2

Pre-Experimental DesignsPre-Experimental Designs

Design 3: The Static-Group Comparison: One group which has experienced a treatment/intervention (X) is compared to another group that has not had the intervention. The groups are not randomly selected nor randomly assigned and are generally pre-existing groups. There is no pre-observation/pretest. – For example, comparison of GRE scores for students who

attended a rural high school versus those who attended an urban high school.

X1 O X2 O

Design 3: The Static-Group Comparison: One group which has experienced a treatment/intervention (X) is compared to another group that has not had the intervention. The groups are not randomly selected nor randomly assigned and are generally pre-existing groups. There is no pre-observation/pretest. – For example, comparison of GRE scores for students who

attended a rural high school versus those who attended an urban high school.

X1 O X2 O

True Experimental DesignsTrue Experimental Designs

Design 4: Pretest-Posttest Control Group Design: One group is administered a treatment while the other is not; all groups are observed before and after the treatment is administered. – For example, 50 freshman students are randomly selected to

participate in a tutoring study. Half are randomly assigned to a tutor for their first semester and half are not. All students are given a pretest at the beginning of the term and a posttest at the end of the term.

R O1 X O2R O1 O2

Design 4: Pretest-Posttest Control Group Design: One group is administered a treatment while the other is not; all groups are observed before and after the treatment is administered. – For example, 50 freshman students are randomly selected to

participate in a tutoring study. Half are randomly assigned to a tutor for their first semester and half are not. All students are given a pretest at the beginning of the term and a posttest at the end of the term.

R O1 X O2R O1 O2

True Experimental DesignsTrue Experimental Designs Design 5: Solomon Four-Group Design: This design involves four

experimental groups. Two of the groups parallel the structure of Design 4 while the remaining two groups include no pre-test (so that the effects of the pretest can be evaluated). – For example, 100 freshman students are randomly selected to

participate in a tutoring study. 25 are randomly assigned to a tutor for their first semester and given a pretest. 25 are randomly assigned to a group where no tutor is assigned and they are given a pretest. Another 25 are randomly assigned to a tutor but not given a pretest. The remaining 25 are randomly assigned to a group where no tutor is assigned and they are not given a pretest. Whew!

R O1 X O2R O1 O2R X O2R O2

Design 5: Solomon Four-Group Design: This design involves four experimental groups. Two of the groups parallel the structure of Design 4 while the remaining two groups include no pre-test (so that the effects of the pretest can be evaluated). – For example, 100 freshman students are randomly selected to

participate in a tutoring study. 25 are randomly assigned to a tutor for their first semester and given a pretest. 25 are randomly assigned to a group where no tutor is assigned and they are given a pretest. Another 25 are randomly assigned to a tutor but not given a pretest. The remaining 25 are randomly assigned to a group where no tutor is assigned and they are not given a pretest. Whew!

R O1 X O2R O1 O2R X O2R O2

True Experimental DesignsTrue Experimental Designs Design 6: Posttest Only Control Group Design: One

group is administered a treatment while the other is not; all groups are observed after the treatment is administered BUT not before the treatment. – For example, students are randomly assigned to two groups of

50 each. The experimental (treatment) group receives a new teaching method during a special class session. The second group (the control) receives a traditional teaching method during a special class session. No pretest is used for each group. Issues such as existing grades, SAT scores, and other factors are examined as covariates.

R X O2R O2

Design 6: Posttest Only Control Group Design: One group is administered a treatment while the other is not; all groups are observed after the treatment is administered BUT not before the treatment. – For example, students are randomly assigned to two groups of

50 each. The experimental (treatment) group receives a new teaching method during a special class session. The second group (the control) receives a traditional teaching method during a special class session. No pretest is used for each group. Issues such as existing grades, SAT scores, and other factors are examined as covariates.

R X O2R O2

Quasi-Experimental DesignsQuasi-Experimental Designs Design 7: The Time-Series Experiment: This

design involves periodic measurements of some group or individuals and the introduction of a change into the conditions during the series. – For example, studying a group of workers over time

and taking several measures of productivity during this period. At some point a new work process is introduced and measures of productivity are taken over several weeks following the intervention.

O1 O2 O3 X O4 O5 O6

Design 7: The Time-Series Experiment: This design involves periodic measurements of some group or individuals and the introduction of a change into the conditions during the series. – For example, studying a group of workers over time

and taking several measures of productivity during this period. At some point a new work process is introduced and measures of productivity are taken over several weeks following the intervention.

O1 O2 O3 X O4 O5 O6

Quasi-Experimental DesignsQuasi-Experimental Designs Design 8: Equivalent Time-Samples Designs:

This design involves periodic introduction of treatments followed by measurements with the treatments varied consistently over time. – For example, to study the effect on student

discussions of having an observer appear in a classroom. At time period one, an observer is present and a measure of discussion level is made. At time two, no observer is present and a measure of discussion level is made. At time three an observer is present, a measure is taken. At time four an observer is not present, a measure is taken. Etc.

X1 O X2 O X1 O X2 O

Design 8: Equivalent Time-Samples Designs: This design involves periodic introduction of treatments followed by measurements with the treatments varied consistently over time. – For example, to study the effect on student

discussions of having an observer appear in a classroom. At time period one, an observer is present and a measure of discussion level is made. At time two, no observer is present and a measure of discussion level is made. At time three an observer is present, a measure is taken. At time four an observer is not present, a measure is taken. Etc.

X1 O X2 O X1 O X2 O

Quasi-Experimental DesignsQuasi-Experimental Designs Design 9: The Equivalent Materials Design: This

design involves giving equivalent samples of materials to subjects, imparting interventions, and then making observations. – For example, subjects are asked to complete a survey

instrument about their opinions related to current events. The students are then split into two groups and given two different sets of (falsified) survey results indicating how other students answered the survey. Both groups are then asked to complete the survey again to observe how they respond.

Experimental Materials A(O) X0 OExperimental Materials B(O) X0 O

Design 9: The Equivalent Materials Design: This design involves giving equivalent samples of materials to subjects, imparting interventions, and then making observations. – For example, subjects are asked to complete a survey

instrument about their opinions related to current events. The students are then split into two groups and given two different sets of (falsified) survey results indicating how other students answered the survey. Both groups are then asked to complete the survey again to observe how they respond.

Experimental Materials A(O) X0 OExperimental Materials B(O) X0 O

Quasi-Experimental DesignsQuasi-Experimental Designs Design 10: Nonequivalent Control Group: This design

involves an experimental and control group with both given pretests and posttest; however, these groups are not randomly selected because they constitute naturally assembled groups (e.g. classrooms). The assignment of X (the treatment) to one group or the other is randomly selected by the researcher. – For example, four sections of a course are chosen to

participate in a study of teaching methods. Half are randomly assigned a new teaching method and half are not. All are given pretests at the beginning of the term and all are given posttests at the end of the semester.

O X OO O

Design 10: Nonequivalent Control Group: This design involves an experimental and control group with both given pretests and posttest; however, these groups are not randomly selected because they constitute naturally assembled groups (e.g. classrooms). The assignment of X (the treatment) to one group or the other is randomly selected by the researcher. – For example, four sections of a course are chosen to

participate in a study of teaching methods. Half are randomly assigned a new teaching method and half are not. All are given pretests at the beginning of the term and all are given posttests at the end of the semester.

O X OO O

Quasi-Experimental DesignsQuasi-Experimental Designs Design 11: Counterbalanced Designs: In this design all subjects

receive all treatments but in a different order. Each treatment occurs once at each time period and once for each treatment group. A Latin-square design is a type of counterbalanced design in which four treatments are applied to four naturally assembled pools of subjects.

– For example, consider a study of the effect of different training methods on learning. Subjects are placed into four groups (A,B,C, D) for different training methods, X1-X4.

Group A X1O X2O X3O X4OGroup B X2O X4O X1O X3OGroup C X3O X1O X4O X2OGroup D X4O X3O X2O X1O

Design 11: Counterbalanced Designs: In this design all subjects receive all treatments but in a different order. Each treatment occurs once at each time period and once for each treatment group. A Latin-square design is a type of counterbalanced design in which four treatments are applied to four naturally assembled pools of subjects.

– For example, consider a study of the effect of different training methods on learning. Subjects are placed into four groups (A,B,C, D) for different training methods, X1-X4.

Group A X1O X2O X3O X4OGroup B X2O X4O X1O X3OGroup C X3O X1O X4O X2OGroup D X4O X3O X2O X1O

Quasi-Experimental DesignsQuasi-Experimental Designs Design 12: The Separate Sample Pretest-Posttest Design: Often

used with large populations (i.e., in public opinion studies) where the researcher cannot randomize or segregate subgroups for different treatments. Two equivalent groups are identified, one sample is measured prior to the treatment and a different (but equivalent) sample is measured after the treatment. This design is also called the "simulated before and after" design.

– For example, 100 community members are randomly surveyed concerning their opinions about local government policies. A PR campaign is then conducted for six weeks. A follow-up survey is then conducted with 100 different residents who are randomly selected.

R O XR X O

Design 12: The Separate Sample Pretest-Posttest Design: Often used with large populations (i.e., in public opinion studies) where the researcher cannot randomize or segregate subgroups for different treatments. Two equivalent groups are identified, one sample is measured prior to the treatment and a different (but equivalent) sample is measured after the treatment. This design is also called the "simulated before and after" design.

– For example, 100 community members are randomly surveyed concerning their opinions about local government policies. A PR campaign is then conducted for six weeks. A follow-up survey is then conducted with 100 different residents who are randomly selected.

R O XR X O

Quasi-Experimental DesignsQuasi-Experimental Designs Design 13: The Separate Sample Pretest-Posttest

Control Group Design: This design is similar to Design 12; however, a control group is added to the design.– For example, consider the PR campaign described in Design

12. In this case, the same design is used, but, in addition, the measurements are made in a similar nearby city where no PR campaign is run.

R O XR X O

R O R O

Design 13: The Separate Sample Pretest-Posttest Control Group Design: This design is similar to Design 12; however, a control group is added to the design.– For example, consider the PR campaign described in Design

12. In this case, the same design is used, but, in addition, the measurements are made in a similar nearby city where no PR campaign is run.

R O XR X O

R O R O

Quasi-Experimental DesignsQuasi-Experimental Designs Design 15: Recurrent Institutional Cycle Design (A "Patched-Up" Design):

This is an approach used in field research. A researcher begins with an inadequate design and then adds features to control for one or more sources of invalidity. The result is an "inelegant accumulation of precautionary checks." The researcher is aware of rival interpretations (sources of internal invalidity) and incrementally identifies other data that would rule out rivals. The design exploits contextual features to refine the research as it progresses.

– For example, this design would combine a longitudinal and cross sectional structure. One group will be exposed to X and measured at the same time as a second group that is just about to be exposed to X. A comparison of the two groups would be able to be made because it is equivalent to a static group comparison. The second group would be remeasured (posttest), which would make the design comparable to the one group pretest-posttest design.

Group A X O1

Group B O1 X O2

Design 15: Recurrent Institutional Cycle Design (A "Patched-Up" Design): This is an approach used in field research. A researcher begins with an inadequate design and then adds features to control for one or more sources of invalidity. The result is an "inelegant accumulation of precautionary checks." The researcher is aware of rival interpretations (sources of internal invalidity) and incrementally identifies other data that would rule out rivals. The design exploits contextual features to refine the research as it progresses.

– For example, this design would combine a longitudinal and cross sectional structure. One group will be exposed to X and measured at the same time as a second group that is just about to be exposed to X. A comparison of the two groups would be able to be made because it is equivalent to a static group comparison. The second group would be remeasured (posttest), which would make the design comparable to the one group pretest-posttest design.

Group A X O1

Group B O1 X O2

My Research AgendaMy Research Agenda

So, what type of research approach do you think I use?

So, what type of research approach do you think I use?

General Research ThemesGeneral Research Themes

Geographic Information Systems (GIS) and Location Intelligence– Studies of the use of GIS as a decision

support tool– The use of GIS in Businesses and

Organizations– Location intelligence and the use of

location in decision making– Perceptions of space and geography

Geographic Information Systems (GIS) and Location Intelligence– Studies of the use of GIS as a decision

support tool– The use of GIS in Businesses and

Organizations– Location intelligence and the use of

location in decision making– Perceptions of space and geography

General Research ThemesGeneral Research Themes

Studies of Teams, Collaboration, and technology– Virtual Teams– Team History– Individual Characteristics

Studies of Teams, Collaboration, and technology– Virtual Teams– Team History– Individual Characteristics

General Research ThemesGeneral Research Themes

Virtual Worlds– The application of VW to education and

learning– Perceptions of avatars, space and location

in VWs– Legal, tax, and social issues in VWs– Communication and collaboration in VWs

Virtual Worlds– The application of VW to education and

learning– Perceptions of avatars, space and location

in VWs– Legal, tax, and social issues in VWs– Communication and collaboration in VWs

General Research ThemesGeneral Research Themes

Mobile Commerce, Computing, and Virtual Teams– Mobile Device Interfaces– Impressions of Mobile Device Users– Applications of Mobile Devices

Mobile Commerce, Computing, and Virtual Teams– Mobile Device Interfaces– Impressions of Mobile Device Users– Applications of Mobile Devices

General Research ThemesGeneral Research Themes

Applications of Conjoint to IS Research– Human Resources– Information Systems Analysis– IT Planning

Applications of Conjoint to IS Research– Human Resources– Information Systems Analysis– IT Planning

General Research ThemesGeneral Research Themes

IT Adoption and Implementation– User Acceptance of Mobile Devices– The Use of Mobile Devices in Commerce

IT Adoption and Implementation– User Acceptance of Mobile Devices– The Use of Mobile Devices in Commerce

General Research ThemesGeneral Research Themes

The Application of IT for Training and Learning– The Application of Technology in

Education– The Role of Communication Technology in

Learning

The Application of IT for Training and Learning– The Application of Technology in

Education– The Role of Communication Technology in

Learning

A Recent StudyA Recent Study

Question: What is the impact of video conference technology and training methodology on student learning

IV: – Training Mode:

• Enactive Mastery• Vicarious Experience

– Communication Media• Face to Face• Video Conferencing

Question: What is the impact of video conference technology and training methodology on student learning

IV: – Training Mode:

• Enactive Mastery• Vicarious Experience

– Communication Media• Face to Face• Video Conferencing

ResultsResults

6

7

8

9

10

11

12

FTF DVC

VE

EM

ResultsResults

Tests of Between-Subjects Effects

Dependent Variable: DQ - Total

2765.596a 5 553.119 11.046 .000

112.075 1 112.075 2.238 .137

339.197 1 339.197 6.774 .010

1506.172 1 1506.172 30.079 .000

198.404 1 198.404 3.962 .048

75.519 1 75.519 1.508 .221

232.381 1 232.381 4.641 .033

7060.445 141 50.074

53232.000 147

9826.041 146

SourceCorrected Model

Intercept

MIDGSETO

PFTSCORE

MEDIA

TRAINING

MEDIA * TRAINING

Error

Total

Corrected Total

Type III Sumof Squares df Mean Square F Sig.

R Squared = .281 (Adjusted R Squared = .256)a.