Slide 1

Dr Chen WenliLearning Sciences and Technologies AGLearning Sciences LabNational Institute of EducationQuantitative Research Methods(II) OutlineLogic of quantitative researchConstructing hypothesisTypes of quantitative research methodsSurvey researchExperimental researchSingle-subject researchCasual-comparative researchQuantitative content analysisValidity and reliability in quantitative research

Experimental ResearchCharacteristics of experimental research

Experimental research designExperimental designQuasi-experimental designFactorial design

Validity of experimental researchControl of extraneous variables

Experimental ResearchResearcher applies some treatments to subjects for an appropriate length of time and then observes the effect of the treatments on the subjects by measuring response variablesIV (experimental or treatment variable)a condition or set of conditions applied to subjectsDV (response, criterion or outcome Variable)results or outcome on the subjects

(to see whether the treatment made a difference).IV (experimental or treatment variable): a condition or set of conditions applied to subjectse.g., methods of instruction, types of assignment, learning materials, rewards given to students, and types of questions asked by teachersDV (Response, criterion or outcome Variable): results or outcome on the subjects e.g. achievement, interest in a subject, attention span, motivation, and attitudes towards learning/school

ExamplesQuality of learning with an active versus passive motivational set (Benware & Deci, 1984)Comparison of computer-assisted cooperative, competitive, and individualistic learning (Johnson, Johnson, & Stanne, 1986)The effect of a computer simulation activity versus a hands-on activity on product creativity in technology education (Kurt, 2001)The effect of language course taught with online supplement material (Shimazu, 2005) CharacteristicsThe only type of research that directly attempts to influence a particular variableThe only type that, when used properly, can really test hypotheses about cause-and-effect relationships.Enable researchers to go beyond description and the identification of relationships, to at least a partial determination of what causes them3 characteristics of experimental research

Manipulation of IVResearcher manipulate the IVDecide the nature of treatment/intervention (what is going to happen to the subjects of the study)To whom it is to be appliedTo what extentWhen, where and how

Comparison of GroupsAt least 2 conditions are compared to assess the effect(s) of particular conditions or treatments (IV)Experimental group (receive a treatment of some sort) Control group (no treatment) or comparison group (receive different treatment)IV may be established in several ways:Presence VS absence of a particular formOne form of variable VS another Varying degrees of the same formPresence VS absence of a particular forme.g., use of computer VS no computer in teaching mathematicsOne form of variable VS another e.g., inquiry method VS lecture method of instructionVarying degrees of the same formDifferent amounts of teacher enthusiasm on students attitude

RandomizationRandom assignment of subjects to groupsan important ingredient in the best kinds of experiments every individual who is participating in the experiment has an equal chance of being assigned to any of the experimental or control conditions being comparedIt takes place before the experiment beginsAllows the researcher to form groups that are equivalentEliminate the threat of extraneous, or additional variables that might affect the outcome of the study

Random selection and random assignment :Distinguish between selection and assignmentBoth help to ensure that groups are equivalent and to control for extraneous variablesIf you incorporate random selection and random assignmentCommonly Used Notation X1=treatment group X2=control/comparison group O=observation (pretest, posttest, etc.) R=random assignmentWeak Experimental DesignsOne-shot case study designa single group is exposed to a treatment or event, and its effects assessed.

One-group pretest-posttest designa single group is measured or observed both before and after exposure to a treatment.

X OTechnologyAttitude scale to measure interest

O X O Pretest Treatment Post testThey are considered weak because they do not have built in controls for threats to internal validity. When a study lacks internal validity, one or more alternative hypotheses exist to explain the outcomes of the study. These alternative hypotheses are referred to by researchers as "threats to internal validity." When a study has internal validity, it means that any relationship observed between two or more variables is unambiguous as to what it means, rather than being due to something else.

only one group; therefore, no random assignmentno control group (for comparison purposes); pretest sensitization; experimenter effects (cannot be sure about conclusions)

True Experimental DesignsRandomized posttest-only control group designinvolves two groups formed by random assignment and receiving different treatments

Randomized pretest-posttest control group designdiffers from the randomized posttest-only control group only in the use of a pretest

Treatment group R O X1 O Control group R O X2 OTreatment group R X1 OControl group R X2 OThese designs do have at least some controls built into the design to control for threats to internal validity

True Experimental DesignsRandomized Solomon four-group designinvolves random assignment of subjects to four groups, with two being pretested and two not.

Treatment group ROX1O Control group ROX2O Treatment group RX1O Control group RX2OBetter control the threat to internal validityDrawbackrequires twice as many participantsQuasi-Experimental DesignsUsed in place of experimental research when random assignment to groups is not feasiblePosttest-only design with nonequivalent groups

Pretest-posttest design with nonequivalent groups:

Treatment group O X1 OControl group O X2 O

Treatment group X1 OControl groupX2 OUses two groups from same populationQuestions must be addressed regarding equivalency of groups prior to introduction of treatment

Quasi-Experimental DesignsCounterbalanced design: all groups are exposed to all treatments, but in a different orderthe order in which the groups receive the treatments should be determined randomlythe number of groups and treatments must be equalComparing the average scores fro all groups on the posttest for each treatment

Group IX1 O X2 O X3 O Group II X3 O X1 O X2 O Group III X2 O X3 O X1 OQuasi-Experimental DesignsTime-series design: involves repeated measurements or observations over time (until scores are stable ), both before and after treatment.

O O O O X O O O OUses a single group of participantsExamines possible changes over time

Study BStudy AXFactorial DesignsFactorial designs extend the number of relationships that may be examined in an experimental study.

Treatment ROX1g1OControlROX2g1 OTreatment ROX1g2 OControl ROX2g2 OIncorporates two or more factors The additional factor could be treatment variable or subject characteristicsEnables researcher to detect differential differences (effects apparent only on certain combinations of levels of IVs)

e.g., two types of factors (e.g., method of instruction) each of which has two levels (e.g., traditional vs. innovative)

A 2 X 2 factorial designBoyGirlTraditionalGame-based learningGroup 1Group 2Group 3Group 4A 2 X 2 factorial designNo interaction between factorsGame -basedTraditionalInteracting factorsBoyGirlBoyAttitudes toward learningGirlAttitudes toward learningTraditionalGame -basedValidityValidity: the experiment tests the variable(s) that it purports to testIf threats are not controlled for, they may introduce error into the study, which will lead to misleading conclusionsThreats to validityInternal: factors other than the IV that affect the DVExternal: factors that affect the generalizability of the study to groups and settings beyond those of the experiment

Both experimental and quasi-experimental research are subject to threats to validity

Threats to Internal ValidityHistory Uncontrolled event that occur during the study that may have an influence on the observed effect other than the IV MaturationFactors that influence a participant's performance because of time passing rather than specific incidents (e.g., the physical, intellectual, and emotional changes that occur naturally)Test practiceThe effects of participants taking a test that influence how they score on a subsequent testInstrumentationInfluences on scores due to calibration changes in any instrument that is used to measure participant performanceStatistical regressionProblem that occurs when participants have been assigned to particular group on the basis of atypical or incorrect scores.

The selection of people for a study may result in the individuals or groups differing (i.e., the characteristics of the subjects may differ) from one another in unintended ways that are related to the variables to be studied. No matter how carefully the subjects of a study (the sample) are selected, it is common to lose some of them as the study progresses. This is known as "mortality." Such a loss of subjects may affect the outcomes of a study. The particular locations in which data are collected, or in which an intervention is carried out, may create alternative explanations for any results that are obtained. The way in which instruments are used may also constitute a threat to the internal validity of a study. Possible instrumentation threats include changes in the instrument, characteristics of the data collector(s), and/or bias on the part of the data collectors. The use of pretest in intervention studies sometimes may create a "practice effect" that can affect the results of a study. A pretest can also sometimes affect the way subjects respond to all intervention. On occasion, one or more unanticipated, and unplanned for, events may occur during the course of a study that can affect the responses of subjects. This is known as a history threat. Sometimes change during an intervention study may be due more to factors associated with the passing of time than to the intervention itself. This is known as a maturation threat. The attitude of subjects toward a study (and their participation in it) can create a threat to internal validity. When subjects are given increased attention and recognition because they are participating in a study, their responses may be affected. This is known as the Hawthorne effect. Whenever a group is selected because of unusually high or low performance on a pretest, it will, on the average, score closer to the mean on subsequent testing, regardless of what transpires in the meantime. This is called a regression threat. Whenever an experimental group is treated in ways that are unintended and not a necessary part of the method being studied, an implementation threat can occur.Controlling Threats to Internal ValidityThere are a number of techniques or procedures that researchers can use to control or minimize threats to internal validity. Essentially they boil down to four alternatives: (1) standardizing the conditions under which the study occurs; (2) obtaining and using more information on the subjects of the study; (3) obtaining and using more information on the details of the study; and (4) choosing an appropriate design.

Threats to Internal ValidityBias in group compositionSystematic differences between the composition of groups in addition to the treatment under study. Experimental mortalityA differential loss of participantsHawthorne effectChange in the sensitivity or performance by the participants that may occur merely as a function of being a part of the studyNovelty effectParticipant interest, motivation, or engagement increases simply because they are doing something differentPlacebo effectThe participants receive no treatment but believe they are

The selection of people for a study may result in the individuals or groups differing (i.e., the characteristics of the subjects may differ) from one another in unintended ways that are related to the variables to be studied. No matter how carefully the subjects of a study (the sample) are selected, it is common to lose some of them as the study progresses. This is known as "mortality." Such a loss of subjects may affect the outcomes of a study. The particular locations in which data are collected, or in which an intervention is carried out, may create alternative explanations for any results that are obtained. The way in which instruments are used may also constitute a threat to the internal validity of a study. Possible instrumentation threats include changes in the instrument, characteristics of the data collector(s), and/or bias on the part of the data collectors. The use of pretest in intervention studies sometimes may create a "practice effect" that can affect the results of a study. A pretest can also sometimes affect the way subjects respond to all intervention. On occasion, one or more unanticipated, and unplanned for, events may occur during the course of a study that can affect the responses of subjects. This is known as a history threat. Sometimes change during an intervention study may be due more to factors associated with the passing of time than to the intervention itself. This is known as a maturation threat. The attitude of subjects toward a study (and their participation in it) can create a threat to internal validity. When subjects are given increased attention and recognition because they are participating in a study, their responses may be affected. This is known as the Hawthorne effect. Whenever a group is selected because of unusually high or low performance on a pretest, it will, on the average, score closer to the mean on subsequent testing, regardless of what transpires in the meantime. This is called a regression threat. Whenever an experimental group is treated in ways that are unintended and not a necessary part of the method being studied, an implementation threat can occur.Controlling Threats to Internal ValidityThere are a number of techniques or procedures that researchers can use to control or minimize threats to internal validity. Essentially they boil down to four alternatives: (1) standardizing the conditions under which the study occurs; (2) obtaining and using more information on the subjects of the study; (3) obtaining and using more information on the details of the study; and (4) choosing an appropriate design.

Threats to External ValidityPopulation-sample differencesThe degree to which the participants in a study are representative of the population to which generalization is desiredArtificial research arrangementsThe degree that a research setting deviates from the participant's usual routineMultiple-treatment interferenceMore than one treatment is administered to the same participants and results in cumulative effects that may not be similar to the outside world and may threaten generalization of the resultsTreatment diffusionThe situation when different treatment groups communicate with and learn from each other

Validity of Different Experimental DesignsPre-Test/Post TestControlGroupRandomi-zationAdditionalGroupsHistoryXMaturationXPre-TestingXMeasuring InstrumentXStatistical RegressionXXDifferential SelectionXXExperimental MortalityXInteraction of FactorsXXPre-TestingXDifferential SelectionXXProceduresXMultiple TreatmentControl of Extraneous VariablesConfounding:the fact that the effects of the IV may intertwine with extraneous variables, such that it is difficult to determine the unique effects of each variableCommon ways to control for extraneous variables RandomizationHolding certain variables constantMatchingComparing homogeneous groups or subgroupsAnalysis of covariance (ANCOVA)

The researcher in an experimental study has an opportunity to exercise far more control than in most other forms of research. Randomization: all individuals in the defined population have an equal and independent chance of being selected Holding certain variables constant: to eliminate the possible effects of a variable by removing it from the studyMatching: pairs of subjects can be matched on certain variables of interestComparing homogeneous groups or subgroups: comparing groups that are similar with respect to that variable Analysis of covariance (ANCOVA)

Single-Subject ResearchMost commonly used to study the changes in behavior an individual exhibits after exposure to a treatment or intervention of some sort. Can be applied in settings where group designs are difficult to put into play.Involves extensive collection of data on one subject at a time. Primarily use line graphs to present their data and to illustrate the effects of a particular intervention or treatment. Adaptations of the basic time-series design

E.g., study children who suffer from multiple handicaps

Single-Subject ResearchA-B design baseline measurements (O) are repeatedly made until stability is established, then the treatment (X) is introduced and an appropriate number of measurements (O) are made during treatment implementation

O O O X O X O X O baseline treatment phase phase A | BSingle-Subject ResearchReversal (A-B-A) designbaseline measurements (O) are repeatedly made until stability is established, then the treatment (X) is introduced and an appropriate number of measurements (O) are made during treatment implementation, followed by an appropriate number of baseline measurements (O) to determine stability of treatment (X)

O O O X O X O X O O Obaseline treatment baseline phase phase phaseA | B | AOther Single-Subject Research DesignsA-B-A-B design2wo baseline periods are combined with two treatment periodsB-A-B designUsed when an individual's behavior is so severe or disturbing that a researcher cannot wait for a baseline to be establishedA-B-C-B design:"C" condition refers to a variation of the intervention in the "B" condition. The intervention is changed during the "C" phase typically to control for any extra attention the subject may have received during the "B" phase.

e.g., effect of praise on a particularly nonresponsive JC student during instruction in mathematicsThreats to Validity in Single Subject ResearchInternal Validitylength of the baseline and intervention conditionsthe number of variables changed when moving from one condition to anotherthe degree and speed of any change that occurswhether or not the behavior returns to baseline levelsthe independence of behaviorsthe number of baselinesExternal Validityweak when it comes to generalizabilityIt is important to replicate single-subject studies to determine whether they are worthy of generalization.

Controlling Threats in Single-Subject StudiesSingle-subject designs are most effective in controlling for subject characteristics, mortality testing, and history threats. They are less effective with location, data collector characteristics, maturation, and regression threats. They are especially weak when it comes to instrument decay, data-collector bias, attitude, and implementation threats.

Causal-Comparative ResearchExplores the possibility of cause-and-effect relationships when experimental and quasi-experimental approaches are not feasibleDiffers from experimental and quasi-experimental researchIV is not manipulated (not ethical or not possible)Focuses first on the effect, then tries to determine possibleRelationships can be identified in causal-comparative study, but causation cannot be fully established.E.g., females have a greater amount of linguistics ability than males , students who were taught by the inquiry method are more critical of information from the Internet than are those who were taught by the lecture method.

Steps in Causal-Comparative ResearchFormulating a problem Identify and define the particular phenomena of interest, and then to consider possible causes for, or consequences of, these phenomena. Selecting a sampleDefine carefully the characteristic to be studied and then to select groups that differ in this characteristic. InstrumentationNo limits to the kinds of instruments that can be usedDesignSelect two groups that differ on a particular variable of interest and then comparing them on another variable or variables.Threats to Internal Validity in Causal-Comparative ResearchWeaknesses :lack of randomizationInability to manipulate an IV A major threat: the possibility of a subject selection bias. The procedures used to reduce this threatmatching subjects on a related variablecreating homogeneous subgroupsthe technique of statistical matching. Other threats to internal validityLocationInstrumentationLoss of subjects. Data Analysis in Causal-Comparative StudiesThe first step: construct frequency polygons. Means and SD are usually calculated if the variables involved are quantitative. The most commonly used test is a t-test for differences between means. ANCOVA is particularly useful in causal-comparative studies. The results of causal-comparative studies should always be interpreted with caution, because they do not prove cause and effect.Common quantitative measure in learning and education Learning gainPost-pre (post-pre)/(1-pre) (Hakes gain)Adjusted post score (through ANCOVA)Learning efficacyDoes it help reduce time spent for problem solving?Users attitudeTeachbacksHow well learner can teach back?

Quantitative Content AnalysisContent analysis is a quantitative research instrument for a systematical and intersubjective description of contentA form of textual analysis *usually*Categorizes chunks of text according to CodeBased on the principles of social science of measuring and countingReduces the complexity of content as it brings out the central patterns of the coverageOne objective is to examine large amounts of content with statistic methodsRough HistoryClassical Content AnalysisUsed as early as the 30s in military intelligenceAnalyzed items such as communist propaganda, and military speeches for themesCreated matrices searching for the number of occurrences of particular words/phrases(New) Content AnalysisMoved into Social Science ResearchStudy trends in Media, Politics, and provides method for analyzing open ended questionsCan include visual documents as well as textsMore of a focus on phrasal/categorical entities than simple word counting

Procedure

The SampleThe sampleWhich types of content?Which period?Which characteristics?Elements of the research instrumentSampling unitsUnits of analysis: unit of the content on which our measurements are based. The categories describe the properties of the media content which is relevant to our research questionValidity in Quantitative ResearchDefinition: the extent to which any measuring instrument measures what it is intended to measureTypes of validityConstruct Validity: examines the fit between the conceptual definitions & operational definitions of the variablesContent Validity : verifies that the method of measurement actually measures the expected outcomes. Predictive Validity : determines the effectiveness of the instrument as a predictor of a future eventStatistical Conclusion Validity: concerned with whether the conclusions about relationships and/or differences drawn from statistical analysis are an accurate reflection of the real world

Reliability in Quantitative ResearchDefinition: refers to the accuracy and consistency of information obtained in a study; important in interpreting the results of statistical analyses; and refers to the probability that the same results would be obtained with different samples (generalizability)3 common methods to check reliabilitytest-retest methodadministering the same instrument twice to the same group of individuals after a certain time interval has elapsed. equivalent-forms method administering two different, but equivalent, forms of an instrument to the same group of individuals at the same time. internal-consistency methodcomparing responses to different sets of items that are part of an instrument.

When reliability and validity are achieved, data are free from systematic errors

SummaryLogic of quantitative researchConstructing hypothesisTypes of quantitative research methodsSurvey researchExperimental researchSingle-subject researchCasual-comparative researchOthersValidity and reliability in quantitative research

Thank You