dr chen wenli learning sciences and technologies ag learning sciences lab national institute of...

Dr Chen Wenli Learning Sciences and Technologies AG Learning Sciences Lab National Institute of Education Quantitative Research Methods (II)

Outline Logic of quantitative research Constructing hypothesis Types of quantitative research methods Survey research Experimental research Single-subject research Casual-comparative research Quantitative content analysis Validity and reliability in quantitative research

Experimental Research Characteristics of experimental research Experimental research design Experimental design Quasi-experimental design Factorial design Validity of experimental research Control of extraneous variables

Experimental Research Researcher applies some treatments to subjects for an appropriate length of time and then observes the effect of the treatments on the subjects by measuring response variables IV (experimental or treatment variable) a condition or set of conditions applied to subjects DV (response, criterion or outcome Variable) results or outcome on the subjects

Examples Quality of learning with an active versus passive motivational set (Benware & Deci, 1984) Comparison of computer-assisted cooperative, competitive, and individualistic learning (Johnson, Johnson, & Stanne, 1986) The effect of a computer simulation activity versus a hands-on activity on product creativity in technology education (Kurt, 2001) The effect of language course taught with online supplement material (Shimazu, 2005)

Characteristics The only type of research that directly attempts to influence a particular variable The only type that, when used properly, can really test hypotheses about cause-and-effect relationships. Enable researchers to go beyond description and the identification of relationships, to at least a partial determination of what causes them 3 characteristics of experimental research

Manipulation of IV Researcher manipulate the IV Decide the nature of treatment/intervention (what is going to happen to the subjects of the study) To whom it is to be applied To what extent When, where and how

Comparison of Groups At least 2 conditions are compared to assess the effect(s) of particular conditions or treatments (IV) Experimental group (receive a treatment of some sort) Control group (no treatment) or comparison group (receive different treatment) IV may be established in several ways: Presence VS absence of a particular form One form of variable VS another Varying degrees of the same form

Randomization Random assignment of subjects to groups an important ingredient in the best kinds of experiments every individual who is participating in the experiment has an equal chance of being assigned to any of the experimental or control conditions being compared It takes place before the experiment begins Allows the researcher to form groups that are equivalent Eliminate the threat of extraneous, or additional variables that might affect the outcome of the study

Commonly Used Notation X 1 =treatment group X 2 =control/comparison group O=observation (pretest, posttest, etc.) R=random assignment

Weak Experimental Designs One-shot case study design a single group is exposed to a treatment or event, and its effects assessed. One-group pretest-posttest design a single group is measured or observed both before and after exposure to a treatment. X O TechnologyAttitude scale to measure interest O X O Pretest Treatment Post test

True Experimental Designs Randomized posttest-only control group design involves two groups formed by random assignment and receiving different treatments Randomized pretest-posttest control group design differs from the randomized posttest-only control group only in the use of a pretest Treatment group R O X 1 O Control group R O X 2 O Treatment group R X 1 O Control group R X 2 O

True Experimental Designs Randomized Solomon four-group design involves random assignment of subjects to four groups, with two being pretested and two not. Treatment group ROX 1 O Control group ROX 2 O Treatment group RX 1 O Control group RX 2 O Better control the threat to internal validity Drawbackrequires twice as many participants

Quasi-Experimental Designs Used in place of experimental research when random assignment to groups is not feasible Posttest-only design with nonequivalent groups Pretest-posttest design with nonequivalent groups: Treatment group O X 1 O Control group O X 2 O Treatment group X 1 O Control groupX 2 O

Quasi-Experimental Designs Counterbalanced design: all groups are exposed to all treatments, but in a different order the order in which the groups receive the treatments should be determined randomly the number of groups and treatments must be equal Comparing the average scores fro all groups on the posttest for each treatment Group IX 1 O X 2 O X 3 O Group II X 3 O X 1 O X 2 O Group III X 2 O X 3 O X 1 O

Quasi-Experimental Designs Time-series design: involves repeated measurements or observations over time (until scores are stable ), both before and after treatment. O O O O X O O O O Uses a single group of participants Examines possible changes over time Study B Study A X

Factorial Designs Factorial designs extend the number of relationships that may be examined in an experimental study. Treatment ROX 1 O ControlROX 2 O Treatment ROX 1 O Control ROX 2 O Incorporates two or more factors The additional factor could be treatment variable or subject characteristics Enables researcher to detect differential differences (effects apparent only on certain combinations of levels of IVs)

A 2 X 2 factorial design BoyGirl Traditional Game- based learning Group 1 Group 2 Group 3Group 4

A 2 X 2 factorial design No interaction between factors Game -based Traditional Interacting factors BoyGirlBoy Attitudes toward learning Girl Attitudes toward learning Traditional Game -based

Validity Validity: the experiment tests the variable(s) that it purports to test If threats are not controlled for, they may introduce error into the study, which will lead to misleading conclusions Threats to validity Internal Internal: factors other than the IV that affect the DV External External: factors that affect the generalizability of the study to groups and settings beyond those of the experiment

Threats to Internal Validity History Uncontrolled event that occur during the study that may have an influence on the observed effect other than the IV Maturation Factors that influence a participant's performance because of time passing rather than specific incidents (e.g., the physical, intellectual, and emotional changes that occur naturally) Test practice The effects of participants taking a test that influence how they score on a subsequent test Instrumentation Influences on scores due to calibration changes in any instrument that is used to measure participant performance Statistical regression Problem that occurs when participants have been assigned to particular group on the basis of atypical or incorrect scores.

Threats to Internal Validity Bias in group composition Systematic differences between the composition of groups in addition to the treatment under study. Experimental mortality A differential loss of participants Hawthorne effect Change in the sensitivity or performance by the participants that may occur merely as a function of being a part of the study Novelty effect Participant interest, motivation, or engagement increases simply because they are doing something different Placebo effect The participants receive no treatment but believe they are

Threats to External Validity Population-sample differences The degree to which the participants in a study are representative of the population to which generalization is desired Artificial research arrangements The degree that a research setting deviates from the participant's usual routine Multiple-treatment interference More than one treatment is administered to the same participants and results in cumulative effects that may not be similar to the outside world and may threaten generalization of the results Treatment diffusion The situation when different treatment groups communicate with and learn from each other

Validity of Different Experimental Designs

Control of Extraneous Variables Confounding: the fact that the effects of the IV may intertwine with extraneous variables, such that it is difficult to determine the unique effects of each variable Common ways to control for extraneous variables Randomization Holding certain variables constant Matching Comparing homogeneous groups or subgroups Analysis of covariance (ANCOVA)

Single-Subject Research Most commonly used to study the changes in behavior an individual exhibits after exposure to a treatment or intervention of some sort. Can be applied in settings where group designs are difficult to put into play. Involves extensive collection of data on one subject at a time. Primarily use line graphs to present their data and to illustrate the effects of a particular intervention or treatment. Adaptations of the basic time-series design

Single-Subject Research A-B design baseline measurements (O) are repeatedly made until stability is established, then the treatment (X) is introduced and an appropriate number of measurements (O) are made during treatment implementation O O O X O X O X O baseline treatment phase phase A | B

Single-Subject Research Reversal (A-B-A) design baseline measurements (O) are repeatedly made until stability is established, then the treatment (X) is introduced and an appropriate number of measurements (O) are made during treatment implementation, followed by an appropriate number of baseline measurements (O) to determine stability of treatment (X) O O O X O X O X O O O baseline treatment baseline phase phase phase A | B | A

Other Single-Subject Research Designs A-B-A-B design 2wo baseline periods are combined with two treatment periods B-A-B design Used when an individual's behavior is so severe or disturbing that a researcher cannot wait for a baseline to be established A-B-C-B design: "C" condition refers to a variation of the intervention in the "B" condition. The intervention is changed during the "C" phase typically to control for any extra attention the subject may have received during the "B" phase.

Threats to Validity in Single Subject Research Internal Validity length of the baseline and intervention conditions the number of variables changed when moving from one condition to another the degree and speed of any change that occurs whether or not the behavior returns to baseline levels the independence of behaviors the number of baselines External Validity weak when it comes to generalizability It is important to replicate single-subject studies to determine whether they are worthy of generalization.

Controlling Threats in Single-Subject Studies Single-subject designs are most effective in controlling for subject characteristics, mortality testing, and history threats. They are less effective with location, data collector characteristics, maturation, and regression threats. They are especially weak when it comes to instrument decay, data-collector bias, attitude, and implementation threats.

Causal-Comparative Research Explores the possibility of cause-and-effect relationships when experimental and quasi-experimental approaches are not feasible Differs from experimental and quasi-experimental research IV is not manipulated (not ethical or not possible) Focuses first on the effect, then tries to determine possible Relationships can be identified in causal-comparative study, but causation cannot be fully established.

Steps in Causal-Comparative Research Formulating a problem Identify and define the particular phenomena of interest, and then to consider possible causes for, or consequences of, these phenomena. Selecting a sample Define carefully the characteristic to be studied and then to select groups that differ in this characteristic. Instrumentation No limits to the kinds of instruments that can be used Design Select two groups that differ on a particular variable of interest and then comparing them on another variable or variables.

Threats to Internal Validity in Causal-Comparative Research Weaknesses : lack of randomization Inability to manipulate an IV A major threat: the possibility of a subject selection bias. The procedures used to reduce this threat matching subjects on a related variable creating homogeneous subgroups the technique of statistical matching. Other threats to internal validity Location Instrumentation Loss of subjects.

Data Analysis in Causal- Comparative Studies The first step: construct frequency polygons. Means and SD are usually calculated if the variables involved are quantitative. The most commonly used test is a t-test for differences between means. ANCOVA is particularly useful in causal-comparative studies. The results of causal-comparative studies should always be interpreted with caution, because they do not prove cause and effect.

Common quantitative measure in learning and education Learning gain Post-pre (post-pre)/(1-pre) (Hakes gain) Adjusted post score (through ANCOVA) Learning efficacy Does it help reduce time spent for problem solving? Users attitude Teachbacks How well learner can teach back?

Quantitative Content Analysis Content analysis is a quantitative research instrument for a systematical and intersubjective description of content A form of textual analysis *usually* Categorizes chunks of text according to Code Based on the principles of social science of measuring and counting Reduces the complexity of content as it brings out the central patterns of the coverage One objective is to examine large amounts of content with statistic methods

Rough History Classical Content Analysis Used as early as the 30s in military intelligence Analyzed items such as communist propaganda, and military speeches for themes Created matrices searching for the number of occurrences of particular words/phrases (New) Content Analysis Moved into Social Science Research Study trends in Media, Politics, and provides method for analyzing open ended questions Can include visual documents as well as texts More of a focus on phrasal/categorical entities than simple word counting

Procedure

The Sample The sample Which types of content? Which period? Which characteristics? Elements of the research instrument Sampling units Units of analysis: unit of the content on which our measurements are based. The categories describe the properties of the media content which is relevant to our research question

Validity in Quantitative Research Definition: the extent to which any measuring instrument measures what it is intended to measure Types of validity Construct Validity: examines the fit between the conceptual definitions & operational definitions of the variables Content Validity : verifies that the method of measurement actually measures the expected outcomes. Predictive Validity : determines the effectiveness of the instrument as a predictor of a future event Statistical Conclusion Validity: concerned with whether the conclusions about relationships and/or differences drawn from statistical analysis are an accurate reflection of the real world

Reliability in Quantitative Research Definition: refers to the accuracy and consistency of information obtained in a study; important in interpreting the results of statistical analyses; and refers to the probability that the same results would be obtained with different samples (generalizability) 3 common methods to check reliability test-retest method administering the same instrument twice to the same group of individuals after a certain time interval has elapsed. equivalent-forms method administering two different, but equivalent, forms of an instrument to the same group of individuals at the same time. internal-consistency method comparing responses to different sets of items that are part of an instrument.

Summary Logic of quantitative research Constructing hypothesis Types of quantitative research methods Survey research Experimental research Single-subject research Casual-comparative research Others Validity and reliability in quantitative research

dr chen wenli learning sciences and technologies ag learning sciences lab national institute of...

Documents