1 Dr. Chang

Lecture Notes for EDM6401, Quantitative Methods in Educational ResearchChang Lei, Ph.D., Professor

1. An Overview of Educational Research

Four Ways of Knowing

Methods of TenacityTruth is true because one believes it even in front of contradicting evidence.

Superstition.

Method of AuthorityTruth is true because an authority says so. Religion.

Method of IntuitionTruth is true because it is logical. It derives from reasoning but does not bear

empirical support. Philosophy.

Method of ScienceScience is a method of seeking truth. This method only accounts for solvable

problems that have empirical solutions based on observable events. Some major components of the scientific method include:

empirical evidence vs. refusal of contradicting evidencerandom sampling vs. isolated or selected eventscountering rival explanations replication and public inquiryTruth is routinely challenged and tested and retested by the public. There is no truth

but a temporary state where it has not been disproved or rejected as untrue.

The "Low" Status of Social SciencesDifficult to replicate

The more developed a discipline, the higher the probability of detection and replication.

Value judgementScience is considered "pure" and free from value judgement.

Too "obvious"People tend to be uninterested in questioning what is already "obvious" to them. Nor

do people want to be bothered with what they do not know.Thorndike: "That is the fate of educational research. If it comes out the way people

thought it should, they ask, 'What is the point?' If startling conclusions emerge, people say, 'I do not believe it'."

Norms of ScienceUniversal Standards

The quality of research is judged by universal standards regardless of the experience, race, sex, affiliation, or the characteristics of the researcher. e.g., Blind review process.

2 Dr. Chang

Common Ownership of InformationScientific information is not proprietary but is owned and freely shared by all;

publication is not only a right but an obligation of a researcher. e.g., Data are to be shared on request. "Publish or perish" enforces this norm.

Integrity in Gathering and Interpreting DataThe researcher displays disinterestedness and impersonal tone when gathering data or

presenting a point of view. e.g., Cite the researcher's own name or use "the author" to minimize personability.

Organized ScepticismIt is the responsibility of the community of scientists to be skeptical of each new

knowledge claim, to test it, to try to think of reasons the claim might be false, to think of alternative explanations. This challenge to new knowledge is sought in science, e.g., conference debate, validation study.

The Roles and Outcomes of ResearchExploratory

Discover new phenomena and relationships among phenomena that are missed by others. Qualitative research plays important role here.

e.g., A counselling psychologist wants to know what things make an effective counsellor.

ExplanatoryDevelop new theories or use existing theories to account for the observations.e.g., Dollard and Dobb (1939) theorized that frustration leads to aggression from the

observation that a child strikes out when deprived of a toy.According to social learning theory, a good role model is important for school

achievement.

ValidationValidating and replicating existing research and theory is an important part of science.

Using different samples, populations, research methods.

Three Components of Educational Research Methodology

1. MEASUREMENT (PSYCHOMETRICS) Instrumentation, Reliability, Validity

2. RESEARCH DESIGN Sampling, Designs, Internal and External Validity

Experiment, quasi-experiment, non-experiment research.

3. DATA ANALYSIS (STATISTICS) Descriptive statistics, Hypothesis testing, and Various analytical techniques

3 Dr. Chang

The Process of Scientific Inquiry1. Identification of a research problem. (why)

2. Consult the literature for a solution. (find out why)

3. Formulation of testable hypotheses on the basis of existing theory and/or experience. (a tentative solution)

4. Design a study with efforts to minimize extraneous factors that may contribute to the same phenomenon or relationship you hypothesized. (design a study)

5. Data collection. When the behaviour of subjects are measured or observed, the measurement or observations become empirical data. (carry out the study)

6. Data analysis. Data are summarized in such a way that the summary bears on the research questions and hypotheses. Statistics are used to generalize from sample to population.

(report the findings)

7. Interpretation of data, adding to the existing body of knowledge.(this is why)

Research Report:* Title Page* Abstract* Introduction

ProblemSignificanceJustificationsHypotheses which are integrated in the literature review

* Methodpopulationsampleproceduresmeasurementdesigns

* ResultsPresent the results in the order of the hypotheses

* Discussion* References* Tables and Figures

The Process of Research: 1. INTRODUCTION SECTION *****1. Identification of a research problem. Objectives and

(Why) Significance of study

4 Dr. Chang

2. Consult the literature for a solution. Literature review(Find out why)

3. Formulation of testable hypotheses on Research questions the basis of existing theory and research. and hypotheses

(Here is a solution) Ind and Dep variables

2. METHOD SECTION *****4.Design a study to minimize extranous Sample, Design (exp

factors that affect the same phenomenon quasi, non), Procedureor relationship you hypothesized. Validity threats(A plan to test the solution)

5.Data collection. When the behaviours Measurements,are experimentally manipulated or Reliability andobserved, the outcome become data. Validity(Carry out the plan.)

3. RESULTS SECTION *****6.Data analysis. Data are summarized in ANOVA vs. Regression

such a way that the summary bears on framework, Sig testthe research questions and hypotheses. Confidence interval(Report it)

4. DISCUSSION SECTION *****7.Interpretation of data, adding to Theory & explanation

the existing body of knowledge. Limitations and(Why? This is why.) Future directions

Ways to Locate a Research Problem1.Identify broad areas that are closely related to your interests and professional goal and write them down.

2.Then choose among the areas that relate to your future career, an area or a research topic that is feasible.

3.Collaborate with other people; join on-going projects.

4.Read text books where rather comprehensive topics in a field are summarized, and problems and future research needs are identified; journal article for the state of art of the field and authors recommendations; reviews articles for both.

5.Test a theory.

6.Replication. Replicate major milestone study. Replicate studies using different population, samples, methods.

7.Observations. Observe carefully the existing practices in your area of interest.

8.Develop research ideas from advanced courses you take.

5 Dr. Chang

9.Get ideas from newspaper and popular magazines.

Variable and ConstantAn attribute or characteristic of a person or object that varies from person to person,

object to object. A constant is an attribute that does not vary from person to person. Student achievement, motivation, blood pressure, etc. Pi, in relation to a particular population, age is a constant for the 6th graders, religion for parochial schools, gender for male prisoners, etc. When you ask a research question, you ask about variables, you want to know the relationship among variables. Why don't students learn, is it because they are in poor health, not motivated, distracted by family problems, by crime, the teachers are not qualified, etc? You end up with a question regarding the relationship among variables. Is there a relationship between motivation and achievement.

Independent variable represents the research interest and is manipulated (experiment) or measured (non-experiment) to see the results of its change on the dependent variable.

Dependent variable is the observed outcome in response to the independent variable. It is used to evaluate the independent variable.

Control variable is a variable that is either made into a constant or is included in the study (even though it is not of interest) to control or neutralize factors extraneous to the research question.

Operational DefinitionsAssign meaning to a construct or a variable by specifying the activities or "operations"

necessary to MEASURE or MANIPULATE it.Redefine a concept in terms of clearly observable operations that anyone can see and

repeat. These observable and replicable operations can take the form of an experiment or of a measurement instrument.

As Cronbach and Meehl (1955) point out, it is rare for a construct to receive one commonly endorsed operational definition. To some researchers, hunger is defined in an animal experiment as "amount of time since last feeding." It may also be defined by others as "amount of energy an animal would expend to seek food." Thus, it is important to be operationally clear about a particular construct, so that other researchers understand, for example, what the construct "hunger" is intended to mean.

Measured operational definition (more often used): Intelligence is defined as scores on the Woodcock-Johnson Test of Cognitive

Abilities.Vagueness of lecturing is defined as using the following words: A couple, a few,

sometimes, all of this, something like that, pretty much.School achievement is defined as one's GPA.Social Economic Status is defined by the number of years of education and the

amount of salary the head of a family receives.Popularity is defined operationally by the number of friendship nominations a student

receives from his/her school mates.

Experimental operational definition:Recall is defined by asking subjects to recite items shown to them from a stimulus list

and assigning a point for each item that matches one on the list.

6 Dr. Chang

Recognition is defined by showing subjects items and asking them to decide whether they were part of the stimulus list.

Aggression is defined as the number of times a child hits a toy doll after watching a violent TV show.Distinguishing Between Two General Types of Literature Reviews

There are two general types of literature reviews, each possessing unique as well as common characteristics. Making the distinction prior to embarking on the review is important to both your own mental health and the quality of the product. The two types are:

1.A critical review of a literature

2.A review of literature relevant to a research proposal

Following are some is and is not for each type:

Critical Review of a LiteratureIs a place where you may review the body of literature that bears on a problematic area -- or even to examine all the research that relates to the specific question raised in a research proposal.

Is an activity the product of which is devoted to critical retrospective on scholarship -- and publishable in such journals as the Psychological Bulletin, Psychological Review.

Is not encumbered with supporting the conceptual framework of a proposed study or of justifying study design and methodology decisions.

Review of Literature Relevant to a Research ProposalIs an obligation to place the question or hypothesis in the context of previous work in such a way as to explain and justify the decisions made.

Is a product that reflects a step-by-step explanation of decisions, punctuated by references to studies that support the conceptual framework and ongoing argument.

Is not a product to educate the reader concerning the state of science in the problem area; nor is it to display the thoroughness with which the author pursued a comprehensive understanding of the literature.

How to Write a Lit ReviewBreak up the review into several topic areas.

Organize all the findings under the various topics into a unified picture of the state of knowledge in the area reviewed. The process of combining and interpreting the literature is more difficult than merely reviewing what has been done.

Use two to three studies that are most pertinent and well done as foundations of your review topics. Use similar studies as support.

Write the review as if you are expressing your own thoughts and developing and

7 Dr. Chang

building your own arguments and themes but not as if you are reporting others' work.

Don't do article by article listing of things.

Don't use the same format, e.g., Baker found...

Rather than citing everything in an article in one place, cite an article multiple times to fit different themes of yours.

Write down your thoughts and paraphrase important points of the articles as you read. It may not be a good idea to read all the articles and then write.

Look over the articles before copying them. Read several carefully before looking for more.

2: Hypothesis Testing1.The hypothesis should state an expected relationship between two or more variables.

2.The researcher should have definite reasons based on either theory or evidence for considering the hypothesis worthy of testing.

3.Hypothesis should be testable. The relationship or difference that is stated in a hypothesis should be such that measurement of the variables involved can be made and necessary statistical comparisons carried out in order to determine whether the hypothesis as stated is or is not supported by the research.

4.The hypothesis should be as brief as possible.

There is a gender difference in the perception of body sensations.Women and men use physiological cues (internal) and situational factors (external)

differently in defining bodily state.Women, compared to men, make greater use of external cues in defining their body

sensations.There is a relationship between information processing techniques and subsequent

recall of information.Visual imagery has a greater enhancing effect on recall than verbal recitation.People tend to apply dispositional attribution to account for behaviours of others and

use situational attribution to explain behaviours of themselves.Induced self-consciousness enhances recall of personal information.Teachers who use specific feedback during lectures obtain higher pupil achievement

gains than teachers who use general feedback.High intimacy self-disclosing statements would be more effective in counselling than

low intimacy self-disclosing statements.

Hypothesis TestingA hypothesis is always about a population.Testing a hypothesis means drawing inference from a random sample to the

population where the sample is taken.

8 Dr. Chang

1. Research hypothesis reflecting your verbal reasoning. The wording often reflects the research design.

e.g., There is a relationship between motivation to learn and math achievement.Girls have higher math achievement than boys.The effect of induced public self-consciousness is stronger adolescents than adults.

2. Statistical hypothesis reflecting the statistics used to summarize your observations.

e.g., ρ > 0: There is a positive correlation between motivation to learn and math achievement. The statistic of correlation is used to summarize data.

μg > μb: Mean math achievement of girls is higher than the mean of boys. Mean is used to summarize data.

3. Null hypothesis representing a way to test the statistical hypothesis.μg = μb. The mean math achievement of girls is the same as the mean of boys.ρ = 0. There is no correlation between motivation to learn and math achievement.

4. Statistical tests are conducted with the assumption that the null hypothesis is true. What is the probability of finding a positive correlation when the truth is there is no

correlation?What is the probability of finding a difference between the two means when there is

no difference?

Statistical SignificanceThe probability level at which you will reject the null hypothesis, or, at which you will

allow yourself the risk of wrongly rejecting the null hypothesis.

Type I ErrorSignificance level is also Type I error rate. It is the probability of rejecting the null

hypothesis when the null hypothesis is true. You make such an error only when the null is rejected.

Type II ErrorIt is the probability of not rejecting the null hypothesis when the null hypothesis is

false. You make such an error only when you fail to reject the null hypothesis.

Sampling distribution of means (or any statistic)Is an imagined or theoretical distribution of an infinite number of means computed

from random samples of the same size. Because of the central limit theorem, this distribution is used as a probability distribution to determine the probability of obtaining a mean larger than or as large as (in absolute value) the one computed from your sample.

Central limit theorem1. If repeated random samples of size n are drawn from a normally distributed

population, the distribution of the sample means is normal.2. As the sample size increases, disregarding the shape of the population distribution,

the sampling distribution of means approximates normality.

9 Dr. Chang

3. The mean of the sampling distribution of means equals the population mean.4. The standard deviation of the sampling distribution of means equals the population

standard deviation divided by the square root of sample size. This is called standard error of means.

If population variance is not known, sample variance can be used as an estimate of population variance in computing the standard error.

Four steps in hypothesis testing: Example1: State the null and H0: μ1-μ2 = 0 alternative hypotheses. H1: μ1-μ2 > 0

2: Set the level of statistical significance α .05 which is the probability at which you'll t (.05, 28)=1.7 reject the null or at which you'll allow yourself to make the type I error.

3: Compute the test statistic which can be t (28) = 2.85 a t-test, z-test, f-test, chi-square, etc.

4: Decision about the null. If you reject the Reject null null, you may make a type I error the and support probability of which is set at step 2. If your research you do not reject null, you are running (alternative) the risk of making a type II error the hypothesis. probability of which can be calculated if you know certain parameters.

An ExampleHypothesis:

People high in public self-consciousness are more conforming to perceived social norms on gender roles (than those low in public self-consciousness).

There is a relationship between public self-consciousness and gender role conformity.

Independent variable is public self-consciousness.

Dependent variable is gender role conformity.

Operational definitions:Public self-consciousness is measured by the Self-Consciousness Scale (Fenigstein,

Scheier, & Buss, 1975; Scheier & Carver, 1985).

Gender role conformity is defined by the following operations: Ten gender role attitudes questions were used to first determine participants' own standings on these gender role questions. The participants were then informed of the mean ratings of their peers on these gender role questions and were asked to re-assess their attitudes toward these gender roles. Conformity to social norms on gender roles is measured by the difference score between the two self-assessment on the ten gender roles questions.

10 Dr. Chang

Statistical hypothesis:μhigh public - μ2high private > 0

This implies that the statistic, mean, is used to summarize sampled data that bear on the hypothesis.or

ρ > 0, implying that the statistic, correlation, is used to summarize data.

Null hypothesis:μhigh public - μ2high private = 0

orρ = 0

Significance level: α <= .05

Hypothesis testing rationale:The hypothesis is regarding population.The null assumes that there is no mean difference between the two populations

(groups).My hypothesis assumes that there is a mean difference (in the direction hypothesized)

between the two populations.The purpose of hypothesis testing is to make the qualitative decision regarding

whether my samples are taken from the populations defined by the null (decision: accept null and your research hypothesis is not supported) or are taken from the populations defined by the alternative (research) hypothesis (decision: reject null and your research hypothesis is supported).

The hypothesis testing starts with the assumption that the null is true. Even though the null is true, there is a good chance that, due to sampling fluctuation, you will find some small magnitudes of difference in your samples. The chance for you to find large differences, however, should be very small. In fact, such chance is so small that you should no longer attribute the difference to sampling fluctuations but to the possibility that the null is not true. This chance is the probability associated with your computed sample statistic. As this probability gets smaller, you grow more doubtful about the truth of the null to the point that you make the qualitative decision that the null is not true (reject the null). This point is your significance level and your decision is associated with the possibility of a type I error.

Another example:Induced state of public self-consciousness increases gender role conformity.State of public self-consciousness is the independent variable. It is induced by video

taping the participants while they are assessing their gender role attitudes in relation to these attitudes of their peers. The knowledge that they are on camera induces the state of public self-consciousness.

3: Measurement and TestingReliability

Classical theory also known as true score theory is mostly concerned with test reliability or the reliability of observed scores of a test in measuring the underlying true abilities or true scores. Reliability can be defined as the strength of the relationship between observed scores and true scores.

11 Dr. Chang

If we were to administer a test to the same person under all different conditions at different times using different items, there would be different observed scores. The mean of all these observed scores is the person's true score, or true ability or personality.

In reality, we only give the person one test and there is only one observed score. This score can be seen as a random variable or as randomly sampled observation from a distribution of all possible observed scores. The observed score can be seen as consisting of the mean of the distribution (or the true score) and a deviation from the mean which is called error or error score. Thus, x = t + e.

The extent to which an observed score represent the true score is reliability. We can use Pearson product moment correlation, ρ, to describe the strength of the relationship between observed scores and true scores, i.e., reliability. Thus, ρxt is called a reliability index. (Note ρxt is not reliability coefficient.) Of course, we don't know the true score and thus, can not solve for ρxt. But assumptions can be made that enable the solving of ρxt. These assumptions, which are not discussed here, make up the classical test theory.

With the assumptions, we can numerically estimate the reliability of a test without knowing its true scores. First, we define the square of reliability, numerically, as the ratio between true score variance and observed score variance. This is called the reliability coefficient. (Note this is ρxx or ρxx'.) Second, we can estimate the reliability coefficient by simply correlating two parallel tests (or two forms of a test, two halves of a test, or two administrations of a test). The result is a Pearson correlation coefficient, r, which is an estimate of the reliability coefficient. Keep in mind, the meaning of a Pearson r when used as an estimate of reliability coefficient, is really r2 representing the proportion of the observed score variance that is true score variance. r ranges from -1 to 1 whereas r2 ranges from 0 to 1. That is why, r as a reliability estimate, ranges from 0 to 1 but not -1 to 1.

Depending on which kinds of two tests are being correlated to arrive at the reliability estimates, these estimates are given different names as shown below:

Test-retest reliability (coefficient of stability)Correlate two administrations of the same test.

Parallel form reliability (coefficient of equivalence)Correlate two forms of the same test

Split half reliability (Spearman-Brown prophecy formula)Correlate two halves of the test

Internal consistency reliability (Cronbach α)Correlate every item with every other item.

When ρxx' = 1,1. the measurement has been made without error (e=0 for all examinees).2. X = T for all examinees.3. all observed score variance reflects true-score variance.4. all difference between observed scores are true score differences.5. the correlation between observed scores and true scores is 1.6. the correlation between observed scores and errors is zero.

When ρxx’ = 0, 1. only random error is included in the measurement.

12 Dr. Chang

2. X = E for all examinees. 3. all observed score variance reflects error variance. 4. all difference between observed scores are errors of measurement. 5. the correlation between observed scores and true scores is 0. 6. the correlation between the observed scores and errors is 1.

When ρxx’ is between zero and 1, 1. the measurement include some error and some truth. 2. X = T + E.3. observed score variance include true-score and error variance. 4. difference between scores reflect true-score differences and error.5. the correlation between observed scores and true scores is reliability. 6. the correlation between observed scores and error is the square root of 1 – reliability.

ValidityThe validity of the use of a test refers to the extent to which the test truly measures

what it is expected to measure. For example, the use of a bathroom scale to measure weight is valid whereas the use of a bathroom scale to measure height is invalid. Commonly discussed validities include content, construct, and predictive validity.

Kinds of Validity EvidenceContent validity refers to the extent to which the items on a test are representative of a

specified domain content. For example, a test that is intended to measure the content of this course should contain items about reliability, validity, intelligence and personality tests. If this test is made up of items on calculus or matrix algebra, the test will have no content validity. Achievement and aptitude (but not personality and attitude) tests are concerned with content validity.

Construct validity refers to the extent to which items on a test are representative of the underlying construct, e.g., personality or attribute. Personality and attitude tests are concerned with construct validity. The process to establish construct validity is referred to as construct validation. Construct validation is complicated, involving testing hypotheses concerning the theories from which the test is derived. A common practice is to compare high scorers with low scorers on the test with respect to some external behavior which is hypothesized to correlate with the test.

Construct validity has often been narrowly interpreted as providing evidence for the internal structure of a test. Another term for this narrow definition of construct validity is factorial validity because such validity evidence is gathered through factor analysis. The correct definition of construct validity refers to gathering evidence for a broad normological network of relations.

In 1959, Campbell and Fiske published an important paper, which has also become the most cited paper in psychology. In this paper, they conceptualize validity issues in a Multitrait-Multimethod (MTMM) correlation matrix. In this MTMM matrix, convergent validity (CV) is the correlation between the different methods of the same trait. This correlation coefficient should be high. Discriminant validity (DV) is the correlation between different traits obtained by the same method. A low value indicates evidence of validity. The correlation between different traits obtained by different methods (HH, heterotrait-heteromethod) should be the lowest in value. This MTMM concept has since become an important part of construct validity, both in terms of its narrow definition and the broad

13 Dr. Chang

definition.Criterion related validity, including predictive validity and concurrent validity, refers

to the extent to which a test correlates with future behaviors which the test is intended to predict. For example, the HKALE is intended to select students who are capable of university studies. For the HKALE to have predictive validity, its test scores should correlate with undergraduate performance, such as the GPA. Predictive validity is simply the correlation between the test and a criterion measure the test is intended to predict. The correlation coefficient is also called validity coefficient. Sometimes, because of lack of a criterion, one correlates the test with another test that perports to measure the same thing. This is called concurrent validity.

Making Sense of ValidityValidation is an ongoing process where one keeps accumulating validity evidence. But

one is not capable of obtaining all the evidence at once. Certain evidence is never attempted. For different uses of a test, the gathering of some evidence becomes more important than others, giving rise to different validity concepts and procedures. Over time, this discriminating use of evidence in relation to different types of tests becomes a tradition so that certain test and its use is routinely associated with one kind of validity evidence but not with other kinds. For example, most education achievement tests are only concerned with content validity; i.e., whether the test items are representative of what have been taught within a specified domain content. One purpose of education is to create a competent work force and, thus, a valid test of such education achievement should be correlated with future job performance. However, such criterion-related validity evidence is seldom gathered for an education achievement test in part because there is also a strong public mentality that students must be assessed for what they have learned. As an another example, the content validity of a personality test is never questioned. The validity concern with a personality test lies in establishing the linkage between the test items and the underlying trait structure as defined by a theory within which the test is conceptualized. Factor analysis is often used to see if the items form clusters according to the theory and to see if the items correlate with measures of other constructs according to specified patterns of relationships defined by the theory. These efforts are referred to as construct validation.

To be simplistic, content validity is associated with an achievement test as construct validity is with a personality and attitude test. The former is concerned with the representativeness of the items with respect to a specified domain content. The content validation procedures are qualitative or judgmental. The latter is concerned with the representativeness of the items with respect to a defined theoretical construct. The procedures are referred to as construct validation which may involve different data collection techniques and strategies, some of which could be thought of as a different validity procedure, e.g., criterion related validity. Finally, some tests do not need validity evidence. For example, a test of typing which is administered to measure the status of an individual's typing skills is its own criterion. Such "obviously valid" tests do not need additional validity evidence.

An important issue associated with criterion validity is what is referred to as the restriction of range effect. Validity coefficient is a correlation coefficient the magnitude of which depends on the ranges of scores for the predictor variable (test) and the criterion (a future behavior to be predicted by the test). A correlation based on the full range of scores will always be higher than that based on a restricted range of scores, independent from the true predictability of the test. In reality, most of such tests are used to make selections (of students or employees for example). The validity study is always conducted on those who

14 Dr. Chang

passed the test and were, thus, given the opportunity to demonstrate the future performance. Thus, the validity is based on a restricted range of scores rather than the full range. The real predictive validity of a test should always be higher than what is obtained from a validity study. There are, however, statistical procedures to adjust the validity coefficient.

Another important issue is the relationship between validity and reliability. Remember that ρxx' = ρ2

xt, the reliability coefficient is the squared correlation between the observed scores and the true scores. If ρxx' = .81, ρxt = .90, (the correlation between the observed scores and true scores is .90). In general, ρxt > ρxx'. That is an observed score will correlate higher with its own true score than with an observed score on a parallel test. Because a test can not correlate more highly with any other test or variable than with its own true score, the maximum correlation between an observed score and another variable is the squared coefficient itself.

Sometimes, the distinction between reliability and validity is blurred. For instance, Lindquist (1942) defined validity as the correlation between the fallible (the test to be validated) and infallible measure of a trait. Assuming the infallible criterion to be perfectly reliable and the (fallible) test to be perfectly representative of the criterion, the maximum validity of the (fallible) test would have to be the correlation of the observed test scores with true scores on the test itself which is the square root of the reliability of the fallible test. This is basically the criterion related validity -- correlation between the observed scores of the test and observed scores of the criterion.

4: Research Validity and Sampling TechniquesResearch Validity

Research validity can be simply understood as the quality of a research study. There are two kinds of quality issues that are referred to as internal validity and external validity. When we try to pinpoint the cause of a phenomenon or behavior either by experimentally manipulating the independent variable or by measuring or observing the independent variable, we are trying to demonstrate that it is the independent variable that “causes” changes in the outcome variable or in the dependent variable. Internal validity is about the extent to which we can make this causal inference. Internal validity is the extent to which the outcomes of a study result from the variables which were manipulated, measured, or selected in the study rather than from other variables not systematically treated. e.g., there might not be a relationship between televised violence and aggressive behavior, rather, children who watch violence programs are more aggressive in the first place. Different brands of deodorants were tested under left and right arm which provide unequal testing conditions. An expectancy effect will also make people feel the advocated brand lasts longer. In other words, internal validity is about how confident we are about the stated “causal” relationship between the independent variables and the dependent variable. That is, the dependent variable is due to independent variable but not due to something else. To improve internal validity and thus the quality of a research study, we need to be able to rule out alternative “causes” that may have done the same thing to the dependent variable as does our independent variable. Some of the commonly encountered alternative causes include:

History: Events take place during the study that might affect its outcome in the same way that the independent variable is hypothesized to affect the outcome. e.g., a study examined whether a certain leadership training program was effective in enhancing students’ sense of competitiveness; in the three months of the experimental study, the TV show, Survivor, was on which could have made the students more aware of competition and become more experienced with some competition strategies.

15 Dr. Chang

Maturation: Especially for developmental studies where children grow with the passage of time to become more mature in certain developmentally related abilities. e.g., a study showing students’ vocabulary increase from a dialectical reading program may not have internal validity due to maturation because children’s vocabulary increases with time independent of the training program.

Testing: When people are measured repeatedly, e.g., pretest-posttest, they become better not because of the independent variable but because they become test smarter. e.g., a study showing math scores improved over the pretest due to a new teaching method might not have internal validity because students work much faster or better because of their experience with the pretest.

Instrumentation: The effect on the dependent variable is not due to the independent variable but due to aspects of the instrument used in the study. Like the testing threat, this one only operates in the pretest-posttest situation. e.g., observed change from pretest to posttest is due not to a math program that is being experimented but rather to a change in the test that was used. The posttest could simply be easier. Instrumentation threats are especially likely when the "instrument" is a human observer. The observers may get tired over time or bored with the observations and thus become more lenient or more stringent or more conservative or liberal in their ratings. Conversely, they might get better at making the observations as they practice more and become more accurate than their ratings at the beginning. In either event, it is the change in instrumentation, not the independent variable, that leads to the observation in the dependent variable.

Regression towards the mean: Particularly problematic when subjects are chosen because of extreme scores where high scoring individuals are more likely to score lower and low scoring individuals are likely to score higher the next time they are tested merely due to random measurement error. e.g., a study showing a teacher is effective in improving the scores of students who are at the bottom in the first term may not have internal validity due to the regression towards the mean artifact because by chance alone these students at the bottom will tend to increase (there is a greater chance for top students to fall towards the mean; while bottom students have a greater chance to rise towards the mean).

Selection: Results due to assignment to different treatment or control groups but not due to the independent variable that makes the two groups. e.g., you want to compare a new coaching method against the existing method used in PE lessons and recruit volunteers to participate in three days of training of the sport. You then compare the results with those of some of the existing PE lessons. Volunteers are simply more motivated than those attending regular PE class and thus may produce better training outcome (dependent variable) which has little to do with training method (independent variable).

Mortality: Especially for longitudinal studies that last for an extended period of time, attrition or dropping off from the study in a non-random manner may affect the outcome of the study. That is, some participants no longer want to continue with the study and the results are based on those who stayed in the study who have are different from those who dropped in some fundamental ways. e.g., a special intervention may be tried out in a school but the results showed that their students’ average HKCE results to be much worse than those from

16 Dr. Chang

regular schools; this study may not have internal validity due to mortality because the weakest students may have dropped out from the regular schools.

Diffusion or imitation of treatment: The control group or one of the treatment groups somehow end up receiving some of the same treatment as the other groups resulting in few differences among the treatments. e.g., a study shows that a new teaching method (tried out in one class) does not lead to better achievement than the traditional method (in another class) in part because students from the two classes constantly compare notes and exchange information.

Hawthorne effect refers to the fact that when participants received unusal treatment in a field experiment, they may temporarily change their behavior or performance not because of the manipulation of the independent variable but because of the special attention they received during the experimentation. The term gets its name from a factory called the Hawthorne Works, where a series of experiments on factory workers were carried out between 1924 and 1932. Among many types of experiments, one stands out and is often talked about is the illumination experiment where researchers came to the factory to change lights all the time (and probably made casual conversations with the workers). In comparison to the control group which did not experience lighting changes, productivity of the experiemental group seemed to increase independent of how lighting was adjusted.

John Henry effect: Whereas the Hawthorne effect is due to some unusal performance of the experiemental group, sometimes the control group may also put up some extraordinary performance to outperform the experimental group due to a sense of demoralization for not being included in the “special treatment” experimental group. The term comes from a true story of a railroad worker, John Henry, who probably felt threatened when a spiking machine (to put spikes to stablize the rails) was introduced to replace the manual work and outperformed the machine but later died of heart attack.

External ValidityExternal validity is the extent to which the findings of a particular study can be

generalized to people or situations other than those observed in the study. Can findings from laboratory be applied to real world where there are many other factors influencing behavior that have been controlled in the lab. All the factors threatening internal validity can be controlled in a lab but the results may become less generalizable outside the lab. e.g., a study on a new teaching method with a certain group of S.4 students does not have external validity if the results cannot be generalized to other topics taught by the same new method, or to other S.4 students. Many threats to external validity can be understood in the form of an interaction:

Treatment-attribute interaction: Certain personality and other characteristics may interact with the independent variable so that the effect of the independent variable may be different on people having different personality characteristics. e.g., A study showing democratic parenting style improves academic achievement may lack external validity if we can argue and show that less parental demand and supervision only work with highly motivated students but not other students.

Treatment-setting interaction: The independent variable may interact with other external factors or contexts to result in different effects for different settings so that the effect cannot be generalized to all settings. e.g., The positive effect of democratic parenting on school achievement may lack external validity if democratic parenting (independent variable)

17 Dr. Chang

works only when schools provide clear structure and strict supervision regarding student learning. Pretest sensitization: The effect of the independent variable may be due to the pretest which serves to sensitize the subjects whereas in the population (the real world to which the findings are to be generalized), there is no pretest and thus the treatment (independent variable) may not work as it does in the study. A study showing that a math teaching method worked because the post test improved over the pretest may lack external validity if the pretest helped the teachers and students to identify learning difficulties or made them more aware of certain weaknesses and the teaching method focused on those weaknesses.

Posttest sensitization: The effect of the independent variable is part due to sensitization or exercising effect of the posttest which is not available in the population to which the results are to be generalized. e.g., in the above study on a new math teaching method, the posttest served to reinforced the effect of the teaching method and thus the students in the study improved their math performance because they had both the new teaching method and the posttest.

Sampling TechniquesAn element is an object on which a measurement is taken. It is not the person or thing

but a particular measurement of the person or thing that is of interest. e.g., persons' height, class size.

A population is all the elements in a defined set about which we wish to make an inference. Examples of target population vs. experimentally accessible population are heights of Chinese vs. heights of Shanghai residents.

Sampling units are non-overlapping collections of elements from the population. The sampling unit can be the element.

Sampling frame is a list of sampling units.A sample is a collection of sampling units drawn from a frame.

Simple random sampleIf a sample of size n is drawn from a population of size N in such a way that every

possible sample of size n has the same chance of being selected.M = Σxi / n is the sample estimate of population mean, μ.

Stratified random samplingThe population of N units is divided into subpopulations of N1, N2... Nh units which

are non-overlapping so that N1 + N2 +..+ Nh = N. The subpopulations are called strata. A sample is drawn from each stratum. The sample is denoted as n1, n2, nh. If a simple random sample is taken from each stratum, the whole procedure is called stratified random sampling.

When the population is heterogeneous, stratified sampling increases the precision in estimating population parameters. Breaking up the population makes each stratum homogeneous (measurement varies little among units) so that a small sample is needed to estimate the population characteristics of the stratum. These strata estimates can be combined into a precise estimate of the whole population.

Wh = Nh / N is stratum weight.

fh = nh / Nh is sampling fraction in the stratum.

Mst = Σ NhXh / N

18 Dr. Chang

= Σ WhXh is the stratified sample estimate of population mean, μ. If in every stratum, nh / n = Nh / N, this means sampling fraction is the same in all

strata. Such stratification is called stratification with proportional allocation of the nh. Using proportional allocation, Mst = Σ nhXh / n

NXst = N1X1 + N2X2 + N3X3 .. + NhXh is stratified sample estimate of population total, τ.

Cluster samplingA cluster sample is a simple random sample in which each sampling unit is a

collection, or cluster, of elements. It is used when 1) a good frame listing population elements either is unavailable, unreliable, or costly; 2) the cost of obtaining observations increases as the distance separating the elements increases. For example, when sampling in the field in agricultural research, it is hard to do random sampling by running around. In quality inspection of light bulbs contained in boxes, light bulbs are the elements and the boxes can be the sampling units. Travelling within a city (to obtain a simple random sample of city residents) is more expensive than travelling in a city block (to get a cluster sample of city blocks as sampling units). The rational for choosing the unit size for cluster sampling is a decision on the unit that gives smaller sampling variance for a given cost or on the cost for a prescribed variance. Used as a general rule, the number of elements within a cluster should be small relative to the population size, and the number of clusters in the sample should be reasonably large.

Mcl = Σxi / Σmi is the cluster sample estimate of population mean, μ, where Σxi is the total of observations (sum of elements) in the ith cluster, and Σxi indicates summing over i=1 to n sampled cluster totals; mi is the size of ith cluster, and Σmi indicates summing over i=1 to n sampled cluster sizes,.

Systematic samplingRandomly selecting one element from the first k elements in the frame and every kth

element thereafter is called a one-in-k systematic sample. It is needed in field study, e.g., select every 10th tree or every 20th file, every 15th shopper who passes by an aisle in a supermarket, until a predetermined n is achieved. If different persons originally handled different sections of the files, or each clerk deals with a surname, then systematic sampling provides more accurate information than simple random sampling which could, by random chance, select n files filed by one clerk. Parameter estimations are the same as those of the simple random sampling.

Sample sizeThe method used to select the sample is of utmost importance in judging the validity

of the inference made from the sample to the population. The representativeness of the sample is more important than the size of the sample. A representative sample of 100 may be preferable to an unrepresentative sample of 100,000. The size of a sample can never compensate for a lack of representativeness (bias).

Having established sample representativeness, using the right sample size becomes an important economical decision. Each observation taken from the population contains a certain amount of information about the population parameter or parameters of interest. Since obtaining the information costs financially, one decides how much information to be sampled. Too little prevents good estimates. Too much may be a waste given a limited economic resource. The quality of information obtained in a sample depends upon the number of elements sampled (sample size) and the amount of variation in the information (population

19 Dr. Chang

variance). Let's look at an example.What proportion of people are left-handed? How big a random sample is needed to

answer the question? Or, in other words, how big an error, or margin of error, is to be tolerated? Let's set the margin of error at no bigger than 10%. If the sample estimate is 20%, you will at least be confident that the population proportion is between 10% and 30%. But you cannot guarantee that every sample, including the one you draw, will have this margin of error unless the whole population is sampled. Sometimes, a sample may have sampling error higher than 10%. Sometimes, a sample may have sampling error lower than 10%. Of course, you would be concerned only with having an error higher than 10%. Then the question becomes how unlikely do you want this unlucky sample having a higher than 10% error to be? Assume you want the unlikelihood to be 5 out 100 samples so that you can be confident, 95% of the times, that a sample does not exceed the specified sampling error of 10%. That is Pr (p - P > 10%) = 5% or Pr (p - P 5) = 95%. You can then use some of the basic statistics to estimate the margin of error by first estimating the population variance (of proportion of left-handers) and standard deviation which is a function of sample size so that you can estimate how big a sample you need so that the chance of making an estimation error of bigger than 10% does not exceed 5%.

5: Experimental, Quasi-Experimental and Non-Experimental DesignsTo ensure research validity or internal validity, researchers develop various ways to

identify, isolate, or nullify variability among subjects in a dependent variable that is presumably "caused" by one or more independent variables that are extraneous to the particular relation or relations under study. Such a research effort is called control, control of variance, or control of extraneous variables. The most powerful way of controlling extraneous variables is experimentation where subjects are randomly assigned to experimental versus control groups. Other things being equal, if random assignment has been used, the groups can be assumed to be equal in all possible characteristics except that due to the manipulation of the independent variable. In other words, variations among subjects due to anything other than the independent variable are scattered or spread out evenly across the randomly assigned groups. The variability due to the manipulation of the independent variable is called systematic variance. The purpose of experimental research is to maximize this source of variance, minimize error variance, and control extraneous variance. Other means of controlling extraneous variance include matching, including the extraneous variable into the study, making the extraneous variable a constant, and using statistical methods to decompose different sources of variance. It is the extent to which extraneous variables are controlled that distinguishes research designs into experimental, quasi-experimental, and non-experimental designs.

Experimental designThe single most important feature of experimental research is the manipulation of the

independent variable. Researchers create changes, called treatment conditions, in variables being researched, called independent variables, to examine the impact of these manipulations on some outcome behaviour or phenomena called the dependent variable. Another important feature of experimental research is the ability to control extraneous variables so that subjects receiving different manipulations of the independent variable are equal except for the manipulation.

Experimental research in social science is developed from both physics research and biological research models. In physics research the primary means of controlling extraneous

20 Dr. Chang

factors is through artificial controls, such as isolation, insulation, sterilization, strong steel chamber walls, soundproofing, lead shielding, etc. These methods ensure the reproduction of similar conditions and the consequent production of certain effects. Biological research moved from the laboratory to the open field, the modern theory of experimental control through randomized assignment to treatment emerges. Agricultural research compares yield per acre for different crops or fertilizers, raking, or plowing method. One of the greatest breakthroughs in experimental design was the realization that random assignment provided a means of comparing the outcomes of different treatments in a manner that ruled out most alternative interpretations. Random assignment requires experimental units, which can be plots of land in agriculture, individual persons in social psychology experiments, intact classrooms in education studies, and neighborhoods in some criminal justice research. Treatments are then assigned to these units by some equivalent of a coin toss.

Quasi-Experimental DesignQuasi-experiments have treatments, outcome measures, and experimental units, but do

not use random assignment to create the comparisons from which treatment-caused change is inferred. Instead, the comparisons depend on nonequivalent groups that differ from each other in many ways other than the presence of a treatment whose effects are being tested. The task is one of separating the effects of a treatment from those due to the initial noncomparability between the average units in each treatment group. In a sense, quasi-experiments require making explicit the irrelevant causal forces hidden within the intact groups. The advantages of experimental control for inferring causation have to be weighed against the disadvantages that arise because we do not always want to learn about causation in controlled settings. Instead, we would like to be able to generalize to causal relationships in complex field settings, and we cannot easily assume that findings from the laboratory will hold in the field.

For lack of randomization, pretest is an integral part of quasi-experiment that enables comparisons among the nonequivalent groups, whereas, in most experiments, a pretest is often unnecessary or undesirable. When a pre-test is not available, researchers may look for some proxy variables to use in an attempt to equate the treatment and control groups. For instance previous academic records can be used as proxies to equate the intact classes where different instructional methods are implemented and post-tests are used to evaluate or compare the different teaching methods.

Non-Experimental or Ex Post Facto ResearchThere is a temporal sequence in the order of the independent variable (which occurs

first) and dependent variable (which is observed subsequently) that allows a causal inference. Specifically, the researcher manipulates the independent variable to "create" changes in the dependent variable. This expected change (in the dependent variable as a consequence of the manipulation of the independent variable) represents a causal relationship between the two variables. This causal relationship is stated in the hypothesis. Thus, an experimental research is guided by a hypothesis stated a priori.

In experimental and quasi-experimental research, inferences are made from the independent variables (the causes) to the dependent variable (the effect). In non-experimental research, also called "ex post facto research", inferences are generally made in the opposite direction. That is beginning with the observation of the dependent variable, attempts are made to uncover, detect, or find the reasons (independent variable) for the existing variations. The

21 Dr. Chang

point is the variations are not the result of the manipulation of the independent variables but are pre-existing. When an experimental researcher manipulates a variable (e.g., administers different treatments), the researcher has some expectations regarding the effect of the manipulation on the dependent variable. These expectations are expressed in the form of hypotheses to be tested. In non-experimental research, a researcher would not have an expectation of the independent variables or sometimes would not even know prior to data collection what "independent" variables are tenable to explain the pre-existing variations on the "dependent" variable. Often, there are no hypotheses associated with a non-experimental study and researchers adopt the position of "letting the data speak for themselves."

This design is most vulnerable to internal validity threats. Two general strategies to protect internal validity are using (1) large samples to compensate for the lack of random assignment and (2) large numbers of "independent" variables to eliminate rival explanations. The latter strategy is intended to overcome the weakness of having no experimental manipulation of the independent variables.

Summary of Research Designs

Manipulation of Random Sample Variable Design Independent V. Assignment Size Number--------------------------------------------------------------Experimental Yes Yes Small SmallQuasi-Exp Yes No Non-Exp No No Large Large--------------------------------------------------------------

Non-Experimental Design: Causal Comparative StudyNon-experimental designs refer to efforts at causal inference based on measures taken

all at one time, with differential levels of both effects and exposures to presumed causes being measured as they occur naturally, without any experimental intervention.

According to some authors, there are two kinds of non-experimental designs, causal comparative and correlational studies. (Others including me do not think such a distinction is necessary.) The difference lies in the measurement of the "independent" variable which can be either categorical or continuous. The quotation mark indicates that an "independent" variable in non-experimental research is not truly what the term stands for since it is not manipulated. For this reason, a categorical or continuous "independent" variable is also called a grouping variable in causal comparative studies and an exogenous variable in correlational studies.

Causal comparative design is different from experimental and quasi-experimental designs in that there is no manipulation of the grouping variable and there is no random assignment of subjects into different groups. The question inspiring gender studies is whether males and females are different with respect to a particular behaviour or trait. In this example, the independent or grouping variable is gender which is not and can not be manipulated and there is no way to randomly assign some subjects into either of the two groups. Another difference is that, as stated earlier, groups in causal comparative research are often formed on the basis of the dependent variable. Researcher are often interested in finding out why, for example, some children are less motivated to learn than others, do not achieve as well as others, have more behavioral problems than others, are more aggressive than others, and turn out to be criminals. The behaviours on which questions are raised are

22 Dr. Chang

the results or outcomes or dependent variables. Researchers try to find answers to the individual differences on these variables by grouping subjects on these variables into, say, high versus low achievers, students with and without behavioral problems or with and without criminal records, and then compare these two groups of subjects on some suspected causes, e.g., parental supervision, peer influence, TV viewing, etc. Such logical thinking and research processes are almost the opposite of the experimental research. For example, in the above example, TV viewing as an independent variable will be manipulated in experimental research and subjects will be randomly assigned to groups with different amounts or different kinds of TV exposure and their subsequent aggressive, antisocial, or criminal behaviours are observed. In this experimental example, because of randomization, different experimental groups can be considered equal except for the manipulation of the TV viewing. Thus different validity threats can be reasonably ruled out and the observed difference among the groups on the dependent variable, aggressive behaviour, can be attributed to the independent variable, TV viewing. In the non-experimental example, however, one can not say that TV viewing causes aggressive behaviour even though the two groups of aggressive versus non-aggressive children were found to differ on TV viewing. The reason is that the two groups can not be assumed to be equal: They may differ in a lot of things in addition to aggressiveness, e.g., different family background, hormonal levels, personalities, etc. The same statistical analyses are used in causal comparative studies, such as ANOVA. The interpretations of the results should be far more cautious in causal comparative research.

Non-Experimental Design: Correlational StudyPartial correlation. The calculation is complicated but the idea of partial correlation is

simple. It is an estimate of the correlation between two variables in a population that is homogeneous on the variable (or variables) that is being controlled, whose effects are being removed, or variability on this variable is made into a constant. For example, a correlation between height and intelligence computed out of a sample that is heterogenous on age, say, ranging from 4 to 15, is a simple correlation which is high and positive. A partial correlation would be an average correlation between height and intelligence within each age group where age is a constant. This partial correlation which is likely zero more truly depicts the relationship between height and intelligence.

Spurious effect. When two variables are correlated solely because they are both affected by the same cause, the correlation between these two variables is spurious. For example, the tobacco industry argues that the correlation between cigarette smoking and lung disease is not causal but spurious in that both these variables may be caused by a common third factor such as stress or an unhappy mental state. Another example will be the positive correlation between height and intelligence often observed in children. Here, the correlation is again spurious because both variables have the common cause of chronicle age.

Mediating variable. The correlation between two variables can be the result of a mediating variable. For example, a strong correlation between SES and academic achievement is often observed and makes some people believe that there is a causal relationship between how rich the parents are and how well the kids do in school. However, such a relationship is now found to be mediated by a third variable, Achievement motivation. That is rich people's children are more motivated to study (by their parents' success) and this motivation leads to good academic performance. This later finding is achieved by correlating SES and Achievement while statistically partialling out Motivation. The correlation is almost

23 Dr. Chang

zero. The important implication of this statistical insight is that the key lies in motivating the poor kids (providing them with role models) whereas giving them material incentives may not make them study.

Suppressor variable. A special case when a partial correlation is larger than its zero-order correlation is called a suppressor variable effect. A suppressor variable is a variable that has a zero, or close to zero, correlation with the criterion or dependent variable but is correlated with the predictor or independent variable. When such suppressor variable is not taken into consideration, the correlation between the independent and dependent variable may be "suppressed" or reduced by this uncontrolled suppressor. For example, a paper-and-pencil pilot test as a predictor was found to predict little of the criterion, flying. The correlation was suppressed by a third variable, verbal ability, which has little to do with flying but a lot to do with test taking. When this suppressor variable was partialled out, the correlation between the pilot test and piloting increased significantly. This is a real example from pilot training during World War II.

Validity IssuesThe only way to enhance research validity of correlational studies which are post facto

or after fact is through careful logical deduction and well-thought-out statistical analyses. The first part requires a strong theory to map out relations among variables and careful thinking to include all the possible extraneous variables that may contribute to the post facto observed variations. The second part involves the use of various rather complicated analytic techniques, such as multiple regression, path analysis, and structural equation modeling. Partial correlation is the basic idea behind these analyses.

Thus, correlational studies often involve more variables than experimental research. The variables in a correlational study are not distinguished between independent and dependent variables. First, since there is no experimental manipulation of the variable of research interest, there is no independent variable. Second, because the inference is often not made from the manipulation of the independent to the outcome of the dependent variable. On the contrary, the post facto outcome is observed first and the research purpose is to account for the observations, the order of what is independent and dependent seems opposite to what is used in experiments. Third, unlike experimental studies where there is usually one dependent variable, there can be and usually are more than one outcome variable in a complex patten of associations. In correlational studies, variables are distinguished between what are referred to as exogenous and endogenous variables. An exogenous variable is one whose variability is assumed to be determined by causes outside the model or study under consideration. It is not the interest of the study to explain the variability of an exogenous variable or its causal relations with other exogenous variables. An endogenous variable is one whose variation is to be explained by the exogenous and/or other endogenous variables in the model. Exogenous and endogenous variables are like independent and dependent variables in experimental studies except that there is no manipulation of the exogenous variables and there are usually more than one endogenous variable in a correlational study.

In correlational studies one needs to think hard to include as many relevant variables as possible. Omission of relevant variables which are correlated with the exogenous or endogenous variables in the model constitutes what is called a specification error which will lead to biased estimates of the relations among the variables in the model. Specification errors are almost unavoidable in correlational research. All one can do is to attempt to minimize them by including in the design major relevant variables. Although it is hard to say

24 Dr. Chang

how many relevant or extraneous variables need to be included in a model, it is fairly certain to say that a correlational study involving only one "independent" variable is bound to be misspecified in virtually any instance that comes to mind.

An Evaluation Checklist for Quantitative Studies

Adopted from McMillan, J. H. (2004). Educational research: Fundamentals for the consumer (4th ed.) Boston: Pearson.

1.0 Research Problem

What are the independent and dependent variables?Is the problem researchable?Is the problem significant? Will the results have practical or theoretical importance? Is the problem stated clearly and succinctly?Does the problem communicate whether the study is descriptive, relational, or experimental?Does the problem indicate the population studied?Does the problem indicate the variables in the study?

2.0 Review of Literature

2.1 Does the review of literature seem comprehensive? Are all important previous studies included? 2.2 Are primary sources emphasized? 2.3 Is the review up to date? 2.4 Have studies been critically reviewed, and flaws noted, and have the results been summarized? (I disagree with 2.1 and 2.4.) 2.5 Does the review emphasize studies directly related to the problem? 2.6 Does the review explicitly relate previous studies to the problem? 2.7 If appropriate, does the review establish a basis for research hypotheses? 2.8 Does the review establish a theoretical framework for the significance of the study? 2.9 Is the review well organized?

3.0 Research Hypothesis

3.1 Is the hypothesis stated in declarative form? 3.2 Does the hypothesis follow from the literature? 3.3 Does the hypothesis state expected relationships or differences? 3.4 Is the hypothesis testable? 3.5 Is the hypothesis clear and concise?

4.0 Selection of Participants

4.1 Are the participants clearly described? 4.2 Is the population clearly defined? 4.3 Is the method of sampling clearly described? 4.4 Is probability sampling used? If so, is it proportional or disproportional? 4.5 What is the return rate in a survey study?

25 Dr. Chang

4.6 Are volunteers used? 4.7 Is there an adequate number of participants?

5.0 Instrumentation

5.1 Is evidence for validity and reliability clearly stated and adequate? Is the instrument appropriate for the participants?5.2 Are the instruments clearly described? If an instrument is designed for a study by the researchers, is there a description of its development? 5.3 Are the procedures for gathering data clearly described? 5.4 Are norms appropriate if norm-referenced tests are used? 5.5 Are standard setting procedures appropriate if criterion-referenced tests are used? 5.6 Do the scores distort the reality of the findings? 5.7 Do response set or faking influence the results? 5.8 Are observers and interviewers adequately trained? 5.9 Are there observer or interviewer effects?

6.0 Design

6.1 Descriptive and Correlational

6.1a If descriptive, are relationships inferred? 6.1b Do graphic presentations distort the findings? 6.1c If comparative, are criteria for identifying different groups clear? 6.1d Are causative conclusions reached from correlational findings? 6.1e Is the correlation affected by restriction in the range and reliability of the instruments? 6.1f If predictions are made, are the based on a different sample? 6.1g Is the size of the correlation large enough? 6.1h If causal-comparative, has the causal condition already occurred? How comparable are the participants in the groups being compared?

6.2 Experimental

6.2a Is there direct manipulation of an independent variable? 6.2b Is the design clearly described? Is random assignment used? 6.2c What extraneous variables are not controlled in the design? 6.2d Are the treatments very different from one another? 6.2e Is each replication of the treatment independent of other replications? Is the number of participants equal to the number of treatment replications?

7.0 Results

7.1 Is there an appropriate descriptive statistical summary? 7.2 Is statistical significance confused with practical significance? 7.3 Is statistical significance confused with internal or external validity? 7.4 Are appropriate statistical tests used?

26 Dr. Chang

7.5 Are levels of significance interpreted correctly? 7.6 How clearly are the results presented? 7.7 Is there a sufficient number of participants to give valid statistical results? 7.8 Are data clearly and accurately presented in graphs and tables?8.0 Discussion and Conclusions

8.1 Is interpretation of the results separate from reporting of the results? 8.2 Are the results discussed in relation to previous research, methodology, and the research problem? 8.3 Do the conclusions address the research problem? 8.4 Do the conclusions follow from the interpretation of the results? 8.5 Are the conclusion appropriately limited by the nature of the participants, treatments, and measures? 8.6 Is lack of statistical significance properly interpreted? 8.7 Are the limitations of the findings reasonable? 8.8 Are the recommendations and implications specific? 8.9 Are the conclusions consistent with what is known from previous research?

Quantitative versus Qualitative ResearchThe purpose of research is to draw some causal inference, A leads to B, A affects B,

so that intervention can be introduced. A particular study may or may not be able to draw causal conclusion, but the eventual goal of research in any discipline is to draw such conclusion. There is a fundamental difference in interpreting a causal relationship that distinguish quantitative vs. qualitative. (The chair example.) Quantitative focuses on the specific or most salient causal link whereas qualitative takes into consideration the chain of events as contributing to a specific social process. However, as a tradition, relationship, process, context are talked about in qualitative whereas "cause and effect" is considered quantitative terminology. More formally, these two philosophies are distinguished in the following formal explanations:

Alternative conditions: sufficient but not necessaryPresence of the conditions associates with the presence of the outcome; but the

absence of conditions does not associate with the absence of the outcome. It is sufficient by itself but not necessary. Flue virus is a sufficient but not necessary condition of headache.

Contingent conditions: necessary but not sufficientAbsence of the conditions indicates the absence of the outcome; but presence of the

conditions does not indicate the presence of the outcome. It is necessary but not sufficient by itself. Ability to discriminate letters is necessary but not sufficient to reading.

Conclusion drawn from the two philosophies:Quantitative: To be able to infer causality, conditions have to be both sufficient and

necessary; i.e., the presence of the conditions is accompanied by the presence of the outcome and the absence of the conditions is accompanied by the absence of the outcome. The only way to test a causal relationship is experiment where the independent variable (the cause) is manipulated to see the corresponding change in the dependent variable (the outcome). There

27 Dr. Chang

are different variations of experiment to fit in the constraints of social science research, e.g., quasi, causal comparative, correlational, but the ideas behind them are the same which are derived from physical science research, physics.

Qualitative: Constellation of conditions that are individually insufficient but necessary and jointly unnecessary but sufficient (INUS) to bring about the outcome. (Meehl's example.) Any social phenomenon resembles the INUS situation. Every social factor itself is not sufficient even though necessary but together they are sufficient to bring about an effect even though that combination is not necessary. The emphasis is on different factors, angles, points of views, culture, the big picture, the chain of events which are necessary but not sufficient. Since the combination of them, which is sufficient, is not necessary and there can be other combinations, there is not a pattern of relationship that is uniformly true. Here the emphasis is on the contexts, the situations. They have to be taken into consideration or findings are context dependent whereas quantitative emphasizes generalization and statistical inference from sample to population.

Qualitative try to study a process (rather than an isolated event) by taking into consideration different factors contributing to the process. But with the qualitative approach, there is discipline emphasis in educational research. An anthropological orientation (ethnography) emphasizes the role of culture in influencing behaviour. Researchers with a sociological bent tend to emphasize symbolic interaction. People are seen as acting according to the meaning of things and persons to them; their reality is socially constructed. People act not according to what the school is supposed to be, but according to how they see it.

Quan:Philosophy: Isolated causal linkMethod: experimentEffort: Control extraneous variable to isolate out the particular linkage; e.g., attitude of subject, history, instrumentation, testing, maturation. standardize data collection. Following physical science tradition, studying only that is observable, measurable, and testable. Latent constructs have to operationalized. Lot of topics are not attempted for research.

Qual:P: INUS, causal chainM: Field study. Consider combinations of factors. Look at context. Use different data collection schemes to obtain all sorts of information. Social phenomenon and human behaviour are not directly observable which include intentions, feelings, aspirations influenced by norms, culture, values. Observable behaviour aren't any real than internal phenomenon.

Quan:P: Following the physical science tradition, study the social phenomenon or human behaviour as an objective and impartial observer. M: control internal validity threats, such as, observer bias and characteristics. Keep the subjects unaware of your research purpose. Hire data collector and standardize the data collecting condition by training them. Structured interview. Let the data speak. Terminology: Subject, researcher

Qual:P: Take the perspective of the people being studied. See the world from the way they see it.

28 Dr. Chang

M: Participation. Researcher is the only or major source of data collection. Subjects play a role in data interpretation. Having subjects read your report and modify afterwards.T: Informants, collaborators, teachers, vs. participant.

Quan:P: Deductive reasoning, formulate theory from previous research and conduct specific empirical test.M: Hypothesis testing. Ask questions before data are collected. Use standardized test. Confirmatory or explanatory studies.

Qual:P: Inductive reasoning, theory grounded in observation. From pieces of specific events and observations, develop an explanation.M: Start from scratch. Extensive and prolonged observation. Going back and forth between data and explanation until a theory is fully grounded by observation. No measurement. Measurement is not just asking questions but knowing what to ask. Exploratory or discovery oriented research.

Quan:P: GeneralizationM: Random sampling, hypothesis testing, inferential statistics.

Qual:P: Context dependent, Generalization with cautionM: Purposive sample to gather data from most representative situation to draw generalization. Informants are selected for their willingness to talk, their sensitivity, knowledge, and insights into a situation, and their ability and influence to gain access to new situations. No intention to use statistical inference. Lengthy text report. Text analysis. Quan vs. qual is more of approach and philosophical difference than methodological difference. Techniques which were traditionally more often used by one approach than the other are now adopted by both. A case study can be used to triangulate on the findings from large scaled surveys. Field notes compiled through participative observations and personal interviews can be quantified and statistically analyzed to draw inference to the population.

Some Details about Research Designs

Randomly assign 3 subjects into each of the 3 treatments, Pu, Pr, and Control.

Treatment

Pu Pr Control

X1,X2,X3 X4,X5,X6 X7,X8,X9

MPu MPr MC MGrand

29 Dr. Chang

Factorial ANOVA:Randomly assign 3 subjects from each gender into each of

the 3 treatments, Pr, Pu, and Control.

Treatment

Pu Pr Control

Male X1,X2,X3 X4,X5,X6 X7,X8,X9

MPuM MPrM MCM MMale

Female X10,X11,X12 X13,X14,X15 X16,X17,X18

MPuF MPrF MCF MFemale

MPu MPr MC Grand

RBANOVA:Assign every subject into each of the 3 treatments, Pr,

Pu, and Control, in a random order.

Subject Treatment

Pu Pr Control Mean

1 X1 X1 X1 1

2 X2 X2 X2 2

3 X3 X3 X3 3

4 X4 X4 X4 4

Mean MPu MPr MC MGrand

SPANOVA:Assign every subject from each gender into each of the 3

treatments, Pr, Pu, and Control, in a random order.

Sex or Subject Treatment or Within-Subject FactorBetween-Sub Factor Pu Pr Control Mean

1 X1 X1 X1 1

30 Dr. Chang

2 X2 X2 X2 2

Male 3 X3 X3 X3 3

4 X4 X4 X4 4

MPuM MPrM MCM MMale

5 X5 X5 X5 5

6 X6 X6 X6 6

Female 7 X7 X7 X7 7

8 X8 X8 X8 8

MPuF MPrF MCF MFemale

MPu MPr MC MGrand

Pu: Experimental manipulation of public self-consciousness is achieved by having the subjects respond to the Spence's Attitudes Towards Women Scale with the awareness that their answers will be evaluated by other people.

Pr: Experimental manipulation of private self-consciousness is acquired by having the subjects respond to the Spence's Attitudes Towards Women Scale in front of a mirror.

Control: Subjects respond to the Spence's Attitudes Towards Women Scale without experimental manipulations.

The Spence's Attitudes Toward Women Scale is so scaled that higher values indicate more stereotyped attitudes towards women.

Between Subject DesignBetween subjects designs are used to draw inference about treatment effects to several

populations. This statement does not mean that subjects are drawn from different populations to start with. In fact you use random assignment to create equal groups. The statement means that the treatment (or experiment) is so effective that after the treatment the behavior of the subjects represent that of a different population.

Although referred to as an experimental design, it is also used for non experimental comparisons of different population means. As a non-experimental design, subjects are sampled from different populations to start with.

Commonly used between-subject designs are the (1) completely randomized design and (2) factorial design. Factorial design deals with more than one independent variable. In the case of two independent variables, the design is called two-way factorial. The two variables can be both experimental variables. More often, one of the two variables is experimental and the other is a measured variable. A special case is the aptitude treatment interaction design where one of the two variables is the state variable, which is experimentally

31 Dr. Chang

manipulated, and the other is a trait variable, which is measured. All these are experimental designs. Of course, when both variables are measured, e.g., gender by race, the design is non-experimental. Three hypotheses are explored in a two-way factorial: (1) the main effect due to one of the variables, (2) the main effect due to the other variable, and (3) the interaction between the two variables. The factorials are also distinguished between “fixed” and “random” depending on whether the categories of the independent variables are assumed to exhaust those in the defined population (fixed design) or are a random sample of the population (random design).

Within Subject DesignIn a between design, each subject is observed under one condition and inference is

drawn between subjects from different conditions. In a within design, each subject is observed under more than one condition and inference is drawn within the same subject across different conditions. There are usually three ways a within-design can be carried out. All three are generally referred to as randomized blocks design.

(1) Subjects are observed under all treatment conditions. The order of the conditions assigned to each subject must be random. For example, each subject experiences three dosage conditions of 1 mg, 5 mg, 10 mg. Some subjects have 5 mg first and some have 10 mg first. The key is that the three conditions are assigned to each subject in a random order. As another example, have the same teachers rate content-identical essays which bear names representing immigration status and gender identities to see if there is discrimination in rating essays. In this example, each teacher rates 4 essays (female-HK, male-HK, female-Mainland, male-Mainland) the order of which is random.

(2) Assign matched subjects randomly into different treatments. K matched subjects form a block. Within a block subjects are randomly assigned to treatment conditions with one subject per condition. Subjects within a block are considered identical as far as the dependent variable is concerned. The design strength lies in that the within-block variability is smaller than between-block variability. In the earlier example, we can match subjects by, for example, age, gender, physical conditions (all of which have to be related to the dependent variable), and randomly assign each matched triplet into the three dosage conditions.

(3) A special case of the first design is the repeated measures design where each subject undergoes repeated observations. For example, in a test-retest design, a subject is observed twice, before and after the treatment. Repeated measures design can also be used in non-experimental studies involving multiple observations.

Mixed DesignIf you add a between-factor to the Randomized Blocks Design, you have the Split Plot

Design (SP) which is a mixed design. There are now two independent variables. One of them varies within the subjects and, thus, is called within-factor. The other varies between subjects and is called between-factor. This basic mixed design can be extended to include more than one within-factor and/or more than one between-factor.

All the requirements for RB design apply to the SP design. All three situations which make up an RB design apply to a SP design. The three situations are: 1. Same subjects are observed under all treatment conditions the order of which is random. 2. Matched subjects form a block within which subjects are considered equivalent and randomly assigned to treatments. 3. Same subjects undergo repeated measures where random order does not exist. The same three situations make up the within-factor of a SP design.

Either of the two factors which make up a SP design can be an experimental or

32 Dr. Chang

measured variable, creating four possible scenarios: 1. Between-factor is an experimental (state) and within-factor is a measured (trait)

variable. For example, the between-factor is Pu (public self-consciousness induced by the camera condition), Pr (private self-consciousness induced by the mirror condition), and a control group. Subjects are randomly assigned to these three conditions. The within-factor is pre-test (before the experimental conditions) and post-test (under the experimental conditions). This is the typical pretest-posttest experimental design. (One typical RB design is the test-retest design which is not experimental because there is no control group.) In this example, the within-factor represents Situation 3 in B above.

2. Between-factor is a measured and within-factor is an experimental variable. In the above example, the Pu, Pr, and Ctl conditions can be made a within-factor by randomly assigning these conditions either to the same subjects or to blocks of matched triplets. The between-factor can be gender. In this example, the within-factor represents Situations 1 or 2 in B.

3. Both factors are experimental. In our earlier Prozac example, the within-factor is the three experimental conditions of 10mg Prozac, 5mg Prozac, and placebo. We can add a between-factor of whether or not receiving therapy to cope with depression. The research question is whether the combination of counselling and Prozac is more effective than either the counselling or Prozac alone. In this case, both the between and within factors are experimental variables. The between-factor conditions are created by randomly assigning subjects into either the counselling or no-counselling condition. The within-factor can be created in two ways. Corresponding to Situation 1 in B, within each of these two between-factor conditions, subjects undergo all three Prozac conditions in a random order. Corresponding to Situation 2 in B, matched triplets are randomly assigned to one of the three Prozac conditions.

4. Finally, both factors can be measured in a non-experimental study. For example, I'm obtaining repeated measures on withdrawn and aggressive behaviors of primary school children. I'm going to give the children three questionnaires at the end of three consecutive semesters. The within-factor is the three waves of questionnaire data. One between-factor could be gender. Another between-factor could be popularity classification of these children. Through peer nomination, children can be classified into popular, rejected, and neglected.

Analysis of Covariance (ANCOVA)ANCOVA is ANOVA with a covariate or covariates. The covariate summarizes

individual differences on the dependent variable. These differences would otherwise be allocated as error in an ANOVA. Apparently, the covariate should be correlated with the dependent variable. By removing variations due to persons from the error term, ANOVA achieves the same goal as that of a within-subject design or a mixed design (RB and SP ANOVA). Three statistical assumptions:

1. Linearity of regression. The relationship between the dependent variable and the covariate is linear. Simply check the scatter plot to examine this assumption.

2. Homogeneity of regression. The regression of the dependent variable on the covariate is the same across different treatment groups. The most intuitive way to check this assumption is to conduct separate regression within the treatment groups to see if the regression coefficients are similar.

3. The covariate is independent from the treatment variable. The way to ensure this assumption is to obtain the covariate before conducting the experiment. That is measuring the covariate before randomly assigning subjects into different treatment conditions and

33 Dr. Chang

conducting the experiment. The purpose of the covariate is to reduce within group variance or difference among

people within groups. If the covariate is highly related with the independent variable, the individual difference between groups will be reduced by the part of the covariate that is related to the independent variable. However, in non-experimental and quasi experimental studies, e.g., teaching methods are compared using intact classes which could differ in ability and ability is used as covariate, the covariate is often correlated with the independent variable and ANCOVA is still used. In this situation, a different question is sought after in non-experimental research. The question is how much is purely due to the grouping variable after the covariate is accounted for. Here researchers may also like to use a covariate which is related to a grouping variable so that "pure" group differences can be identified after controlling for the covariate. For example, in comparing gender role attitudes among several age groups, one may want to control for education which is related to age groups. In this design, one can find out how much is due to education and how much is due to pure age.