Validity, Reliability and Validity, Reliability and GeneralizabilityGeneralizability

ValidityValidity

HOW MIGHT YOU BE WRONG?HOW MIGHT YOU BE WRONG?

Internal ValidityInternal Validity

External ValidityExternal Validity

Construct ValidityConstruct Validity

ValidityValidity

Are your findings really about what they Are your findings really about what they appear to be about or did something else appear to be about or did something else cause them?cause them?

Internal ValidityInternal Validity

Did your treatment actually cause the Did your treatment actually cause the outcomes?outcomes?

Did anything else potentially cause the Did anything else potentially cause the outcome instead of your treatment?outcome instead of your treatment?

INTERNAL VALIDITYINTERNAL VALIDITY

Internal validity addresses the "true" causes of the Internal validity addresses the "true" causes of the outcomes that you observed in your study.outcomes that you observed in your study. Strong Strong internal validity means that you not only have internal validity means that you not only have reliable measures of your independent and reliable measures of your independent and dependent variables BUT a strong justification that dependent variables BUT a strong justification that causally links causally links your independent variables to your your independent variables to your dependent variables. At the same time, you are able dependent variables. At the same time, you are able to rule out extraneous variables, or alternative, often to rule out extraneous variables, or alternative, often unanticipated, causes for your dependent variables. unanticipated, causes for your dependent variables. Thus strong internal validity refers to the Thus strong internal validity refers to the unambiguous assignment of causes to effects.unambiguous assignment of causes to effects. Internal validity is about Internal validity is about causal controlcausal control..

EXTERNAL VALIDITYEXTERNAL VALIDITY

External validity addresses the ability External validity addresses the ability to generalize your study to other to generalize your study to other people and other situations.people and other situations. To have To have strong external validity (ideally), you strong external validity (ideally), you need a probability sample of subjects need a probability sample of subjects or respondents drawn using "chance or respondents drawn using "chance methods" from a clearly defined methods" from a clearly defined populationpopulation

External ValidityExternal Validity

When you have strong external validity, When you have strong external validity, you can generalize to other people and you can generalize to other people and situations with confidence. situations with confidence. – External validity refers to the population we External validity refers to the population we

use and the setting. If I do a study with fifth use and the setting. If I do a study with fifth graders, then the results can be applied to a graders, then the results can be applied to a similar set of fifth graders. similar set of fifth graders.

Construct ValidityConstruct Validity

Does your test measure what you think it Does your test measure what you think it really measures. What are you measuring really measures. What are you measuring or observing?or observing?

Keep it simple and account for it.Keep it simple and account for it.

CONSTRUCT VALIDITYCONSTRUCT VALIDITY

Construct validity is about the correspondence Construct validity is about the correspondence between your concepts (constructs) and the actual between your concepts (constructs) and the actual measurements that you use.measurements that you use. A measure with high A measure with high concept validity accurately reflects the abstract concept concept validity accurately reflects the abstract concept that you are trying to study. Since we can only know that you are trying to study. Since we can only know about our concepts through the concrete measures that about our concepts through the concrete measures that we use, you can see that construct validity is extremely we use, you can see that construct validity is extremely important. It also becomes clear why it is so important to important. It also becomes clear why it is so important to have very clear conceptual definitions of our variables. have very clear conceptual definitions of our variables.

Construct ValidityConstruct Validity

Construct validity is often established Construct validity is often established through the use of through the use of a multi-trait, multi-method a multi-trait, multi-method matrix.matrix. At least two constructs are measured. At least two constructs are measured. Each construct is measured at least two different Each construct is measured at least two different ways, and the type of measures is repeated ways, and the type of measures is repeated across constructs. For example, each construct across constructs. For example, each construct first might be measured using a questionnaire, first might be measured using a questionnaire, then each construct would be measured using a then each construct would be measured using a similar set of behavioral observation categories. similar set of behavioral observation categories.

RELIABILITYRELIABILITY

In order to make any kind of causal In order to make any kind of causal assessments in your research situation, assessments in your research situation, you must first haveyou must first have reliable reliable measures, measures, i.e., stable and/or repeatable measures. If i.e., stable and/or repeatable measures. If the random error variation in your the random error variation in your measurements is so large that there is measurements is so large that there is almost no stability in your measures, you almost no stability in your measures, you can't explain anything!can't explain anything!

ReliabilityReliability

Researcher Bias-How might you have Researcher Bias-How might you have effected the study? Account for it!effected the study? Account for it!

RELIABILITYRELIABILITY

Picture an intelligence test where an Picture an intelligence test where an individual's scores ranged from moronic to individual's scores ranged from moronic to genius level. No one would place any faith genius level. No one would place any faith in the results of such a "test" because the in the results of such a "test" because the person's scores were so unstable or person's scores were so unstable or unreliable. unreliable.

ReliabilityReliability

Reliability is required to make statements about Reliability is required to make statements about validity. However, reliable measures could bevalidity. However, reliable measures could be biasedbiased and hence "untrue" measures of a and hence "untrue" measures of a phenomenon) phenomenon) or confoundedor confounded with other factors with other factors such as acquiescence response set. Picture a such as acquiescence response set. Picture a scale that always weighs five pounds too light. scale that always weighs five pounds too light. The results are reliable, but inaccurate or The results are reliable, but inaccurate or biased. Or, picture an intelligence test on which biased. Or, picture an intelligence test on which women or people of color always score lower women or people of color always score lower ((even if this doesn't occur on other testeven if this doesn't occur on other tests). Again, s). Again, the measure may be reliable but biased. the measure may be reliable but biased.

ReliabilityReliability

Identifying and controlling threats to Identifying and controlling threats to reliability and validity in qualitative reliability and validity in qualitative research is critical. Asking the question of research is critical. Asking the question of whether another researcher, going into the whether another researcher, going into the field and asking the same interview field and asking the same interview questions would get the same answers is questions would get the same answers is the overarching premise of reliability. the overarching premise of reliability.

Generalizability (Be Honest)Generalizability (Be Honest)

The study is generalizable to other school heads The study is generalizable to other school heads from international schools in Asia. However, one from international schools in Asia. However, one limit to this study was that the study will be limit to this study was that the study will be conducted with schools only in Asia. Therefore, conducted with schools only in Asia. Therefore, the findings will not apply universally. A second the findings will not apply universally. A second limit to the study is that the entire study was limit to the study is that the entire study was funded by the researcher, with limited funds for funded by the researcher, with limited funds for air travel and miscellaneous expenses. air travel and miscellaneous expenses. Therefore, schools were selected within a four Therefore, schools were selected within a four hour flight range of the researcher’s home town. hour flight range of the researcher’s home town. Telephone interviews were deemed less reliable Telephone interviews were deemed less reliable as they did allow for interpretation of body as they did allow for interpretation of body language and visual clues. language and visual clues.