Validity

RMS – May 28, 2014

Measurement Reliability

• The extent to which a measurement gives results that are consistent

Measurement Validity

• The degree to which an assessment measures what it is supposed to measure

Construct validity

• The extent to which an operationalization of a construct (i.e., practical tests developed from a theory) actually measure what the theory says they do– Does an IQ questionnaire actually measure

intelligence?

Content Validity

• Systematic examination of the test content to determine whether it covers a representative sample of the behaviour domain to be measured– Does the IQ questionnaire have items covering all

areas of intelligence known in the literature?

Face validity

• An estimate of whether a test appears to measure a certain criterion (no guarantee that it actually does!)– Related to content validity, but you don’t have to

know all the theories to make the judgment

Criterion validity

• Are the test and a criterion variable(s) representative of the same theoretical construct related– Compares the test with other measures already

held to be valid• IQ tests validated against measures of academic

performance (the criterion)

Assignment 2 – Variables & Measurement

1. Recap your research question and hypotheses2. Identify the independent and dependent variables of interest for each hypothesis. For each variable:

a. How could you measure the variable (i.e., what is its operational definition)?b. Should it be continuous or discrete?c. What is its scaling type?d. What descriptive statistics are appropriate?e. How would you estimate reliability and validity of measurement?

Questions about assignment 2?

Experimental validity

Validity

• Experimental validity – the soundness of the experimental design– Not the same as measurement validity (the

goodness of the operational definition)

• Internal validity• Construct validity• External validity

Internal Validity

• Concerns the logic of the relationship between the independent and dependent variables

• It is the extent to which a study provides evidence of a cause-effect relationship between the IV and the DV

Confounding variables

• Error that occurs when the effects of two variables in an experiment cannot be separated – results in a confused interpretation of the results

• Example– Group A (IV = 0)– Group B (IV = 1)– Behaviour observed

• Want to say that changes in behaviour are due to IV

How do you know what could be confounding?

• Need to make judgments as you design the experiment

• Be particularly careful with subject variables

Construct Validity

• Extent to which the results support the theory behind the research– Would another theory predict the same

experimental results?

• Hypotheses cannot be tested in a vacuum– The conditions of a study constitute auxilliary

hypotheses that must also be true so that you can test the main hypothesis

Example

• H1: Anxiety is conducive to learning• Participants selected on basis of whether they bit their

fingernails (sign of anxiety)• Observe how fast they can learn to write by holding a

pencil in their toes (a learning task)• Did not just test impact of anxiety of learning– Tested that fingernail biting is a measure of anxiety (H2)– Tested that writing with toes is a good learning task (H3)

• If either is false, you could have negative results even if the main hypothesis is true

Similarities

Construct Validity• Try to rule out other

possible theoretical explanations of the results

• Must design a new study to help you choose between competing theoretical explanations

Internal Validity• Try to rule out alternative

variables as potential causes of the behaviour of interest

• May be able to redesign the study to control for the source of confounding

External Validity

• How well the findings of an experiment generalize to other situations or populations– Strictly speaking results are only valid for other

identical situations– Can be hard to know which situational variables

are important

• Ecological validity is related– Extent to which an experimental situation mimics

a real-world situation

Statistical Validity• Extent to which data are shown to be the result of a cause-

effect relationship rather than an accident– Does the relationship exist or was it caused by pure chance

• Similar to internal validity• Notion of power (is n big enough?)• Is measure accurate?• Is the statistical test appropriate?

– 3 kinds of lies: lies, damn lies, and statistics• Statistical test establishes that an outcome has a certain

low probability of happening by chance alone– No guarantee that it’s not a random error in sampling or

measurement

So we did the green study again and got no link. It was probably a – RESEARCH CONFLICTED ON GREEN JELLY BEAN/ACME LINK; MORE STUDY RECOMMENDED - xkcd.com/882/

Threats to Internal Validity - 1• Events outside the laboratory (history):– Can occur when different experimental conditions are

presented to subjects at different times• Example: Feelings of depression (failure condition on

Monday, success condition on Wednesday)– What if Monday was rainy and Wednesday was sunny?

• Maturation:– A source of error in an experiment related to the

amount of time between measures– Subjects may change between conditions because of

naturally occurring processes• Aging doesn’t just occur with people!

Threats to Internal Validity - 2• Effects of Testing:– Changes caused by testing procedure (not processes

unrelated to the experiment)• Learn how to do the task, learn the style of a test

• Regression Effect:– tendency of subjects with extreme scores on a 1st measure

to score closer to the mean on a second testing– Not a perfect correlation between 2 variables

• SAT and GPA or repeated SAT • Arises when there is an error associated with the measurement of

the variables (e.g., students know some answers and make lucky/unlucky guesses at the rest)

• Random error – that part of the value of a variable that can be attributed to chance

Threats to Internal Validity - 3

• Mortality:– The dropping out of some subjects before an

experiment is completed– Selective subject loss– Can introduce bias depending on the reason they

are dropping out

• Selection:– Any bias in selecting groups can undermine

internal validity

Threats to Construct Validity – 1

• Difficult as there are an indefinite number of theories that may account for a given relationship

• Strategy: ask whether alternative theoretical explanations of the data are less plausible than the theory believed to be supported by the research

Threats to Construct Validity - 2

• Loose connection to theory and method– The anxiety/learning example– Often research suffers from poor operational

definition of theoretical constructs

• Ambiguous effect of IV– Experimenter can control all reasonable confound

variables– Participant may compromise result by seeing the

situation differently than the experimenter

Human-subject research

• Good subject tendency:– Participants want to help the research achieve

their goals (and may not understand the goals)

• Evaluation apprehension:– Tendency of participants to alter their behaviour

to appear as socially desirable as possible

Threats To External Validity

• Other subjects– Are the participants truly representative of the

sample to which you are trying to generalize?• Other times– Would the same experiment conducted at another

time produce the same results?– Caution: the web is rapidly changing

• Other settings– Lab vs field– Structured vs unstructured environments

Reading: Exp. Challenges in Cyber Security

• What’s the research problem?• What was their approach?• What types of validity do they discuss?• Hypotheses?– IV, DV– Measurement validity

• Internal validity• Construct validity• External validity