What Every Stat Student Ought To Know About Causation

Paul W. Holland Educational Testing Service

“There are three kinds of liesLies, Damn Lies,and Statistics.”

Quoted from Mark Twain, who attributes it to Benjamin Disraeli (PM 1868-1880) (and who

knows if he ever really said it?)But if he did, why would such a significant

policy-maker say such a thing?Was it just another Statistics Joke like:

“Figures never lie, but liars always figure”or

“You can prove anything with Statistics”???

Perhaps Disraeli had some bad experiences with what we would now call “misuses” of

statistical data.Perhaps “statistics” were used to shoot down

some of his favorite policies.Its not too hard to do this.

Some Easy Causal “Misuses” of Data from theNational Assessment Of Educational Progress

(NAEP).Widely practiced Educational Tools (reading groups, work sheets, drill and practice) are associated with lower student performance.

Preferred Educational Practices (smaller classes, computers, a stable teaching force) are

associated with higher student performance.

IT’S ALL OLD STUFF, BUT STILL WORTH REPEATING.

Before leaping to causal conclusions we need to first consider other plausible (causal?) explanations.

Here is an alternative to the claim that the poor test performance is caused by the widely-used

educational practice.

EducationalPractice

Earlier TestPerformance

NAEPScores

This is often called “reverse causation,” i.e., when the “effect is really the cause”

Here is an alternative to the claim that the good test performance is caused

by the preferred educational practice.

This is usually called the “common cause” explanation.

EducationalPractice

NAEPScores

SES/SocialSegregation

Simpson’s Paradox is behind both of these. It’s recognition is probably 100 years old.

Yule, an English Statistician, born in 1871, was too young to be a statistical consultant for Disraeli

What Is There About Causationthat Statisticians Really Ought To Tell People?

There are several lists going back at least 2000 years!

Here are some well-known ones.

Aristotle’s Types of Causes: Material, Efficient, Formal, and Final. (He liked multiple views).

Hume’s Three Conditions: Temporal Succession, Spatial/Temporal Contiguity and Constant Conjunction. (He thought “causes” were illusions).

Mill’s Methods: Agreement, Difference, Residues and Concomitant Variation. (An empiricist).

Koch and Henle’s Three Conditions: to establish that a micro-organism is the cause of an infectious disease. (Koch solved anthrax, tuberculosis & cholera.)

Sir Bradford Hill’s Nine Conditions: to go from

Association to Causation (I.e., Smoking and Health).

Campbell’s and Stanley’s List of: “Threats to Validity” and “Plausible Rival Hypotheses” in Non-experimental Studies. (To aid those carrying out real program evaluations).

In the rest of this talk I will discuss a list of my own that is somewhat

different from these.

Natural Language Is Not Very Good At Making Important Causal

Distinctions

Interrogatives:

“Who is”, “What is”, “Where is, and “When is” (Description)

Versus

“Why is”, “What if” and “How does” (Causation).

The main words for causal explanation are: “Because” and “Due to”, but they are ambiguous.

“She did well on the exam because she is a woman.”“She did well on the exam because she was well prepared

for it.”

“The lecture put me to sleep due to its soporific nature.”

Other causal words: Determines, drives, impacts, affects.

The Weasel words: Risk Factor, association, correlates.

2. The Great Divide: Description Versus Causation

Description: Ethnography, case studies, anecdotes, surveys, polls; Percents, means, distributions, anatomy, maps;Sample versus Population, biased Samples, representativeness.

Causation: Comparative studies, quasi-experiments, controlled or randomized experiments; Correlations, regression coefficients, mean differences; Biased comparisons.

The slippery slope from Description to Causation:

Description often invites COMPARISONS, and comparisons often lead to CAUSAL questions.

“Casual comparisons inevitably initiate careless causal conclusions.”

--PWH, 2000

“The shift from casual to causal conclusions is more than a mere vowel movement.”

--HW, 2000

“…all those earlier texts had concerned themselves with pinning down the cause of motion. Galileo proposed to strike out on a

different course—to drop all Aristotelian talk of why things moved, and focus instead on the how, through painstaking observations

and measurements.”--Dava Sobel,

2000Galileo’s Daughter

“Newton did not show the cause of the apple falling, but he showed a similitude. …. between the apple and the stars.

…(he) was well content if he could bring diverse phenomena under ‘two or three Principles of Motion’ even though ‘the causes of these

Principles were not yet discovered’.”--D’Arcy Thompson, 1961

Description is nothing to be ashamed of.

Without Tycho Brae there could be no Kepler.

3. The Three Kinds Of Causal Questions.

Proposing/Identifying Causes,? ----------> E

What are the causes of child abuse?What caused the Great Depression?

What caused the accident?

Assessing Effects,C ----------->?

What will class size reduction do to test scores in California’s schools?Will eliminating social promotion increase student learning?

What about the drop-out rates?

Proposing/Describing Mechanisms,C ----------->? ----------->E

How does Aspirin reduce heart attack risk?By reducing inflammation or thinning blood ?

How will class size reduction improve student learning?By increasing teacher’s time with each student individually

or reduce instructional disruptions?

4. Proposing Causes Or Causal Mechanisms Are Examples Of Forming Causal Hypotheses

They can be wrong.They can change as new information comes in.

The Aristotelian Conceit is that human beings can figure out the causes of all things.

Assessing causal effects is different.Effects can be assessed with bias (be wrong) or without bias.

If biased they might change with improved study designIf unbiased they (won’t?) change.

“Old replicable experiments never die, they just get reinterpreted.”

5. Assessing The Effects Of Causes Is What Statistics Does Best.

This is the main purpose of all Causal Studies—Controlled Experiments, Randomized Experiments and Observational Studies/Quasi-Experiments.

The Assessment of a Causal Effect may be biased but bias may be reduced by improved design of the Causal Study.

Assessing Causal Effects is to Proposing Causes or Causal Mechanisms as Data is to Theory.

(Relevant previously assessed Causal Effects often inform the proposal of Causes and Mechanisms.)

Assessed Effects are often the sort of causal answers that policy makers want to hear.

As opposed to Causal Theories as to why the effects are observed.

6. Intuitions behind assessing effects.

The Minimal Ideal Controlled Experiment has three parts i) Two identical units of study ii) Two precisely defined and executed experimental conditions. iii) Precisely measured outcome observed on each unit an appropriate time after exposure to the experimental conditions.

There are then three “Loci of Control”i) The homogeneity of the unitsii) The precision of the conditionsiii) The accuracy of the measurement of the outcomes.

7. Causal Studies Can Lose Control At Each Of The Three Loci

Good causal study design tries to maintain some measure of control at these loci.

Examples:A) HETEROGENEOUS UNITS. Blocking and Random Assignment.

Blocking uses the available unit homogeneity to group “identical” units to be treated DIFFERENTLY.

Random assignment then spreads the remaining unit inhomogeneity across the treatment conditions to avoid systematic bias.

Matching and covariance adjustments are versions of blocking.All address concerns about the initial comparability of the groups

being treated differently.

B) THE CAUSES/TREATMENTSThe integrity of treatments being compared, were the units treated as we thought? (Blindness of patients is to help insure treatment integrity)Can we control the treatment doses that we want to study?

C) THE OUTCOMESThe comparability of the outcomes being measured.(Blindness of physicians is to help insure the comparability of subjective assessments.)

Replication is to reduce the amount of measurement error, but it only reduces unbiased error, and not systematic biases.(A big study is not necessarily an unbiased study)

The ability to randomize often,but not always,

implies the ability to exert controlat the other loci of the study.

Lack of controlat each of the three loci of an studycan lead to different sorts of biasesin the assessment of causal effects.

8. Good Causal Studies Require Attention At All Three Loci To Improve Control

There are two approaches to attending to “lack of control”:

A) Make untestable assumptions(Strong Ignorability, Instrumental Variables, Selection Modeling, “Natural” Experiments.)

B) Collect relevant data(Pre-intervention covariates, detailed records of the treatments actually received, multiple outcome measures.)

A good principle for the design of observational studies is to MAXIMIZE the collection and use of relevant data and to MINIMIZE the use of untestable assumptions.

9. Focussing On Effects Raises The Problem Of “WHAT Can Be A CAUSE?”

In most studies of interest to Social Scientists it is usually pretty obvious what are the UNITS (of study), and the OUTCOMES (or dependent variables) of interest.

What is more difficult, and “error prone”, in my opinion, are decisions about WHAT can be a CAUSE, i.e., which independent variables are CAUSAL.

My simple rule of thumb is:

If IT can be a TREATMENT in a (comparative) EXPERIMENT, then IT is a CAUSE and can have a CAUSAL EFFECT.

Otherwise, IT isn’t and can’t.

Manipulations, policy variables, treatments can be causesBUT“Unchanging” attributes such as Age, Gender, Race, or Pretest scores can not.

The key idea is that a “cause” is something that could have been different from what it was—like the experimental treatment received could have

been different from what it actually was.

This is a very hard requirement for some to swallow.

I think this happens when people are thinking about identifying causes or proposing causal mechanisms without paying attention to the

more basic process of assessing causal effects.

Whenever Race or Gender is used as a “Cause” with an “Effect”, then the “explanation” is Descriptive rather than Causal.

Studies of racial or gender bias in salaries or other social outcomes are places where this mistake is made every day.

Discussions of “Nature versus Nurture” are some of the most harmful version of this confusion.

10. So What is the Causal Role Of Attributes Of UnitsLike Race or Gender?

The ubiquity of a treatment effect versus the treatment acts differently on different units (statistical interaction).

One size might not fit all.

In a world of heterogeneity, Hume’s“Constant Conjunction” is not a usefulidea. (It’s a non-Humean world outthere)

SUMMARY OF WHERE WE HAVE COME SO FAR

1. Natural Language Is Not Very Good At Making Important Causal Distinctions.

2. The Great Divide: Description Versus Causation3. There are Three Kinds Of Causal Questions, Answers Or Inferences4. Proposing Causes Or Causal Mechanisms Are Examples Of Forming Causal Hypotheses.

5. Assessing The Effects Of Causes Is What Statistics Does Best.6. The Minimal Ideal Comparative Experiment.7. Causal Studies Can Lose Control At Each Of The Three Loci.8. Good Causal Studies Require Attention At All Three Loci For Improving

Experimental Control.

9. Focusing On Effects Raises The Problem Of “WHAT Can Be A CAUSE?”10. What is the Causal Role Of Attributes Of Units?