1

Is a “Discussion” on

“Are Observational Studies Any Good” Any Good

Don HooverMay 2, 2014

2

Everyone Already Knows Observational Studies Are Not Perfect … Right?

• But who thinks– the real type 1 error is 0.55 when the nominal is 0.05?– The real coverage of a 95% confidence interval is 25%?– That’s what David Madigan and the OMAP team find

• This obviously makes such results meaningless

• But how many papers with these properties are being (and will continue to be) published ???

3

But Does David’s Talk Really Apply to ALL Observational Studies?

• They Only Look at Observational Studies of Drug Use and Adverse Consequences

• There’s other kinds of Observational Studies … on HIV, Epi, Health Behaviors, Nutrition, etc.– No one has looked at these types of studies

• These other studies must have similar problems• Maybe at a smaller magnitude

– But there are no “negative controls” for these settings … so no one can check this

4

The Approach here is Creative and Innovative

• Finding Negative Control Exposures or Outcomes to derive empirical distribution of the test statistic somewhat equalizes assumptions and unmeasured confounding

• With a given Drug Use as the exposure and a given Disease the outcome, such negative controls are readily available in many data sets

• So maybe something like it should be used when possible

• But now some questions ……

5

Q1- Why were Negative Control Drugs More Associated With Outcomes than by Chance?• People put on Any Drug are Sicker?• Those receiving a negative (control) drug are

more likely to receive some other positive drug?• Those apriori more likely to have a given disease

outcome are steered to the negative drugs?• Incorrect statistical models used?

6

Q2- Is this Approach Practical?

• A lot more work to fit many models than the standard approach which only fits one

– More money as well - A grant application using it would be less likely to get funded

– More work also means more chance for error in implementation

7

Q3 – How does one interpret a positive drug with empirical P < 0.05?

Positive Drug with empirical P < 0.05

The use of an “empirical” approach acknowledges we do not know what is going on so maybe the P < 0.05 is from model artifact not

causal

Calibrated Normal Scores of Negative Controls

8

Q4 – What is done with “Negative Drugs” more extreme than the Positive One

Calibrated Normal Scores of Negative Controls Positive Drug with P < 0.05

Should these Negative Controls all be

Examined for Causal Association as their

Signal is larger than the positive drug?

9

Q5 - How to handle Heterogeneity in Denominator of Calibration Statistic

Variance may introduce Apples to Oranges comparisons especially if although such does not appear to be the case in the examples

David used

From Schumie … Madigan Stat Med 2014 33; 209-18

( )Log RR1 ,ˆ2 2 2and other 'sτ <σ τn