a blind analysis you are not allowed to peek! prof. donald koetke senior research professor of...

A Blind AnalysisA Blind Analysis

You are not allowed to peek!You are not allowed to peek!

Prof. Donald Koetke

Senior Research Professor of Physics

Valparaiso University

3/30/07 VU Colloquium 2

OutlineOutlineOutlineOutline

1. What is a “blind analysis”?

• What is a “bias”?

• Does “bias” = “systematic error”?

• What is the special (subtle) bias?

2. What is the history of the blind analysis”?

3. What are examples of “blind analysis” in physics, astronomy, astrophysics, and, … ?

4. Some thoughts and reflections --


A Blind AnalysisA Blind AnalysisA Blind AnalysisA Blind Analysis

What is a “bias”?

Does “bias” = “systematic error”?

What is the special (subtle) bias?

“A prejudice in favor of or against” -- may bedue to computer codes, equipment performanceor setting, decisions/selections/cuts imposed, etc.

Yes - if it causes a systematic shift in a result(This is not a “mistake” or “blunder”.)

Experimenter bias in making decisions/choices to achieve a desired answer. Examples …



1. You know what the answer “should be” -i.e., you know what the “accepted” answer is -- but your answer is different -outside of errors.What do you do?

2. You make a measurement and within errorsit agrees with the “accepted” answer -- What do you do?

You do an experiment and…

Asymmetry!!



The goal of a “blind analysis” is to preventthe experimenter from (unconsciously - or consciously) making decisions in the analysis that would affect the result based on:

• The predictions of a model or theory (e.g., the Standard Model for particle physics)

• Previous measurements known to the experimenter

• The experimenter’s intuition or other predisposition

These are all examples of personal bias




A “blind analysis” is an analysis of measured data in which the final answer is kept hidden from the experimenters until all of the decisions about the analysis have been made:• Computer codes have been developed and tested

• Decisions about the number of events (trials) needed have been made (cuts have been selected)

• Apriori agreements are reached about what to do when the real answers from the experiment are revealed; no further analysis!


OutlineOutline









HistoryHistoryHistoryHistory

Blind analysis begins in ~1930s with -

Medical research -- blind tests!

Patients don’t know whether they are getting --

a) the medicine/treatment, or,

b) a placebo

Patients are assigned to (a) or (b) ramdomly

Therefore - the patients are “Therefore - the patients are “blindblind” to their treatment” to their treatmentTherefore - the patients are “Therefore - the patients are “blindblind” to their treatment” to their treatment

They may imagine symptoms or cure, etc., but only the researcher/physician knows if these can be due to the medicine/treatment.





• Patients talk to medical researcher

e.g., how are you feeling? is pain less or more? are the other problems? serious or not so serious? before or after you take the medicine? …etc.

• Researcher hears what patient says

• Researcher examines the patient

• Researcher records all this as “data”

Do you see a Do you see a problem here?problem here?

Do you see a Do you see a problem here?problem here?

c.f.,1937 JAMA 26 June 2178/2.





• Therefore, the medical researcher can (and will) unconsciously and unintentionally interpret and record the information (data) with this bias (knowledge).

The patients are “The patients are “blindblind” to their treatment - ” to their treatment - butbut,,

the researcher is the researcher is NOTNOT!!

The patients are “The patients are “blindblind” to their treatment - ” to their treatment - butbut,,

the researcher is the researcher is NOTNOT!!

The solution is…


Medical research -- The double blind test!



• The list of patients and their treatment is prepared and maintained by someone who is NOT participating in the research. The list is sealed in the “black box”.

• The list can be retrieved from the “black box” only after all the analysis is completed.

The patients are “The patients are “blindblind” to their treatment - ” to their treatment - andand,,

the researcher is “the researcher is “blindblind” to who is getting treated!” to who is getting treated!

The patients are “The patients are “blindblind” to their treatment - ” to their treatment - andand,,

the researcher is “the researcher is “blindblind” to who is getting treated!” to who is getting treated!

1948 Am. Heart Jrnl., XXXVI, 529.

1950 Am. Jrnl. Med., IX, 142/1.





Consider a simple analysis -- “Was the medicine effective?”

• The data on all patients will indicate whether the medication was effective for each patient.

• When this data analysis is complete (and free from mistakes) and any disagreement among the researchers have been settled and any concerns about bias have been removed, i.e., then (and only then) -

• The “black box” can be opened and the list can be retrieved.

• The data on all patients is now grouped into two groups: group (a) and group (b)





The simple analysis -- “Was the medicine effective?”

• For what fraction of group (a) [fa] and group (b) [fb] was the medicine effective?• Do a statistical analysis to determine whether the difference in these two fractions is statistically significant - or whether it is consistent with a random occurrence. The result is now known!

• The researchers agree to accept the results obtained and no further analysis of the data is permitted - unless there has been a blunder (mistake) - in which case fix the blunder and report both results and the nature of the “fix”.



Blind analysis has become the standard methodology in clinical trials.

Blind analysis has been used in the physical sciences only in recent years.

Physicists (and astronomers) are, of course ---- Careful-- Quantitative-- Attentive to bias (systemtic errors)

Why should they need a blind analysis?



Ernest Rutherford (1934) --

“It seems to me that in some way it is regrettable that we had a theory of the positive electron before the beginning of the experiments. Blackett* did everything possible not to be influenced by the theory, but the way of anticipating results must inevitably be influenced to some extent by the theory. I would have liked it better if the theory had arrived after the experimental facts had been established.” Ernest Rutherford, Proc. Solvay Conference,

(Gauthier-Villars, Paris 1934), p 177.* Nobel prize in physics for discovering the positron (1948)



Gregor Mendel (1865) --

The classic case is Gregor Mendel’s work on inheritance.

84 degrees of freedom, giving a probability that hewould have measured this well to be only 7x10-5 !!

In every case the data agreed with the theoretical ratios within less than the standard errors.Taking the whole together, 2 was 41.6 on



Modern examples --

From nuclear and subnuclear physics ---

Reason: It’s the field in which the blind analysis techniques have been widely used - and -

It is the field with which I am most familiar.


B-meson lifetime ratio

The mean ratio has a 2 of 4.5 for 13 degrees of freedom; P=0.985

8 Major experiments

Over 2000 physicists

Bottom line:

The agreement appears to be

too good!!



An example of a “search” for an uncommon occurrence

A reaction that violates the Standard Modelconservation of lepton number

If you do not find the reaction, you have not shown the Standard Model to be incorrect

If you do find the reaction, you had better be very sure that you have got it right!!!


MEGA

Search for e

Question: When and How are the values of determined?

Signal: Ee = E = 52.8 MeV

Boxes represent 2 boundaries

ExamplesExamplesExamplesExamples


MEGA

Search for e

Blind Analysis: The boundaries aredetermined before the analysis of

(most of) the data!

Signal: Ee = E = 52.8 MeV

Boxes represent 2 boundaries



TWIST

High precision ee


A search for physics beyond (not included in) the Standard Model.

p = 29.8 MeV/c

Measure: pe()


Nu

mb

er

X = E/Emax

N(x) = a + bxa,b from theory


Simulation = “Monte Carlo”

Simulation ofthe experiment

using a,b

Nexp(x) = a’ + b’x

Analyzed expimental

data

Find a’ & b’ bycomparison of

real data with

simulateddata

Find a’ & b’ bycomparison of

real data with

simulateddata

The detector system distorts the distribution


Muon decay spectrum

Current SM = -0.007 ± 0.013 0 = 0.7518 ± 0.0026 3/4 = 0.7486 ± 0.0026 ± 0.0028 3/4P = 1.0027 ± 0.0079 ± 0.0030 1

)1(3)34(3

2)33(

)(cos0

2

2

xx

xxx

dxdx

d−+−+−∝

Γ

⎥⎦⎤

⎢⎣⎡ −+−+ )34(

3

2)1(cos xxP δθξμ



EnergyCos()


Muon decay spectrum from Standard Model


EnergyCos()

TWIST measured spectrum


EnergyCos()

Standard Model spectrum

Monte Carlo computer code

,,,


EnergyCos()

TWIST simulated spectrum

,,,

EnergyCos()



Problem!

This spectrum is the S.M. spectrum -- nothing is

hidden; the experimenter is not “blinded”.

Compare spectra ’,’,’,’

’,’,’,’


EnergyCos()



EnergyCos()

Standard Model spectrum

Monte Carlo computer code

o,o,o,o

Unknown spectrum

o,o,o,o are generated randomly, are encrypted, stored secretly, and used to generate the simulated data. Nobody knows what the offsets from ,,, are.

o,o,o,o are generated randomly, are encrypted, stored secretly, and used to generate the simulated data. Nobody knows what the offsets from ,,, are.


EnergyCos()

TWIST simulated spectrum

?

EnergyCos()



Compare spectra to get , , ,

’,’,’,’

= ’ - o

= ’ - o = ’ - o = ’ - o



Before you can open the black box:

• Get all computer codes working and tested

• Identify all sources of systematic error and evaluate the size of each one

• Take all of the data you will need including data to help estimate the systematic errors

• Be sure that the Monte Carlo programs accurately simulate your experiment or you will have false values

• Analyze all of the data you intend to use to get the result

o,o,o,o



o,o,o,o = ’ - o

= ’ - o = ’ - o = ’ - o

MeasureCalculate the results

Compare with Standard Model predictions

Write the paper



Let’s look at one of your experiments Let’s look at one of your experiments

from PHYS-245from PHYS-245



Present accepted value: c = 299,792,458 m/sc 3 x 108 m/s

A measurement of the speed of light

12

€

T

osc

€

t =θ

2πT ⇒ c =

D

t

D



Present accepted value: c = 299,792,458 m/sc 3 x 108 m/s

A measurement of the speed of light

12

€

T

osc

€

t =θ

2πT ⇒ c =

D

t

D

€

To = T +δT

€

To

€

co

Blinded


ReflectionsReflectionsReflectionsReflections

1. Guide to a blind analysis: If my answer were to come out to be six standard deviations from the expected result, what would I do? Make the list, and then -do all of that before you look at the answer!

2. A blind analysis is intended to guard against experimenter bias. It will not guard against fraud; that requires integrity and honesty.

3. A blind analysis is likely not the fastest way to an answer.



4. It is not always possible to achieve perfect “blindness” -- e.g., drug testing.

5. Don’t need to plan everything in the analysisbefore beginning; just keep the answer hidden.

6. A blind analysis may not work for every experiment -but it is worth investigating before you begin.

7. In a blind analysis you want to hide the answer from anyone else who might want to offer advice that may be based on the answer.


Thankyou


8. A blind analysis removes the “comfort factor” - knowing what answer you are getting so you can make changes, do more analysis, repeat measurements, etc., if the answer is not what you expect.

But, that may not be the best science!But, that may not be the best science!But, that may not be the best science!But, that may not be the best science!

a blind analysis you are not allowed to peek! prof. donald koetke senior research professor of...

Documents

examples of blind analysis

experimenter bias

special subtle bias

systematic error

vu colloquium2 outlineoutline

vu colloquium3 outlineoutline

vu colloquium4 outlineoutline

vu colloquium5 outlineoutline