1 1 the scientist game scott s. emerson, m.d., ph.d. professor of biostatistics university of...
Post on 19-Dec-2015
217 views
TRANSCRIPT
![Page 1: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/1.jpg)
11
The Scientist GameThe Scientist Game
Scott S. Emerson, M.D., Ph.D.
Professor of Biostatistics
University of Washington
Scott S. Emerson, M.D., Ph.D.
Professor of Biostatistics
University of Washington
© 2002, 2003 Scott S. Emerson, M.D., Ph.D.© 2002, 2003 Scott S. Emerson, M.D., Ph.D.
![Page 2: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/2.jpg)
22
OverviewOverview
• A simplified universe– One dimensional universe observed over time
– Each position in the universe has an object
– Goal is to discover any rules that might determine which objects are in a given location at a particular time
• A simplified universe– One dimensional universe observed over time
– Each position in the universe has an object
– Goal is to discover any rules that might determine which objects are in a given location at a particular time
![Page 3: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/3.jpg)
33
ObjectsObjects
• Objects in the universe have only three characteristics, each with only two levels
• Color: White or Orange• Size: BIG or small• Letter: A or B
A a B b A a B b
• Objects in the universe have only three characteristics, each with only two levels
• Color: White or Orange• Size: BIG or small• Letter: A or B
A a B b A a B b
![Page 4: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/4.jpg)
44
Universal LawsUniversal Laws
• The level of each characteristic (color, size, letter) for the object at any position in the universe is either
• completely determined by the prior sequence of that characteristic for objects at that position,
OR• is completely random (anything is permissible)
• (No patterns involving probabilities less than 1)• (Adjacent positions have no effect)
• The level of each characteristic (color, size, letter) for the object at any position in the universe is either
• completely determined by the prior sequence of that characteristic for objects at that position,
OR• is completely random (anything is permissible)
• (No patterns involving probabilities less than 1)• (Adjacent positions have no effect)
![Page 5: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/5.jpg)
55
Examples of Universal LawsExamples of Universal Laws
• Color only:
b a A b a a a B A a A A
• The next object in the sequence must be white, but any size or letter will do:
a A b B
• Color only:
b a A b a a a B A a A A
• The next object in the sequence must be white, but any size or letter will do:
a A b B
![Page 6: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/6.jpg)
66
Examples of Universal LawsExamples of Universal Laws
• Size and letter:
A a B b A a B b A a B b
• The next object in the sequence must be a big A, but any color will do
A A
• Size and letter:
A a B b A a B b A a B b
• The next object in the sequence must be a big A, but any color will do
A A
![Page 7: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/7.jpg)
77
Examples of Universal LawsExamples of Universal Laws
• Size only:
B a B b A b B a B b A a
• The next object in the sequence must be big, but any color or letter will do:
A B A B
• Size only:
B a B b A b B a B b A a
• The next object in the sequence must be big, but any color or letter will do:
A B A B
![Page 8: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/8.jpg)
88
Examples of Universal LawsExamples of Universal Laws
• No discernible pattern (in available data):
A a b a A b B a B B A a
• If there is truly no deterministic pattern, then any object may appear next:
a A b B a A b B
• No discernible pattern (in available data):
A a b a A b B a B B A a
• If there is truly no deterministic pattern, then any object may appear next:
a A b B a A b B
![Page 9: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/9.jpg)
99
Scientific TaskScientific Task
• Goal is therefore to decide for some position– whether a rule governs the level of each
characteristic, and– if so, what that rule is (pattern to the
sequence)
• Goal is therefore to decide for some position– whether a rule governs the level of each
characteristic, and– if so, what that rule is (pattern to the
sequence)
![Page 10: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/10.jpg)
1010
Hypothesis GenerationHypothesis Generation
• Initially we have observational data gathered over time– Amount of available information varies from
position to position– We want to identify some position that is the
most likely to be governed by some deterministic rule
• Initially we have observational data gathered over time– Amount of available information varies from
position to position– We want to identify some position that is the
most likely to be governed by some deterministic rule
![Page 11: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/11.jpg)
1111
Observational DataObservational Data
TimePstn -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 …114. b a A b a a B a A A a b ? ?115. A a B b A a b b A a a A ? ?116. B b A a a b ? ?117. b B A b b b a ? ?118. A b B A B b A B B a B B ? ?119. B b B b A a A a B A b ? ?120. B B b ? ?121. B A a B b b a b A ? ?…
TimePstn -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 …114. b a A b a a B a A A a b ? ?115. A a B b A a b b A a a A ? ?116. B b A a a b ? ?117. b B A b b b a ? ?118. A b B A B b A B B a B B ? ?119. B b B b A a A a B A b ? ?120. B B b ? ?121. B A a B b b a b A ? ?…
![Page 12: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/12.jpg)
1212
Observational DataObservational Data
TimePstn -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 …
118. A b B A B b A B B a B B ? ?
…
TimePstn -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 …
118. A b B A B b A B B a B B ? ?
…
![Page 13: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/13.jpg)
1313
Next Step? Next Step?
• Further observation?– Might take too long– Won’t really establish cause and effect
• Further observation?– Might take too long– Won’t really establish cause and effect
![Page 14: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/14.jpg)
1414
Experimentation Experimentation
• You can try to put an object in the position– If it cannot come next, it disintegrates and you
can try another– If it can come next, it stays and you can try a
different object to follow it
• You can try to put an object in the position– If it cannot come next, it disintegrates and you
can try another– If it can come next, it stays and you can try a
different object to follow it
![Page 15: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/15.jpg)
1515
Experimental Goal Experimental Goal
• You need to devise a series of experiments to discover– whether a deterministic rule governs the
sequence of objects at position 118, and– if there is such a rule, what it is
• You need to devise a series of experiments to discover– whether a deterministic rule governs the
sequence of objects at position 118, and– if there is such a rule, what it is
![Page 16: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/16.jpg)
1616
Real WorldReal World
• Problem:– You must buy objects to experiment with
• (apply for a grant)
• Question: – What object should you try next in the
sequence in order to determine the rule?
• Problem:– You must buy objects to experiment with
• (apply for a grant)
• Question: – What object should you try next in the
sequence in order to determine the rule?
![Page 17: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/17.jpg)
1717
Possible ExperimentsPossible Experiments
Time
Pstn -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2
118. A b B A B b A B B a B B ? ?
Possible Experiments a A b B a A b B
• Which experiment do you do first?
Time
Pstn -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2
118. A b B A B b A B B a B B ? ?
Possible Experiments a A b B a A b B
• Which experiment do you do first?
![Page 18: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/18.jpg)
1818
Results of ObservationResults of Observation
Time
Pstn -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 118. A b B A B b A B B a B B ? ?
• We identified position 118 which had some regular patterns– Color cycle of length 2: (orange, white)– Size cycle of length 4: (big, little, big, big)– Letter cycle of length 3: (A, B, B)
Time
Pstn -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 118. A b B A B b A B B a B B ? ?
• We identified position 118 which had some regular patterns– Color cycle of length 2: (orange, white)– Size cycle of length 4: (big, little, big, big)– Letter cycle of length 3: (A, B, B)
![Page 19: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/19.jpg)
1919
Define HypothesesDefine Hypotheses
• Deterministic pattern versus random chance for each characteristic– Recognize that some or all observed patterns
might be coincidence• Chance observation of a pattern for a single
characteristic (e.g., color) with sample size 12 (assuming each level equally likely)
– 1 out of 1024 for a cycle of length 2– 1 out of 512 for a cycle of length 3– 1 out of 256 for a cycle of length 4
• Deterministic pattern versus random chance for each characteristic– Recognize that some or all observed patterns
might be coincidence• Chance observation of a pattern for a single
characteristic (e.g., color) with sample size 12 (assuming each level equally likely)
– 1 out of 1024 for a cycle of length 2– 1 out of 512 for a cycle of length 3– 1 out of 256 for a cycle of length 4
![Page 20: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/20.jpg)
2020
Possible HypothesesPossible Hypotheses
• Assuming sufficient data to see any rule118. A b B A B b A B B a B B ? ? HypothesesColor, Size, and LetterColor, SizeColor, LetterSize, LetterColorSizeLetterAll coincidence
• Assuming sufficient data to see any rule118. A b B A B b A B B a B B ? ? HypothesesColor, Size, and LetterColor, SizeColor, LetterSize, LetterColorSizeLetterAll coincidence
![Page 21: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/21.jpg)
2121
Most Popular First ChoiceMost Popular First Choice
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter Color, Size Color, Letter Size, Letter Color Size Letter All coincidence
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter Color, Size Color, Letter Size, Letter Color Size Letter All coincidence
![Page 22: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/22.jpg)
2222
Most Popular First ChoiceMost Popular First Choice
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size Color, Letter Size, Letter Color Size Letter All coincidence
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size Color, Letter Size, Letter Color Size Letter All coincidence
![Page 23: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/23.jpg)
2323
Most Popular First ChoiceMost Popular First Choice
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size +Color, Letter Size, Letter Color Size Letter All coincidence
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size +Color, Letter Size, Letter Color Size Letter All coincidence
![Page 24: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/24.jpg)
2424
Most Popular First ChoiceMost Popular First Choice
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size +Color, Letter +Size, Letter Color Size Letter All coincidence
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size +Color, Letter +Size, Letter Color Size Letter All coincidence
![Page 25: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/25.jpg)
2525
Most Popular First ChoiceMost Popular First Choice
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size +Color, Letter +Size, Letter +Color Size Letter All coincidence
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size +Color, Letter +Size, Letter +Color Size Letter All coincidence
![Page 26: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/26.jpg)
2626
Most Popular First ChoiceMost Popular First Choice
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size +Color, Letter +Size, Letter +Color +Size Letter All coincidence
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size +Color, Letter +Size, Letter +Color +Size Letter All coincidence
![Page 27: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/27.jpg)
2727
Most Popular First ChoiceMost Popular First Choice
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size +Color, Letter +Size, Letter +Color +Size +Letter All coincidence
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size +Color, Letter +Size, Letter +Color +Size +Letter All coincidence
![Page 28: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/28.jpg)
2828
Most Popular First ChoiceMost Popular First Choice
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size +Color, Letter +Size, Letter +Color +Size +Letter +All coincidence
118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size +Color, Letter +Size, Letter +Color +Size +Letter +All coincidence
![Page 29: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/29.jpg)
2929
Most Popular First ChoiceMost Popular First Choice
• A noninformative experiment118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size +Color, Letter +Size, Letter +Color +Size +Letter +All coincidence +
• A noninformative experiment118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses AColor, Size, and Letter +Color, Size +Color, Letter +Size, Letter +Color +Size +Letter +All coincidence +
![Page 30: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/30.jpg)
3030
Next Worse ChoiceNext Worse Choice
• If all hypotheses equally likely, a 7-1 split118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses A bColor, Size, and Letter + -Color, Size + -Color, Letter + -Size, Letter + -Color + -Size + -Letter + -All coincidence + +
• If all hypotheses equally likely, a 7-1 split118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses A bColor, Size, and Letter + -Color, Size + -Color, Letter + -Size, Letter + -Color + -Size + -Letter + -All coincidence + +
![Page 31: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/31.jpg)
3131
Other Suboptimal ExperimentsOther Suboptimal Experiments
• If all hypotheses equally likely, a 6-2 split118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses A b b B aColor, Size, and Letter + - - - -Color, Size + - - - -Color, Letter + - - - -Size, Letter + - - - -Color + - + - -Size + - - + -Letter + - - - +All coincidence + + + + +
• If all hypotheses equally likely, a 6-2 split118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses A b b B aColor, Size, and Letter + - - - -Color, Size + - - - -Color, Letter + - - - -Size, Letter + - - - -Color + - + - -Size + - - + -Letter + - - - +All coincidence + + + + +
![Page 32: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/32.jpg)
3232
Optimal ExperimentsOptimal Experiments
• Based on a binary search118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses A b b B a B a AColor, Size, and Letter + - - - - - - -Color, Size + - - - - + - -Color, Letter + - - - - - + -Size, Letter + - - - - - - +Color + - + - - + + -Size + - - + - + - +Letter + - - - + - + +All coincidence + + + + + + + +
• Based on a binary search118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses A b b B a B a AColor, Size, and Letter + - - - - - - -Color, Size + - - - - + - -Color, Letter + - - - - - + -Size, Letter + - - - - - - +Color + - + - - + + -Size + - - + - + - +Letter + - - - + - + +All coincidence + + + + + + + +
![Page 33: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/33.jpg)
3333
What If Data Insufficient?What If Data Insufficient?
• Suppose deterministic cycle length > 12118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses A b b B a B a AColor, Size, and Letter + - - - - - - -Color, Size + - - - - + - -Color, Letter + - - - - - + -Size, Letter + - - - - - - +Color + - + - - + + -Size + - - + - + - +Letter + - - - + - + +All coincidence + + + + + + + +Cycle length > 12 ? ? ? ? ? ? ? ?
• Suppose deterministic cycle length > 12118. A b B A B b A B B a B B ? ? Possible ExperimentsHypotheses A b b B a B a AColor, Size, and Letter + - - - - - - -Color, Size + - - - - + - -Color, Letter + - - - - - + -Size, Letter + - - - - - - +Color + - + - - + + -Size + - - + - + - +Letter + - - - + - + +All coincidence + + + + + + + +Cycle length > 12 ? ? ? ? ? ? ? ?
![Page 34: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/34.jpg)
3434
MoralMoral
• The goal of the experiment should be to “decide which” not “prove that”
• A well designed experiment discriminates between hypotheses– The hypotheses should be the most
important, viable hypotheses
• The goal of the experiment should be to “decide which” not “prove that”
• A well designed experiment discriminates between hypotheses– The hypotheses should be the most
important, viable hypotheses
![Page 35: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/35.jpg)
3535
MoralMoral
• All other things being equal, an experiment should be equally informative for all possible outcomes– In the presence of a binary outcome, use a
binary search• (using prior probability of being true)
– But may need to consider simplicity of experiments, time, cost
• All other things being equal, an experiment should be equally informative for all possible outcomes– In the presence of a binary outcome, use a
binary search• (using prior probability of being true)
– But may need to consider simplicity of experiments, time, cost
![Page 36: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/36.jpg)
3636
In the Presence of VariabilityIn the Presence of Variability
• We use statistics to quantify the precision of our inference– We will describe our confidence/belief in our
conclusions using frequentist or Bayesian probability statements
• We use statistics to quantify the precision of our inference– We will describe our confidence/belief in our
conclusions using frequentist or Bayesian probability statements
![Page 37: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/37.jpg)
3737
Statistical Experimental DesignStatistical Experimental Design
• I believe a scientific approach to the use of statistics is to– Decide a level of confidence used to construct
frequentist confidence intervals or Bayesian credible intervals
– Ensure adequate statistical precision (sample size) to discriminate between relevant scientific hypotheses
• The intervals should not contain two hypotheses that were to be discriminated between
• I believe a scientific approach to the use of statistics is to– Decide a level of confidence used to construct
frequentist confidence intervals or Bayesian credible intervals
– Ensure adequate statistical precision (sample size) to discriminate between relevant scientific hypotheses
• The intervals should not contain two hypotheses that were to be discriminated between
![Page 38: 1 1 The Scientist Game Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics University of Washington Scott S. Emerson, M.D., Ph.D. Professor of Biostatistics](https://reader030.vdocuments.site/reader030/viewer/2022013011/56649d2b5503460f94a015a9/html5/thumbnails/38.jpg)
3838
Impact on Statistical PowerImpact on Statistical Power
• I choose equal one-sided type I and type II errors– E.g., 97.5% power to detect the alternative in
a one-sided level 0.025 hypothesis test
• In this way, at the end of the study, the 95% CI will not contain both the null and alternative hypotheses– I will have discriminated between the
hypotheses with high confidence
• I choose equal one-sided type I and type II errors– E.g., 97.5% power to detect the alternative in
a one-sided level 0.025 hypothesis test
• In this way, at the end of the study, the 95% CI will not contain both the null and alternative hypotheses– I will have discriminated between the
hypotheses with high confidence