computer intensive statistical methods
DESCRIPTION
A little summary on several types of computer intensive statistical methods developed from the fantastic book by Malcolm Haddon titled "Modeling and Quantitative Methods in Fisheries" Authors: Daniele Baker and Stephanie JohnsonTRANSCRIPT
Computer Intensive ModelsDaniele and Steph
Playing Card Example
•52 cards are population▫25 card sample…. What is the mean, median, SE
•Randomization▫Randomly reallocate cards to groups (i.e. diamonds,
spades..)▫ 4 people divide up cards▫Do 1000+ times randomly allocating to the 4 groups▫Graph up distribution of parameter▫If your original parameter is outside of the 95% CI then
it is significantly different from random
Playing Card Example • JackKnife
▫Take 24 cards… (leave 1 out) and complete test statistics
▫Redo for all possible combinations (25 x)
•Bootstrapping▫Pick out 25 cards, but put them back each time▫Calculate new parameters
Redo 1000+ times▫If your sample parameter falls within the 95% CI of
the distribution, then it isn’t statistically different from random
Playing Card Example
•Monte Carlo▫ Find a model that would fit the card trend
Relative frequency plot, examine shape▫ Randomly select value for model parameters or data (cards
picked) Complete 1000+ times Analyze your parameters to fit the data vs. random generated
parameters
http://www.vertex42.com/ExcelArticles/mc/MonteCarloSimulation.html
COMPARISONRandomize Jackknife Bootstrap Monte Carlo
With Replacement
No No Yes Yes
Exact P-values Yes Likely no Yes Yes
Resample a theoretical PDF
Parametric
Resample an empirical
distribution
Yes Yes Non-parametric
Non-parametric
Good to… Deal with unknown
distribution
Detect bias, calc. SE, good
for biases parameters
Calc. sample size for exp.
Design, CI, SE and Test Hypot.
Flexible, generic, SE,
CI, Test Hypot.
Good for sparse data sets
Limitations Can’t calc SE, or CI (weak)
Bad CI
4 Methods
•Randomization▫Ho: each group of obs. is a random sample of 1 pop.▫Must be characterized by a test stat▫Combine all groups, then reallocate, and compare
Repeat 1000+ times▫Compare obs. Test stat with empirical distribution of
that test stat given available data. A sig diff is when obs. test stat is beyond empirical
distribution
4 Methods
• Jackknife▫Sample could be from one arm of the distribution▫Subset data (for all combinations of all data minus 1
pt) (total = n-1)▫Calculate pseudo-values , diff. btw. this and obs =
estimate of bias▫Good when estimating something other than mean▫Calculate Jackknife SE and parameter of interest
CI can be fitted but issues with DFs
4 Methods
•Bootstrapping▫Random samples of the observation (with
replacement Each treated as a separate random sample
▫Should = the distribution if you had repeatedly sampled the original population
▫Provides better CI than the Jackknife Can determine SE, CI and test hypotheses
▫Have been used for multiple regression and stratified sampling
4 Methods
•Monte Carlo▫Mathematical model of the situation + model
parameters▫Randomly select variable, parameter or data values
and then use to determine model output Do 1000+ times and use to test hypotheses or
determine Confidence Intervals▫*Resampling from a theoretical distribution▫Compare observations with data from a model of the
system + for use in risk assessment
Chapter 5: Randomization Chapter 5: Randomization TestsTests•ANOVA (e.g.) vs. Randomization
▫Assumptions•Hypothesis testing
▫Determinations of likelihood that observations in nature could have arisen by chance
▫Relativity: group 1 vs. group 2, hypothesis
Standard Significance Testing
• Test statistic (e.g. t-test)• Significant difference• Statistically 3 things
needed to test hypothesis:
1. Formally stated hypothesis (Ho and Ha)
2. Test statistic: t-test, F-ratio, r correlation, etc.
3. Means of generating PD of test statistic under assumption Ho is true
•idrc.ca
idrc.ca
•Observed vs. expected values (d.f., etc.)•Determine how likely observed values,
assuming Ho to be true
•α-value: 0.05, 0.01, 0.001
0
0.2
0.4
0.6
0.8
1
0 5 10 15 20 25
X2 Statistic
Prob
abili
ty
α 0.05 = ~15
However…• Test statistics are not valid if assumptions are
falsely made• Can, therefore, reject a real difference in data or
accept a non-existent difference• Problem with theoretically derived PDF:
▫ If test statistic is not significant, cannot tell without further analyses if the test failed b/c:1. Samples are not independently distributed (thing being tested)
or,2. The data failed to conform to the assumptions necessary for the
validity of the test (e.g. samples were not normally distributed)
This is where randomization tests come into play…
Significance Testing by Randomization
Independent of any determined parametric PDF
Generates empirical PDF for the test-statistic
Given a null hypothesis, the expected PDF for a test statistic can be generated by repeatedly randomizing the data with respect to sample group membership and recalculating the test statistic
1. Done many times (min. 1000)2. Test statistic values tabulated3. Compared with original value from un-
randomized data4. If original value is unusual relative to
permutations, Ho can be rejected
The Three R’s
sharonlanegetaz.v2efoliomn.mnscu.edu/Research
Key Points•Essentially, the null-hypothesis is that the
groups being compared are random samples from the same population
•Thus, a test of significance is an attempt to determine whether observed samples could have been randomly drawn from the same population
•Answer = probability
▫PROBABILITY = possibly from same pop. (never claim definitively)
▫ PROBABILITY = not likely from same pop.
Ex: Fish length for in-shore fish vs. off-shore fish•Ha: In-shore fish are smaller on average than off-
shore fish
0 50 100 150 200 250 300
Fork Length
Off-shore
In-shore
Ex. 5.2•Randomization can be used to test mean
difference•Original mean difference occurred 25 x’s out of
1000•What does this tell us about the data?•Weight of evidence, not significant difference
(e.g. p=0.05)
0100200300400500600700800900
1000
0 5 10 15 20 25 30 35 40 45 50 55
Abs(Mean Difference)
So
rted
Ran
do
miz
atio
n
Rep
lica
tes
p=0.025
Selection of a Test Statistic•Chosen based on sensitivity to the hypothetical
property being checked or compared (e.g. mean vs. median (ex.5.3))
•Precaution should be taken when selecting a non-standard test statistic
•Multivariate comparisons•Determine exactly which hypothesis is being
tested by the test statistic▫“When in doubt, be conservative”
•Ex 5.3
Ideal Test Statistics
•Greatest statistical power•Significance – probability of making a Type I
error•Power – probability of making a Type II error•Unbiased – using a test that is more likely to
reject a false hypothesis than a true one
No difference
Null true
Difference exists
Null False
Null accepted OK TYPE II ERROR
Null rejected TYPE I ERROR OK
Randomization of Structured Data•Restricted to comparison tests (cannot be used
for parameter estimation)•Differences in variation – randomize residuals
instead of data values•Basic rule: unbalanced and highly non-normal
data a randomization procedure should be used•Question: what should be randomized?
▫Raw data,▫Sub-set of raw data, or▫ residuals from model
Take Home Message…
With structured or non-linear data, care needs to be taken in what components should be randomized
Summary•Randomization requires less assumptions
than standard parametric stats•Significance tests test whether the
observed samples could be from the same pop.
•State hypotheses and determine significance level
•Test statistics that yield the greatest power should be utilized