practical statistics for neuroscience miniprojects steven kiddle slides & data :

Practical statistics for Neuroscience miniprojects

Steven Kiddle

Slides & data : http://bit.ly/1Jaor2r

We are unlikely to finish all the slides

Keep them, they may be helpful for your miniproject

Lecture outline

• Taught component– How to present statistics– Hypothesis testing– Normal distribution

• Practical component– Plots– Statistical tests– Multiple testing corrections

Taught component

Why are statistics important?

• Help to make science more repeatable and objective

• Help you to interpret your results• Help you to assess the level of evidence you

have supporting a hypothesis

• A vital skill for a scientific career!

How to report statistics

• Always report:(1) Statistical software you used(2) Statistical tests you used(3) Significance level you used(4) Sample size

I checked these in 17 randomly chosen neuroscience project posters

How not to report statistics!

• I found that:(1) 16/17 didn’t report the statistical software used(2) 11/17 didn’t report the statistical tests used(3) 9/17 didn’t report the significance level used(4) 2/17 didn’t report the sample size!

Commonly used analysis methods

• Plotting:- Box plots- Line plots

• Hypothesis testing– T-test– ANOVA– Chi-squared

Hypothesis testing

• Two types of hypothesis– Null hypothesis (H0)• Usually that there are no differences between groups

or that two variables are unrelated– Example : (H0) Smoking and lung cancer are unrelated

– Alternative hypothesis (H1)• There are differences between groups, or that two

variables are related– Example : (H1) Smoking and lung cancer are associated

Significance levels

• You accept the alternative hypothesis if the chance of your data being generated under the null hypothesis (the ‘p-value’) is beneath a pre-specified significance level α– Typically α = 0.05

• You should state the significance threshold you use in your report

Multiple hypothesis testing I• Suppose you have a significance threshold of α = 0.05

• Suppose that you measure 100 variables that are NOT related to a disease

• You perform 100 hypothesis tests to compare your variables to disease state• H0 : Variable is not affected by disease state

• H1 : Variable is affected by disease state

• For how many variables do you expect to reject the null hypothesis (H0) even though its true?

Multiple hypothesis testing II

• α = 0.05 means that if the null hypothesis (H0) is true, we would expect to reject it 5% of the time

• So if H0 is true and we did 100 tests, we would expect to reject H0 5 times by chance alone• That is bad, these findings will not replicate

• How do we stop it?• Multiple testing corrections

Multiple testing corrections

• Bonferroni correction– If we want α = 0.05 , instead use α = 0.05/n

where n is the number of tests you want to use– So for 100 tests, we would use α = 0.0005,

and would only have 5% chance of any test rejecting the null hypothesis

• Benjamini-Hochberg correction– Popular alternative

Normal distribution

Tests that rely on assumptions of normality

• T-tests• ANOVA / linear models

How to check if you data is normally distributed

• Histograms• Statistical tests

• Can apply to data

• But better to apply to residuals of the models– For t-test, that means looking at the groups separately– For ANOVA, that means extracting residuals from the

model

What do you do if your data is not normally distributed?

• If sample size is really small– Nothing you can do – use test anyway

• If data is skewed– Transform data (e.g. log? square root?)

• Use non-parametric tests– Mann-whitney U instead of T-test– Spearman’s Rank Correlation

• Last resort - Remove outliers?– Systematically and preferably only if you know what causes them

How to present plots

• Label both axes– Large enough to read

• Show units• If using stars (*) for significance levels, explain

what *, **, *** means

Lunnon et al., (2012) Journal of Alzheimer’s Disease

How to present statistics I

• Say what statistical software used, e.g.– SPSS, STATA, R, MATLAB, etc

• Say what the sample size is

• Say what statistical test is being performed– T-test, ANOVA, chi-squared, etc

• Say what significance level you are using for the study– Think, is it appropriate given my sample size and number of

hypotheses being tested?

How to present statistics II

• Report p-value– And/or multiple testing corrected p-value• E.g. Q-values for Benjamini-Hochberg

• Report coefficient (β), and ideally it’s standard error for each reported statistic– This can be more informative than a p-value,

especially for small datasets

How to present statistics III

A more complete guide, tailored to SPSS and specific tests is given at:

http://statistics-help-for-students.com/

Be cautious in your interpretations• Correlation does not equal causation!

• Can you hypothesise a mechanism by which causation could occur?

Why does correlation not equal causation?

• It looks like the variables are correlated when they are not– How this happens?• By chance, especially when multiple testing is

performed but not corrected for

• Variables are truly correlated but there is either:– Reverse causation– Confounding by other variables

Confounding

Statistical software

• Excel– Point and click, quite limited

• SPSS– Point and click, a little limited

• STATA– Command line

• R, MATLAB, etc– Command line, very useful, steep learning curve

http://www.r-project.org/http://www.rstudio.com/

R introduction

Practical component - SPSS

Data is faked to show large differences, real data will not be so

clear cut

Outline

• Data• Tests for normality• Plots• T-test• ANOVA• Chi-squared• Non-parametric tests

Data

• Create folder in ‘My Documents’• Download data and save in your new folder:

Slides & data : http://bit.ly/1Jaor2r

• Open zip folder• Double click on ‘neuroscience_example.sav’ to

open SPSS

Introduction to the data

• 5 variables– a , b , c , d , e

• 2 are binary– a , b

• 3 are continuous– c , d , e

Normality checks I

• Need to check data is normally distributed when we want to apply– T-test– ANOVA– Linear regression

Normality checks II

Let’s see if the variable ‘d’ is normally distributed

Normality checks III

Can see that the data has two peaks

Rejects the null hypothesis that the data is normally distributed

Normality checks IIINow we take into account variable ‘a’, we find that ‘d’ is normally distributed when we take into account ‘a’

Plots

• Histograms (shown in normality check)– Show distribution of a continuous variable

• Boxplots– Show the distribution of a continuous variable between

groups

• Line plot/scatter plot– Shows the relationship between two continuous

variables

Generating a boxplot I

Generating a boxplot II

Generating a boxplot III

Generating a boxplot IV

Double click on plot to label axis

Labelling a plot I

Double click to change label

How to present plots

• Label both axes– Large enough to read

• Show units• If using stars (*) for significance levels, explain

what *, **, *** means

Lunnon et al., (2012) Journal of Alzheimer’s Disease

Labelling a plot II

Saving plots

Rename document and save in your folderYou can now open the document and extract the plot as an image

Boxplot exercises

• Make a few more boxplots comparing binary variables to continuous variables

• Try adding labels• Try saving• Try to interpret the boxplot– Do you see differences between the groups

Generating a line plot I

Generating a line plot II

T-test

• Compares a binary variable (yes/no) to a continuous variable

• Example null hypothesis– Mean height is the same across males and females

• Example alternative hypothesis– Mean height is different between males and

females

Performing a t-test I

Performing a t-test II

Descriptive statistics

Performing a t-test II

Levene test null hypothesis : the variance (and standard deviation) of the two groups are the same.

At the significance level 0.05 we can reject the null hypothesis.

Therefore we should use the second row (‘Equal variances not assumed’).

Performing a t-test II

T-test null hypothesis : the mean of the two groups are the same

At the significance level 0.05 we can reject the null hypothesis (p-value is less than 0.001).

I.e. the data supports the fact that the variable ‘d’ is different between the two groups.

How to report findings (option A)

• A two-sample t-test assuming unequal variances performed in SPSS showed differences in variable ‘d’ between groups 0 (N = 19) and 1 (N = 31) in variable ‘a’ at the 5% significance level (mean difference = 1.6, standard error = 0.89, p-value < 0.001 ).

How to report findings (option B)

• Materials and methods:– Statistical analysis

• Statistical analysis was performed in SPSS 20.• Group differences were analysed using two sample t-test

assuming unequal variances.• A significance level of 5% was applied to all hypothesis tests.

• Results:– Variable ‘d’ was found to differ between groups 0 (N = 19)

and 1 (N = 31) in variable ‘a’ (mean difference = 1.6, standard error = 0.89, p-value < 0.001 ).

How that might look

• Materials and methods:– Statistical analysis

• Statistical analysis was performed in SPSS 20.• Group differences were analysed using two sample t-test

assuming unequal variances.• A significance level of 5% was applied to all hypothesis tests.

• Results:– Change in blood glucose levels differed between males (N

= 19) and females (N = 31) (mean difference = 1.6 ng/ml , standard error = 0.89, p-value < 0.001 ).

How to present statistics

A more complete guide, tailored to SPSS and specific tests is given at:

http://statistics-help-for-students.com/

One-way ANOVA

• Extension of t-test idea• Compares a binary variable (yes/no) to a several

variables, continuous or nominal (including binary)• Example null hypothesis– Mean height is the same across males and females,

regardless of age• Example alternative hypotheses– Mean height is different between males and females– Mean height differs across ages

Performing ANOVA I

Performing ANOVA II

Which variables differ between variable ‘a’ group 0 and 1?

How would you perform a Bonferroni multiple testing correction?

How would you report these findings? (clue: http://statistics-help-for-students.com/ )

Chi-squared test

• Compares multiple nominal variables• Example null hypothesis– Lung cancer and smoking are unrelated

• Example alternative hypotheses– Smokers are more likely to have lung cancer

Performing chi-squared I

Performing chi-squared II

How would you report these findings? (clue: http://statistics-help-for-students.com/ )

Non-parametric tests

• If a continuous variable is not normally distributed, using parametric tests may give you misleading results– T-test and ANOVA are parametric tests

• Solution, use non-parametric tests– Such as• Mann-Whitney U• Spearman’s Rank Correlation

Mann-Whitney U

• A non-parametric equivalent of a t-test

Performing Mann-Whitney U I

Performing Mann-Whitney U II

How would you report these findings? (clue: http://statistics-help-for-students.com/ )

Spearman’s Rank Correlation

• A non-parametric equivalent of correlation, or a one-way ANOVA between a binary variable and a continuous variable

Performing Spearman’s Rank Correlation I

Performing Spearman’s Rank Correlation II

How would you report these findings? (clue: http://statistics-help-for-students.com/ )