research methods: measurement
DESCRIPTION
lecture 6 from a college level research methods in psychology course taught in the spring 2012 semester by Brian J. Piper, Ph.D. ([email protected]) at Linfield College, includes categorical, ordinal, interval, and ratio levelsTRANSCRIPT
Statistics II: Measurement & Data Analysis
Brian J. Piper
Goals
• Operationalization• Psychometrics (Reliability & Validity)• Scales of Measurement• Hypothesis Testing• Advanced Topics– Power– Effect Size
Operationalization
• What areas of the brain are important for music appreciation?
• Important = Active– EEG (electrical activity)– PET (sugar use)– fMRI (oxygen use)
Operationalization 1: Spatial Function
• Do men and women differ in their spatial abilities?
• Spatial Function = mental rotation
Operationalization 2: Motor Function
• Do men and women differ in their fine-motor abilities?
• Motor Function = rotary pursuit
Reliability
• Consistency of measurement• Types– Test-re-test reliability
Time 1
Time 2
Split-Half Reliability
• Consistency of measurement on two-halves of test
• Foundations for repeated measurements
Even
Odd
Extension: Short-Form of Wisconsin (Berg) Card Sorting Test (BCST)
Extension: Short-Form of Wisconsin (Berg) Card Sorting Test (BCST)
r(205) = +0.77
Fox et al. (in review). J Biol Biomed Reports.
Validity
• Does a test measure what it claims to?• face “faith” validity: does it seem valid based on
intuition (non-numerical)
Criterion Validity
• Does performance on new measure match with older “gold standard” measure?
• Continuous Performance Tests Example
Reaction Time (Conner’s)
Reac
tion
Tim
e (P
EBL)
Construct Validity
• Does a test measure the construct it claims to?
• Convergent Validity: Does test A correlate (converge) with test B?
• Discriminant Validity: Does test A measure something different (discriminate) than test C?
Measurement Scales (Self-Test)Level Definition
Nominal
Ordinal
Interval
Ratio
Measurement Scales (Self-Test)Level Definition
Nominal categorical, e.g. sex
Ordinal ranking, e.g. Olympic medal
Interval equal spacing, e.g. IQ, ACT, SAT
Ratio true zero, e.g. Reaction Time
Hypothesis Testing
• Null hypothesis (H0): A = B
• Alternative hypothesis (HA): A ≠ B
Alpha
• The cut-off used to decide between H0 and HA
• Probability that finding is not due to chance (p value)
• .05: conventional• .10: liberal (some medical environments)• .01: conservative, large N
Alpha
P value obtained
Decision
.50 H0
.11 H0
.06 H0
.0500000001 H0
.0499999999 HA
Decision Making
HO is True HO is False
Fail to reject H0 Correct decision
Reject H0 Correct decision
Reality
Decision
Decision Making
HO is True HO is False
Fail to reject H0 Correct decision
Reject H0 Type I error Correct decision
Reality
Decision
Type I Error: rejecting H0 when it is true
Decision Making
HO is True HO is False
Fail to reject H0 Correct decision Type II error
Reject H0 Type I error Correct decision
Reality
Decision
Type I Error: rejecting H0 when it is trueType II Error: fail to reject H0 incorrectly
Publication Bias
• H0 results often don’t get shared• Reasons: – Journal prestige– Research ego– Higher standard
• Solution: registry?– Replication?
Solution 1: Effect Size Distribution
• A quantitative index of the magnitude of group difference’s
• Calculated as (Mean1 – Mean2)/SD
# St
udie
s
Solution 1: Effect Size Distribution
• A quantitative index of the magnitude of group difference’s
• Calculated as (Mean1 – Mean2)/SD
# St
udie
s
# St
udie
s
Solution 2: Power Analysis
• Power: the probability that a real effect will be detected
• Probability of Type II error: Beta• Power = 1 - Beta
N Power
50 0.40
100 0.70
500 0.80
1000 0.85
Other Terminology
• Population: all members of identifiable group• Sample: a subset of the population• Confidence Interval: inferential statistic,
contains range of where population mean sits
Margin of Error
• Is accurate if sample is representative of population.
Summary• Operationalization• Reliability & Validity (face, criterion, construct)• Scales of Measurement (nominal, ordinal,
interval, ratio)• Hypothesis Testing: Type I versus Type II error• Advanced Topics– Power– Effect Size