validity and reliability dr. voranuch wangsuphachart dept. of social & environmental medicine
DESCRIPTION
Validity and Reliability Dr. Voranuch Wangsuphachart Dept. of Social & Environmental Medicine Faculty of Tropical Medicine Mahodil University 420/6 Rajvithi Road Bangkok 10400, THAILAND E-mail. Objective: at the end of lecture, student would be able to:. - PowerPoint PPT PresentationTRANSCRIPT
Validity and Reliability
Dr. Voranuch WangsuphachartDept. of Social & Environmental Medicine
Faculty of Tropical MedicineMahodil University
420/6 Rajvithi RoadBangkok 10400, THAILAND
1. Know concepts & definition of validity & reliability
2. List importance and impact of validity & reliability
3. Specify strategies to assess validity & reliability
4. List strategies to enhance validity & reliability
5. Describe major types of bias
Objective: at the end of lecture, student would be able to:
1. Validity 2. Reliability
• Definition and synonyms• Important points• Accessing validity & reliability• Strategies to enhance validity & reliability
3. Major types of bias
Contents:
Medical or epidemiological study, major consideration is to obtain:
Valid measurement Reliable measurement
on the exposure factors and outcomes of interest in the study population
“WITHOUT BIAS and ERRORS” or to minimize them to the least as possible
To achieve a high standard quality study:
Ensure right answers to study questions
Good the study design
Valid and reliable the measurements . Control for any possible bias
Good cooperation between * research group and * study population
Screening for fasting blood cholesterol profile among people
x1 x2 x3
x5x6
X11, x12, x13 X21, x22, x23X31, x32, x33
X41, x42, x43 X51, x52, x53X61, x62, x63
x4
Screening for fasting blood cholesterol profile among people
x1 x2 x3
x5 x6x4
X11 X21 X31
X41 X51 X61
Instrument or Research Tool
• “equipment hard ware”– a red blood cell counter – a PH meter
– an electronic weighing machine • “paper ware”
– a questionnaire
– a weekly diet diary • “people ware”
– observers/investigators
– technicians
How good is the instrument or tool?
• instrument• tool
measurement
– without bias
or error– minimize bias
• true value truth
• measurement
– valid/accurate
– precise/reliable
What is accuracy & precision?
• What do you think of first when talking about validity & reliability?
• What is the different between validity & reliability?
• Why are validity & reliability important in conducting any medical research
- both in laboratory & field setting?
PRECISION
DEFINITION :
A precise measurement in
one that has nearly the same value each time
it is measured
SYNONYM
• reliability
• repeatability
• reproducibility
• consistency
• agreement
IMPORTANT POINTS
• precision depends on:
– sample size– efficiency of the study
• VIP influence on the power of a study
• precision, reliability and consistency
affected by RANDOM ERROR
ASSESSING PRECISION
• Using S.D.Variance (2)
• Using Coefficient of variation = S.D.
X
• Using Kappa statistic
• Using Cronbach's alpha
Strategies for enhancing precision
1. standardizing measurement methods
• preparing study protocols
• preparing operations manual
• writing specific guidelines or instructions
for making each measurement • serving as basis for describing methods
when results are reported
Strategies for enhancing precision
• preparing operations manual
– write down precisely :
- how to prepare environment
and subject
- how to carry out
and record interview
- how to calibrate instrument
Strategies for enhancing precision
• writing specific guidelines or instructions for making
the measurement
uniform performance over
the duration of study
Strategies for enhancing precision
2. Training and certifying the observers
• improving consistency of measurement techniques (several observers)
• performing pilot study
– to test the power of techniques
specified in operations manual
3. Refining the instruments
• writing or spelling out questionnaires and interviews to increase clarity
4. Automating the instruments
• using automatic mechanical devices
Strategies for enhancing precision
5. Repeating the measurement
• impact of random error of any source can be reduced by
– repeating measurement
– using mean of the two or more readings
Strategies for enhancing precision
ACCURACY
DEFINITION :
The degree to
which the results
of a measurement
correspond to
the true state
or truth
SYNONYM:
• validity
• conformity
IMPORTANT POINTS
• accuracy is a function of
“SYSTEMATIC RROR”
• VIP influence on the internal and
external validity of the study
• the greater the systematic error,
the less accurate the variable
IMPORTANT POINTS
• It is attributed to:
– Methodological aspect of
study design or analysis
– Selection of subject
– Quality of information obtained
– Confounding
– Effect Modification
– Misclassification
ASSESSING ACCURACY
Comparison with reference techniques
Gold standards
Strategies for enhancing accuracy
1. Standardizing measurement methods 2. Training and certifying the observers
3. Refining the instruments
4. Automating the instruments
5. Making informal measures
6. Blinding
7. Calibrating the instrument
MAJOR TYPES OF BIAS
• Observer bias
• Subject bias
• Instrument bias
• Information bias
• Selection bias
MAJOR TYPES OF BIAS
• Observer bias consistent distortion in reporting
measurement by observer
- more intensive measurements in
certain subjects
- ask questions about specific exposures several times of cases but only once of controls
MAJOR TYPES OF BIAS
• Observer bias
Ex. a tendency to underestimate blood
pressure in cases known to be
receiving treatment
Ex. a more persistent search of medical
records for a history of smoking
cigarettes in patients known to
have lung cancer
MAJOR TYPES OF BIAS
• Subject bias consistent distortion of measureme
nt by study subject
- selective recall or reporting of an event
respondent bias
or recall bias
MAJOR TYPES OF BIAS
• Instrument bias
- may result from faulty function of a mechanical instrument
- may result from inappropriate use of technique or tool to objective of
measurement
leading questions on questionnaire
MAJOR TYPES OF BIAS
• Information bias a distortion in the estimate of
effect or variable due to:
* measurement error
* misclassification of subjects on
measurement variable
* invalid measurement
MAJOR TYPES OF BIAS
• Information bias
* incorrect diagnostic criteria
* inadequacies in previously recorded data
* unequal diagnostic surveillance
among exposure study groups
in follow up studies
• Selection bias a distortion in the estimate of effect
resulting from how subjects are selected
for study population
“self-selection bias”
MAJOR TYPES OF BIAS
Selection bias can result from:
- choice of groups to be compared
(in all types of studies)
- choice of sampling frame
- loss to follow up or NON RESPONSE
during data collection
(in follow-up studies)
MAJOR TYPES OF BIAS
Selection bias can result from:
- selective surveillance/diagnostic surveillance varies with exposure status
- more intensive measurements in
certain subjects
MAJOR TYPES OF BIAS
SUMMARY
1. Reliability : Precision, Reproducibility
Random Error
2. Validity : Accuracy , Conformity
Systematic Error
Bias
MAJOR TYPES OF BIAS
• Observer Bias
• Subject Bias Recall Bias
Respondent Bias
• Instrument Bias
• Information Bias
• Selection Bias
Reliability and validity of measurement
Reliability ValidityDefinition
Best way toassess
The degree to which avariable has nearly the same value whenmeasured several times
The degree to which avariable actually represents what it is supposed to represent
Comparison among repeated measures
Comparison with a reference standard
Increase powerto detect effects
Increase validityof conclusions
Value tostudy
Threatened by
Random error(variance)
contributed by :
Systematic error(Bias)
contributed by :
The observerThe subjectThe instrument
The observerThe subjectThe instrument
Reliability Validity
.
....... ..... ....
. ...
Illustration of the difference between Precision and Accuracy
....... ..... ....
. ...
good precision
poor accuracy
poor precision good precision poor precision
good accuracy poor accuracygood accuracy
Illustration of the difference between Precision and Accuracy
Frequency
True valueMeasurement
A C
Unreliable
Invalid
B D
DIFFERENCES BETWEEN VALIDITY AND RELIABILITY
Frequency
True valueMeasurement
A C
Unreliable
Invalid
B D
DIFFERENCES BETWEEN VALIDITY AND RELIABILITY
A- Valid and reliableB- Valid but not reliableC- Not valid but reliable D- Not valid and not reliable