intermediate methods in observational epidemiology 2008 quality assurance and quality control
Post on 19-Dec-2015
222 views
TRANSCRIPT
Intermediate methods in observational epidemiology
2008
Quality Assurance and Quality Control
Threats to Causal Inference in Epidemiologic Studies
Confounding• Experimental Design
• Adjustment/Control
Threat Solution
Bias • Quality Assurance
• Quality Control
QA: Activities to assure quality of data that take place prior to data collection (through protocol and manuals of operation)
QC: Efforts during the study to monitor the quality of data at identified points during the collection and processing of data
Definitions of Quality Assurance and Quality Control
STEPS IN QUALITY ASSURANCE
(1) Specify hypothesi(e)s
(2) Specify general design -- develop protocol
(3) Select or prepare data collection instruments,and develop procedures for data collection/ processing -- develop operation manuals
(4) Train staff -- certify staff
(5) Using certified staff, pre-test and pilot studyinstruments and procedures. In the pilot study, assessalternative strategies for data collection- eg,telephone vs. in-person interviews
(6) Modify (2) and (3) and retrain staff onthe basis of results of (5)
(1) Specify hypothesi(e)s
(2) Specify general design -- develop protocol
(3) Select or prepare data collection instruments,and develop procedures for data collection/ processing -- develop operation manuals
(4) Train staff -- certify staff
(5) Using certified staff, pre-test and pilot studyinstruments and procedures. In the pilot study, assessalternative strategies for data collection- eg,telephone vs. in-person interviews
(6) Modify (2) and (3) and retrain staff onthe basis of results of (5)
Based on a “grab” sample
STEPS IN QUALITY ASSURANCE
STEPS IN QUALITY ASSURANCE
(1) Specify hypothesi(e)s
(2) Specify general design -- develop protocol
(3) Select or prepare data collection instruments,and develop procedures for data collection/ processing -- develop operation manuals
(4) Train staff -- certify staff
(5) Using certified staff, pre-test and pilot studyinstruments and procedures. In the pilot study, assessalternative strategies for data collection- eg,telephone vs. in-person interviews
(6) Modify (2) and (3) and retrain staff onthe basis of results of (5)
Based on a sample as similar as possible to the study population
STEPS IN QUALITY ASSURANCE
(1) Specify hypothesi(e)s
(2) Specify general design -- develop protocol
(3) Select or prepare data collection instruments,and develop procedures for data collection/ processing -- develop operation manuals
(4) Train staff -- certify staff
(5) Using certified staff, pre-test and pilot studyinstruments and procedures. In the pilot study, assessalternative strategies for data collection- eg,telephone vs. in-person interviews
(6) Modify (2) and (3) and retrain staff onthe basis of results of (5)
QUALITY CONTROL PROCEDURES: TYPES
1. Observation monitoring
“Over the shoulder” observation of staff by experienced supervisor(s) to identify problems in the implementation of the protocol.
Example:
- Taping of interviews
QUALITY CONTROL PROCEDURES: TYPES
1. 1. Observation monitoringObservation monitoring
2. Quantitative monitoring
-Random repeat (phantom) measurements based on either internal or external pools (biologic samples) to examine:
. Intra-observer
. Inter-observer
Advantages. Better overall quality of data. Measurement of reliability
variability
Phantom sample based on an internal pool
Internal phantom sample
STUDY BASE BLOOD
SAMPLES OF 7 PARTICIPANTS
Aliquot 2: measurement in
study lab
Aliquot 1: measurement in
gold standard lab
Aliquot 1: measurement in
gold standard lab
Aliquot 2: measurement in
study lab
Phantom sample based on an external pool
Phantomsample from the gold standard lab
STUDY BASE BLOOD
SAMPLES OF 7 PARTICIPANTS
QUALITY CONTROL PROCEDURES: TYPES
1. Observation monitoringObservation monitoring
2. Quantitative monitoring
- Random repeat measurementsRandom repeat measurements
- Monitoring of individual technicians for deviations from expected values
Example: monitoring of digit preferencefor blood pressure (expected: 10%for each digit)
Digit Preference in Systolic Blood Pressure (SBP) Measurements
Last digit of SBP (mmHg)
Observer A
Observer B
0 11% 15% 1 10% 5% 2 9% 13% 3 9% 7% 4 10% 17% 5 10% 3% 6 12% 12% 7 8% 8% 8 10% 18% 9 11% 1%
Digit Preference in Systolic Blood Pressure (SBP) Measurements
Last digit of SBP (mmHg)
Observer A
Observer B
0 11% 15% 1 10% 5% 2 9% 13% 3 9% 7% 4 10% 17% 5 10% 3% 6 12% 12% 7 8% 8% 8 10% 18% 9 11% 1%
Quality Control Indices
• Validity (Accuracy)
• Precision (Repeatability, Reliability)
Validity: Usually estimated by calculating sensitivity and specificity. The study (observed) measurement (“test”) is compared with a more accurate method (“gold standard”).
When clearcut gold standard notavailable: “inter-method reliability”
Problem: Limited to 2 x 2 tables
...Thus, traditional reliability indices (e.g., kappa, correlation
coefficient) can be also used to estimate validity of continuous
variables or variables with more than 2
categories
Gold
Sta
nd
ard
resu
lts
Study results
•
• • •••
•
•
••
•
• •
•
•
Reliability: Sources of Variability
• Measurement Error
– Instrument/Technique/Lab
– Observer/Technician• Intra-observer• Inter-observer
• Intra-individual (physiologic)
Blood collected from an individual(1st measurement)
To examine within-technician variability?Aliquot 1.2: Lab
determination done by same technician
Aliquot 1.2: measurement done
by same technician in a masked
fashion
To measure within-individual variability? Blood collected from the individual
(replicate measurement)Repeat blood collection in same
individual X time later
To examine between-lab variability?Send Aliquot 1.3 to a different lab
Aliquot 1.3: Lab determination
done at a different lab
Time Design of a study to evaluate sources of variability
(Based on Chambless et al, Am J Epidemiol 1992;136:1069-1081)
For other sources of
variability, use phantom samples
Phantom sample
Aliquot 1.2
Aliquot 1.3
Aliquot 1.1: Study lab determination
Aliquot 1.4
To examine between-technician variability? Aliquot 1.3: Lab determination done by a
different technician at study lab
Aliquot 1.2: measurement done by a different technician in a masked fashion at study lab
Indices of Reliability (also used for validity)
• % differences between repeat measurements (expected if no bias: ½ positive and ½ negative)
• % observed agreement
• Kappa
• Correlation coefficient
• Coefficient of variation
• Bland-Altman plot
Indices of Reliability (also used for validity)
• % differences between repeat measurements (expected if no bias: ½ positive and ½ negative)
• % observed agreement
• Kappa
• Correlation coefficient
• Coefficient of variation
• Bland-Altman plot
Agreement Between First and Second Readings to Identify Atherosclerotic Plaque in the Left Carotid Bifurcation by B-
Mode Ultrasound in the ARIC Study (Li et al, Ultrasound Med Biol 1996;22:791-9)
986777209Total
79472569Normal
19252140Plaque
TotalNormalPlaqueSecond Reading
First Reading
Percent Observed Agreeement: [140 + 725] ÷ 986 = 88%
Shortcomings• Chance agreement is not taken into account• If most observations are in one of the concordance cell(s), % Observed Agreement overestimates agreement
Agreement Between First and Second Readings to Identify Atherosclerotic Plaque in the Left Carotid Bifurcation by B-
Mode Ultrasound in the ARIC Study (Li et al, Ultrasound Med Biol 1996;22:791-9)
986777209Total
79472569Normal
19252140Plaque
TotalNormalPlaqueSecond Reading
First Reading
Percent Observed Agreeement: [140 + 725] ÷ 986 = 88%
Shortcomings• Chance agreement is not taken into account• If most observations are in one of the concordance cell(s), % Observed Agreement overestimates agreement
Indices of Reliability (also used for validity)
• % differences between repeat measurements (expected if no bias: ½ positive and ½ negative)
• % observed agreement
• Kappa
• Correlation coefficient
• Coefficient of variation
• Bland-Altman plot
986777209Total
794725 69 Normal
19252 140 Plaque
TotalNormalPlaqueSecond Reading
First Reading
The most popular measure of agreement: Kappa Statistics
E
EO
P
PP
0.1
PO Observed agreement proportionPE Expected (chance) agreement proportion
986777209Total
794725 69 Normal
19252 140 Plaque
TotalNormalPlaqueSecond Reading
First Reading
PO = [140 + 725] ÷ 986 = 0.88
Kappa Statistics
986777209Total
794725 69 Normal
19252 140 Plaque
TotalNormalPlaqueSecond Reading
First Reading
PO = [140 + 725] ÷ 986 = 0.88
Expected agreement: (1) multiply the marginals converging on the concordance cells, (2) add the products, and (3) divide by the square of the total:
Kappa Statistics
986777209Total
794725 69 Normal
19252 140 Plaque
TotalNormalPlaqueSecond Reading
First Reading
PO = [140 + 725] ÷ 986 = 0.88
Expected agreement: (1) multiply the marginals converging on the concordance cells, (2) add the products, and (3) divide by the square of the total:
Kappa Statistics
986777209Total
794725 69 Normal
19252 140 Plaque
TotalNormalPlaqueSecond Reading
First Reading
PO = [140 + 725] ÷ 986 = 0.88
Expected agreement: (1) multiply the marginals converging on the concordance cells, (2) add the products, and (3) divide by the square of the total:
Kappa Statistics
Shortcomings• Kappa is a function of the prevalence of the condition• Can be calculated only for categorical variables (2 or more)
Maximum agreement not due to chance
Agreement not due to chance
P P
PO E
E1 0
0 8 8 0 6 8
1 0 0 6 80 6 3
.
. .
. ..
PE = [(209 x 192) + (777 x 794)] ÷ 9862= 0.68
Thus, kappa values obtained from
different populations may
not be comparable
Interpretation of Kappa values
(Altman & Bland, Statistician 1983;32:307-17)
1.0
0.8
0.6
0.4
0.2
0
-1.0
VERY GOOD
GOOD
MODERATE
FAIR
POOR
Indices of Reliability (also used for validity)
• % differences between repeat measurements (expected if no bias: ½ positive and ½ negative)
• % observed agreement and % observed positive agreement
• Kappa
• Coefficient of variation
• Bland-Altman plot
Coefficient of variation (CV)
General definition: Standard Deviation(SD) as a percentage of the mean
value
2
1
2)(j
iiji XXV
Calculation of the Coefficient of Variability
Xi1 and Xi2 = values of repeat measurementson same lab sample
Xi = mean of these measurements
For each pair of values: iVsd
The mean overall CV over all pairs is the average of all pair-wise CVs
and
For each pair of repeatmeasurements: CV
sd
X 1 0 0
Example of Calculation of the Coefficient of Variation - I
Phantoms
1
2
Replicates (e.g., 2 different observers, 2 measurements done by same observer, 2 different labs, etc.)PAIR No.
1
2
3
4
k
......
Pair (Split samples) No. 1: Measurement of total cholesterol
Measurement No. 1 (X11)= 154 mg/dL
Measurement No. 2 (X12)= 148 mg/dL
24.41811 vsd
V1= (154 - 151)2
+ (148 - 151)2
= 18 mg/dL
Phantoms
1
2
ReplicatesPAIR No.
1
Do the calculations for each pair of replicate samples
Mean= [154 + 148] / 2= 151 mg/dL
Example of Calculation of the Coefficient of Variation - I
%8.2100151
24.41001
1 X
sdCV
Repeat the
calculation for all
pairs of
measurements
and calculate
average to obtain
overall CV
Analyte Intra-Class Correlation Coefficient*
Coefficient of variation (%)**
Total serum cholesterol 0.94 5.1
HDL 0.94 6.8
HDL2 0.77 24.8
Reliability in the ARIC study (Am J Epi 1992;136:1069)
*Best: as high as possible
**Best: as low as possible
Indices of Reliability (also used for validity)
• % differences between repeat measurements (expected if no bias: ½ positive and ½ negative)
• % observed agreement and % observed positive agreement
• Kappa
• Coefficient of variation
• Bland-Altman plot