intermediate methods in observational epidemiology 2008 quality assurance and quality control

Intermediate methods in observational epidemiology

2008

Quality Assurance and Quality Control

Threats to Causal Inference in Epidemiologic Studies

Confounding• Experimental Design

• Adjustment/Control

Threat Solution

Bias • Quality Assurance

• Quality Control

QA: Activities to assure quality of data that take place prior to data collection (through protocol and manuals of operation)

QC: Efforts during the study to monitor the quality of data at identified points during the collection and processing of data

Definitions of Quality Assurance and Quality Control

STEPS IN QUALITY ASSURANCE

(1) Specify hypothesi(e)s

(2) Specify general design -- develop protocol

(3) Select or prepare data collection instruments,and develop procedures for data collection/ processing -- develop operation manuals

(4) Train staff -- certify staff

(5) Using certified staff, pre-test and pilot studyinstruments and procedures. In the pilot study, assessalternative strategies for data collection- eg,telephone vs. in-person interviews

(6) Modify (2) and (3) and retrain staff onthe basis of results of (5)

(1) Specify hypothesi(e)s

(2) Specify general design -- develop protocol

(3) Select or prepare data collection instruments,and develop procedures for data collection/ processing -- develop operation manuals

(4) Train staff -- certify staff

(5) Using certified staff, pre-test and pilot studyinstruments and procedures. In the pilot study, assessalternative strategies for data collection- eg,telephone vs. in-person interviews

(6) Modify (2) and (3) and retrain staff onthe basis of results of (5)

Based on a “grab” sample

STEPS IN QUALITY ASSURANCE

STEPS IN QUALITY ASSURANCE

(1) Specify hypothesi(e)s

(2) Specify general design -- develop protocol

(3) Select or prepare data collection instruments,and develop procedures for data collection/ processing -- develop operation manuals

(4) Train staff -- certify staff

(5) Using certified staff, pre-test and pilot studyinstruments and procedures. In the pilot study, assessalternative strategies for data collection- eg,telephone vs. in-person interviews

(6) Modify (2) and (3) and retrain staff onthe basis of results of (5)

Based on a sample as similar as possible to the study population

STEPS IN QUALITY ASSURANCE

(1) Specify hypothesi(e)s

(2) Specify general design -- develop protocol

(3) Select or prepare data collection instruments,and develop procedures for data collection/ processing -- develop operation manuals

(4) Train staff -- certify staff

(5) Using certified staff, pre-test and pilot studyinstruments and procedures. In the pilot study, assessalternative strategies for data collection- eg,telephone vs. in-person interviews

(6) Modify (2) and (3) and retrain staff onthe basis of results of (5)

QUALITY CONTROL PROCEDURES: TYPES

1. Observation monitoring

“Over the shoulder” observation of staff by experienced supervisor(s) to identify problems in the implementation of the protocol.

Example:

- Taping of interviews

QUALITY CONTROL PROCEDURES: TYPES

1. 1. Observation monitoringObservation monitoring

2. Quantitative monitoring

-Random repeat (phantom) measurements based on either internal or external pools (biologic samples) to examine:

. Intra-observer

. Inter-observer

Advantages. Better overall quality of data. Measurement of reliability

variability

Phantom sample based on an internal pool

Internal phantom sample

STUDY BASE BLOOD

SAMPLES OF 7 PARTICIPANTS

Aliquot 2: measurement in

study lab

Aliquot 1: measurement in

gold standard lab

Aliquot 1: measurement in

gold standard lab

Aliquot 2: measurement in

study lab

Phantom sample based on an external pool

Phantomsample from the gold standard lab

STUDY BASE BLOOD

SAMPLES OF 7 PARTICIPANTS

QUALITY CONTROL PROCEDURES: TYPES

1. Observation monitoringObservation monitoring

2. Quantitative monitoring

- Random repeat measurementsRandom repeat measurements

- Monitoring of individual technicians for deviations from expected values

Example: monitoring of digit preferencefor blood pressure (expected: 10%for each digit)

Digit Preference in Systolic Blood Pressure (SBP) Measurements

Last digit of SBP (mmHg)

Observer A

Observer B

0 11% 15% 1 10% 5% 2 9% 13% 3 9% 7% 4 10% 17% 5 10% 3% 6 12% 12% 7 8% 8% 8 10% 18% 9 11% 1%

Digit Preference in Systolic Blood Pressure (SBP) Measurements

Last digit of SBP (mmHg)

Observer A

Observer B

0 11% 15% 1 10% 5% 2 9% 13% 3 9% 7% 4 10% 17% 5 10% 3% 6 12% 12% 7 8% 8% 8 10% 18% 9 11% 1%

Quality Control Indices

• Validity (Accuracy)

• Precision (Repeatability, Reliability)

Validity: Usually estimated by calculating sensitivity and specificity. The study (observed) measurement (“test”) is compared with a more accurate method (“gold standard”).

When clearcut gold standard notavailable: “inter-method reliability”

Problem: Limited to 2 x 2 tables

...Thus, traditional reliability indices (e.g., kappa, correlation

coefficient) can be also used to estimate validity of continuous

variables or variables with more than 2

categories

Gold

Sta

nd

ard

resu

lts

Study results

•

• • •••

•

•

••

•

• •

•

•

Reliability: Sources of Variability

• Measurement Error

– Instrument/Technique/Lab

– Observer/Technician• Intra-observer• Inter-observer

• Intra-individual (physiologic)

Blood collected from an individual(1st measurement)

To examine within-technician variability?Aliquot 1.2: Lab

determination done by same technician

Aliquot 1.2: measurement done

by same technician in a masked

fashion

To measure within-individual variability? Blood collected from the individual

(replicate measurement)Repeat blood collection in same

individual X time later

To examine between-lab variability?Send Aliquot 1.3 to a different lab

Aliquot 1.3: Lab determination

done at a different lab

Time Design of a study to evaluate sources of variability

(Based on Chambless et al, Am J Epidemiol 1992;136:1069-1081)

For other sources of

variability, use phantom samples

Phantom sample

Aliquot 1.2

Aliquot 1.3

Aliquot 1.1: Study lab determination

Aliquot 1.4

To examine between-technician variability? Aliquot 1.3: Lab determination done by a

different technician at study lab

Aliquot 1.2: measurement done by a different technician in a masked fashion at study lab

Indices of Reliability (also used for validity)

• % differences between repeat measurements (expected if no bias: ½ positive and ½ negative)

• % observed agreement

• Kappa

• Correlation coefficient

• Coefficient of variation

• Bland-Altman plot

Indices of Reliability (also used for validity)

• % differences between repeat measurements (expected if no bias: ½ positive and ½ negative)

• % observed agreement

• Kappa

• Correlation coefficient

• Coefficient of variation

• Bland-Altman plot

Agreement Between First and Second Readings to Identify Atherosclerotic Plaque in the Left Carotid Bifurcation by B-

Mode Ultrasound in the ARIC Study (Li et al, Ultrasound Med Biol 1996;22:791-9)

986777209Total

79472569Normal

19252140Plaque

TotalNormalPlaqueSecond Reading

First Reading

Percent Observed Agreeement: [140 + 725] ÷ 986 = 88%

Shortcomings• Chance agreement is not taken into account• If most observations are in one of the concordance cell(s), % Observed Agreement overestimates agreement

Agreement Between First and Second Readings to Identify Atherosclerotic Plaque in the Left Carotid Bifurcation by B-

Mode Ultrasound in the ARIC Study (Li et al, Ultrasound Med Biol 1996;22:791-9)

986777209Total

79472569Normal

19252140Plaque

TotalNormalPlaqueSecond Reading

First Reading

Percent Observed Agreeement: [140 + 725] ÷ 986 = 88%

Shortcomings• Chance agreement is not taken into account• If most observations are in one of the concordance cell(s), % Observed Agreement overestimates agreement

Indices of Reliability (also used for validity)

• % differences between repeat measurements (expected if no bias: ½ positive and ½ negative)

• % observed agreement

• Kappa

• Correlation coefficient

• Coefficient of variation

• Bland-Altman plot

986777209Total

794725 69 Normal

19252 140 Plaque

TotalNormalPlaqueSecond Reading

First Reading

The most popular measure of agreement: Kappa Statistics

E

EO

P

PP

0.1

PO Observed agreement proportionPE Expected (chance) agreement proportion

986777209Total

794725 69 Normal

19252 140 Plaque

TotalNormalPlaqueSecond Reading

First Reading

PO = [140 + 725] ÷ 986 = 0.88

Kappa Statistics

986777209Total

794725 69 Normal

19252 140 Plaque

TotalNormalPlaqueSecond Reading

First Reading

PO = [140 + 725] ÷ 986 = 0.88

Expected agreement: (1) multiply the marginals converging on the concordance cells, (2) add the products, and (3) divide by the square of the total:

Kappa Statistics

986777209Total

794725 69 Normal

19252 140 Plaque

TotalNormalPlaqueSecond Reading

First Reading

PO = [140 + 725] ÷ 986 = 0.88

Expected agreement: (1) multiply the marginals converging on the concordance cells, (2) add the products, and (3) divide by the square of the total:

Kappa Statistics

986777209Total

794725 69 Normal

19252 140 Plaque

TotalNormalPlaqueSecond Reading

First Reading

PO = [140 + 725] ÷ 986 = 0.88

Expected agreement: (1) multiply the marginals converging on the concordance cells, (2) add the products, and (3) divide by the square of the total:

Kappa Statistics

Shortcomings• Kappa is a function of the prevalence of the condition• Can be calculated only for categorical variables (2 or more)

Maximum agreement not due to chance

Agreement not due to chance

P P

PO E

E1 0

0 8 8 0 6 8

1 0 0 6 80 6 3

.

. .

. ..

PE = [(209 x 192) + (777 x 794)] ÷ 9862= 0.68

Thus, kappa values obtained from

different populations may

not be comparable

Interpretation of Kappa values

(Altman & Bland, Statistician 1983;32:307-17)

1.0

0.8

0.6

0.4

0.2

0

-1.0

VERY GOOD

GOOD

MODERATE

FAIR

POOR

Indices of Reliability (also used for validity)

• % differences between repeat measurements (expected if no bias: ½ positive and ½ negative)

• % observed agreement and % observed positive agreement

• Kappa

• Coefficient of variation

• Bland-Altman plot

Coefficient of variation (CV)

General definition: Standard Deviation(SD) as a percentage of the mean

value

2

1

2)(j

iiji XXV

Calculation of the Coefficient of Variability

Xi1 and Xi2 = values of repeat measurementson same lab sample

Xi = mean of these measurements

For each pair of values: iVsd

The mean overall CV over all pairs is the average of all pair-wise CVs

and

For each pair of repeatmeasurements: CV

sd

X 1 0 0

Example of Calculation of the Coefficient of Variation - I

Phantoms

1

2

Replicates (e.g., 2 different observers, 2 measurements done by same observer, 2 different labs, etc.)PAIR No.

1

2

3

4

k

......

Pair (Split samples) No. 1: Measurement of total cholesterol

Measurement No. 1 (X11)= 154 mg/dL

Measurement No. 2 (X12)= 148 mg/dL

24.41811 vsd

V1= (154 - 151)2

+ (148 - 151)2

= 18 mg/dL

Phantoms

1

2

ReplicatesPAIR No.

1

Do the calculations for each pair of replicate samples

Mean= [154 + 148] / 2= 151 mg/dL

Example of Calculation of the Coefficient of Variation - I

%8.2100151

24.41001

1 X

sdCV

Repeat the

calculation for all

pairs of

measurements

and calculate

average to obtain

overall CV

Analyte Intra-Class Correlation Coefficient*

Coefficient of variation (%)**

Total serum cholesterol 0.94 5.1

HDL 0.94 6.8

HDL2 0.77 24.8

Reliability in the ARIC study (Am J Epi 1992;136:1069)

*Best: as high as possible

**Best: as low as possible

Indices of Reliability (also used for validity)

• % differences between repeat measurements (expected if no bias: ½ positive and ½ negative)

• % observed agreement and % observed positive agreement

• Kappa

• Coefficient of variation

• Bland-Altman plot