Interpreting numbersScotPHO training course

March 2011

Dr Gerry McCartney

Head of Public Health Observatory Division

NHS Health Scotland

gmccartney@nhs.net

Approaching numbers: some questions to ask

• What is being counted?– Definitions– Type of numbers - counts, means, etc.

• Who/where (population): what is the denominator?• When (time): what time period do they cover?• How (source): where did they come from?• Why were they produced: is there an agenda?

68% of Doctors don’t

listen to their patientsSMOKERS ON PILL DOUBLE STROKE RISK

Incidence and Prevalence

• Dealt with in more detail later in course• Incidence describes the number of new cases in the

population over a period of time• Prevalence describes the number of cases present in a

population at any one point in time

Framework for interpreting numbers

Could your interpretation be affected by either:

• Error• Chance• Confounding (the mixing of two effects) • Bias (systematic departure from truth – either deliberate

or unintentional)

Example: COPD (lung disease) variation

• Is COPD more common in Board A or Board B?• Errors (e.g. different definitions used in each Board)?• Chance (e.g. no confidence intervals used)?• Bias (e.g. are there systematic differences in how disease is

recorded)?• Confounding (e.g. are there mixed effects – such as age

structure)?

NHS Board A Cases per 1,000 population

NHS Board B Cases per 1,000 population

Anytown 230 Smalltown 190

Othertown 280 Seatown 210

Bigtown 159 Hilltown 149

Overall 223 Overall 183

Sources of error:

• Mistakes in data collection, data recording, data storage, data transmission

• Coding errors, transcription errors• Can be random or systematic

• Do the numbers add up?• Are the number plausible?

Chance and interpreting numbers

• Most figures report data for a sample from a larger population

• A different sample would give a different result• Year to year fluctuation can be due to chance • The size of the sample dictates the degree to which a

difference is likely to be due to chance• Confidence intervals and p-values give estimates of the

precision of a value– E.g. Relative risk of heart disease amongst diabetics is 7.4 (95% CI 6.5-

8.6) means that there is a less than 1 in 20 chance of the true value lying outwith the range 6.5 to 8.6

Bias – identification and interpretation

• Bias is a systematic alteration of figures away from the true value

Examples• Selection bias – critically appraise sampling strategy, loss to

follow-up, response rate• Information bias – completeness of data, calibration,

participant self-report, recall time • Publication bias – think about a funnel plot

Confounding: when separate effects are mixed together

• In this example, the effect of location is mixed with (confounded by) the effect of age

• The population of Western Isles is older so has higher rates of CHD admission

• Is CHD more common in Western Isles after taking age into account?

NHS Western Isles NHS Lothian

CHD admissions per 100,000 per year

350 200

Methods for dealing with confounding

Design•Randomisation (only for experimental studies)•Restriction (e.g. narrow the comparison groups by age, sex,

ethnicity, socioeconomic status)•Matching

Analysis• Stratification (i.e. compare sub-groups, but has dangers) • Standardisation*• Multivariate analysis*

*dealt with in more detail elsewhere

Standardisation: brief interpretation

•A method of “removing” the effect of other factors to allow a “fair” comparison

•The other factors are most commonly age and sex, but standardisation can be used for other factors

•Standardisation shows the rates you would get if the population had a “standard” age and sex structure

NHS Western Isles NHS Lothian

Crude CHD admissions per 100,000 per year

350 200

Directly age standardised CHD admissions per 100,000 per year

250 220

Standardised Mortality Ratios (SMRs):

• This is a comparison of mortality in a population with a ‘standard’ population taking account of age structure

• The standard population is allocated a value of 100 for whatever the mortality rate is

• The age and sex standardised mortality of the population of interest is then divided by that in the standard population to give a figure for comparison with the 100

• An SMR of 150 indicates that mortality is 50% higher after accounting for age and sex differences

Interpreting associations: does A cause B?

A B

A B

A

B

A B

C

?

Causal relationship

Confounding

Chance

Some quick notes on interpreting graphs:

• Beware of: ambiguity, distortion and distraction

Data Ambiguity

Data distortion (1)10

.110

.210

.310

.4

0 1 2 3 4 5 6 7

Data Distortion (2)

Data Distortion (3)

Data distortion (4)

0

200

400

600

800

1000

1200

1400

Financial year

Ra

te p

er

10

0'0

00

po

pu

lati

on

Male

Female

General acute inpatient discharges with an alcohol-related diagnosis in any position, by gender, Scotland, 1982/3 - 2009/10

Data distortion (5)

0

200

400

600

800

1000

1200

1400

Financial year

Ra

te p

er

10

0'0

00

po

pu

lati

on

Male

Female

General acute inpatient discharges with an alcohol-related diagnosis in any position, by gender, Scotland, 1982/3 - 2009/10

Summary

• Always ask the questions: what, who, where, when, how and why

• Think about possible problems with data: errors, chance, bias and confounding

• Even when things are associated they may not be cause and effect

• Beware of the possibility of graphs creating distortions, distractions or ambiguity

Questions

gmccartney@nhs.net