critical appraisal skills quantitative reviews

North Cumbria Informatics Service

Pippa OrrKnowledge Support Librarian

Critical Appraisal Skillsquantitative reviews

With acknowledgements to CASP for their slides


Critical appraisal is the process of weighing upevidence to see how useful it is in decision making

Critical Appraisal Skills Programme (CASP)

http://www.phru.nhs.uk/casp/critical_appraisal_tools.htm

http://www.health.adelaide.edu.au/img/photo/dentist.jpg


Effectiveness of Health Care

• doing the right thing• to the right patient• in the right way• at the right time• at the right cost• in the right place



Kinds of evidence

• Descriptive– cross-sectional, longitudinal

• Analytic– case-control study– cohort study

• Experimental– randomized controlled trial



Hierarchy of evidence



Why does good evidence from research fail to get into practice??

- 75% cannot understand the statistics

- 70% cannot critically appraise a research paper

Using Research for Practice: A UK Experience of the barriers scale Dunn V, Crichton C, Williams K, Roe B, Seers K



Critical appraisal helps the reader Critical appraisal helps the reader of research ………...of research ………...

• Decide how trustworthy a piece of research is (validity)

• Determine what it is telling us (results)• Weigh up how useful the research will be (relevance)



Robust randomisation procedures:

•to ensure that the variables are equal in both groups

•to remove all bias

•to ensure that the results are generalisable

Primary Research Evidence:

Randomised Controlled Trials (RCT)



Randomised controlled trial

population

group 1

group 2

Outcome

Outcome

new treatment

control treatment



Blinding = participants don’t know what intervention they are getting

Double blinding = those giving the intervention don’t know what the participant is receiving

Blinding



It is important to ensure that all those that are randomised into the trial are followed up to the trial’s conclusion

Loss to follow-up



Analysing people, at the end of the trial, in the groups to which they were randomised, even if they did not receive the intended intervention.

Intention to treat analysis



Types of review:

Meta-analysis

Systematic reviews

Reviews



Publication bias

Papers with "interesting" results are (or may be) more likely to be:

• submitted for publication

• accepted for publication

• published in a major journal and in English Language

• quoted by authors

• quoted in newspapers



Odds Ratio, Relative RiskMeasures of risk

The likelihood of something happening

V

The likelihood of something not happening



Odds ratio (OR)

• The odds of an event happening in the experimental group expressed as a proportion of the odds of an event happening in the control group

• The closer the OR is to 1, the smaller the difference in effect, i.e. no effect: OR = 1



Confidence intervals/ limits

• Presents the range of likely effects• The 95% confidence interval, for example,

includes 95% of results from studies of the same size and design in the same population

• This is close, but not identical, to saying that the true size of effect (never exactly known) has 95% chance of falling within the confidence interval

• The narrower/ shorter the confidence interval, the more precise/ confident we can be about the estimate



Forest plots

• Common approach to presenting the results of a meta-analysis

• Also known as a ‘blobbogram’ or ‘odds ratio diagram’

• Graphical representation of individual trial results included in a review, together with the combined meta-analysis result



meta-analysis result

line of no effect

confidence interval



p-value

• The probability (ranging from 0 to 1) that the results observed in a study (or results more extreme) could have occurred by chance if in reality the null hypothesis was true, ie if you did nothing.

• If this probability is less than 1/20 (which is when the p value is less than 0.05), then the result is conventionally regarded as being “statistically significant”.



The p-value in a nutshell

p < 0.05a statistically significant result

p = 0.05

or 1 in 20result fairly unlikely to

be due to chance

0 1

Could the result have occurred by chance?

The result is unlikely to be due to chance

The result is likely to be due to chance

1 20

p > 0.05not a statistically significant result

p = 0.5

or 1 in 2result quite likely to be

due to chance

1 2



Number needed to treat

Is the number of people you would need to treat with a specific intervention to see one additional occurrence of a specific beneficial outcome.



Critical appraisal:questions to apply to reviews

• is it trustworthy? validity• what does it say? results• will it help? relevance


critical appraisal skills quantitative reviews

Documents

research paperusing

reader of research

piece of research

right cost

right thing

right patient

right way

right time