introduction to critical appraisal
TRANSCRIPT
Introduction to Critical Appraisal
Dominic GilroyLibrary & Knowledge Service Manager
Leeds & York Partnerships NHS FT
Objectives• Why we need to evaluate literature• How to form a question that will result in a
targeted literature search• Critical appraisal of the evidence – using
appraisal tools/checklists• Thoughts on statistics
What is Critical Appraisal
“Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value
and relevance in a particular context”
Burls (2009) What is critical appraisal?http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/What_is_critical_appraisal.pdf
More than just reading and summarising a paper.
Critical Appraisal is:
• Balanced assessment of strengths of research against its weaknesses
• Assessment of research process and results
• Consideration of quantitative and qualitative aspects of research
• To be undertaken by all health professionals as part of their work as just one part of the decision making process
Critical Appraisal is not:
• Negative dismissal of a piece of research
• Assessment of results alone
• Based entirely on detailed statistical analysis
• To be undertaken by expert researchers only
Why bother?
• Published evidence is not always reliable – a way of separating “wheat from the chaff”
• Only an estimated 2% is judged clinically relevant
• Patients are better informed and increasingly present having read material on the web or in the media
Challenges:• Can be time-consuming
• It does not always give you an easy answer – there are still judgements and decisions to be made
• It may not give the expected or hoped for result
• It can be dispiriting if it highlights a lack of good evidence – need determination to persist with an area of interest when access to good research in the area is limited
• Maths and statistics …..!
When to do Critical AppraisalAsk the question
Find the evidence
Appraise and interpret the evidence
Act on appropriate evidence
Evaluate and reflect
Step 1: Asking the questionA well structured question is the key to finding evidenceContains 4 parts (usually) … PICO
PICO
P = patient or problemHow would you describe a group of patients similar to yours?
I = intervention or treatmentWhat is the main action you are considering?
C = comparison or alternativeWhat is/are the other option(s) available?
O = outcomeWhat do you/the patient want to happen?
Step 2: Finding the evidence IDENTIFICATION OF THE BEST AND MOST RELEVANT RESOURCE(S) TO SEARCH SHOULD START FROM THE TOP OF THE EVIDENCE HIERARCHY
Systematic Reviews
Expert Opinion
Randomised Controlled Trials
Anatomy of a paperLook for:
Abstract - abbreviated form of the study
Introduction - problem being studied and explains why the research is being carried out
Methods - information concerning the experiment design, including study sample, treatment allocation, outcome measures
Results - clear presentation of the experiments results, including any adverse effects or variables that may have affected the outcome
Discussion/conclusion - gives the author a chance to give their viewpoint and explain any clinical importance of the experiment
Appraisal Tools• Several different appraisal tools are available
• Check sheets of question
• Different tools for different types of research
• Helps make appraisal quick, simple, effective and consistent
• Can help get rid of personal bias
• Help identify if results are significant and valid
Implications of the research:Consider:• Does it produce new knowledge?• How applicable is the research to my own patients?
Perhaps the ultimate question …• Should you act on the findings and change practice?
Some numbers and statistics• Presentation of statistics
• Sample Size
• P Values
• Confidence Intervals
• Risk Reduction
• Number Needed to Treat
Disclaimer:
(Most) Librarians are not statisticiansThis session is not about in depth-statistics but about how to interpret statistics when reading the paper
Presentation of Statistics• The same statistics can be presented in numerous ways
• The way in which statistics are presented can affect how we interpret the results
• Authors may use this when writing papers to suggest a particular view of the results
Shock Horror!Daily Mail Online
Why eating just one sausage a day increases your risk of cancer by 20% http://www.dailymail.co.uk/health/article-550729/Why-eating-just-sausage-day-raises-cancer-risk-20-cent.html
Critically AppraisedWhy eating just one sausage a day raises your cancer
risk by 20 per cent
• Specifics – This refers to Bowel Cancer• Baseline – the usual risk for bowel cancer in a lifetime is: - 1 in 18 for women (5.6%) - 1 in 20 for men (5%)• The risk with a “sausage a day” is not 25% (5% + 20%) or 1 in 4• The risk with a “sausage a day” increases by 20% of the original risk (20% of 5% = 1%)
So the new risk if you eat a sausage is a far less dramatic: - 1 in 15 for women (6.6%)
- 1 in 17 for men (6%)
Which would you prefer to fund?The following are four different rehabilitation programmes for heart attack victims:
Programme A – which reduced the rate of deaths by 20%
Programme B – which produced an absolute reduction in deaths of 3%
Programme C – which increased patients’ survival rate from 84% to 87%
Programme D – which meant that 31 people needed to enter the programme to avoid one death
Which would you prefer to fund?The following are four different rehabilitation programmes for heart attack victims:
Programme A – which reduced the rate of deaths by 20%
Programme B – which produced an absolute reduction in deaths of 3%
Programme C – which increased patients’ survival rate from 84% to 87%
Programme D – which meant that 31 people needed to enter the programme to avoid one death
They all refer to the same study!(Taken from: How to read a paper / Trisha Greenhalgh, 2010, 4th ed.)
Sample Size“A trial should be big enough to have a high chance of detecting… a worthwhile effect if it exists, and thus to be reasonably sure that
no benefit exists if it is not found in the trial”Practical Statistics for Medical Research/ Doug Altman, 1991
• A sample size may be given in the paper
• Don’t worry about how they have calculated it but……
• If they do provide a sample size check if they actually achieve the stated number in their recruiting figures
• If they do not provide the sample size look at the numbers included in the trial as low numbers can mean a less reliable study
P Values
Are useful in deciding if a result could be just by chance:
0 1Impossible that result is by chance Certainty that result is by chance (Good) (Bad)
P= 0.001 Very unlikely 1 in 1000P= 0.05 Fairly unlikely 1 in 20P= 0.5 Fairly likely 1 in 2P = 0.75 Very likely 3 in 4
A P value of <0.05 is generally considered statistically significantIf P <0.05 then the result in question is likely to be more than just chance
Confidence Intervals #1
“A confidence interval….allows you to estimate for both positive trials and negative ones whether the strength of the evidence is strong or weak.”
Greenhalgh, Trisha (2014) How to read a paper
If the same trial was run many, many times you might not expect the exact same results every time – but the results should lie within a specific range. With a 95% CI we are saying that 95% of the time the results lie within a specific range.
Trial result
Range of results of 95%
Exceptions to the rule (outliers)(The other 5%)
Confidence Intervals Commute Example
John’s journey home
Average = 50 minutesCan often take up to 55 minutesCan be as quick as 45 minutes
Very occasionally (5% of time or less) it can take over an hour(eg: traffic accidents etc.) or can be quicker if he travels at off peak times
John’s journey time could be expressed as: 50 min (95% C.I. 45 min – 55 min)
Avishai Teicher via the PikiWiki - Israel free image collection project
Confidence Intervals #2
The range of the confidence interval (CI) is important.
The narrower it is the more accurate and precise the results
The broader it is the wider the possibility and range of likely results
Referring back to our commute example:It is more useful to tell someone it may take between 45 and 55 minutes to travel somewhere than it would be to say it could take between 30 minutes and 2 hours. It would help them to plan better.Precision is usually preferable.
Risk ReductionAbsolute Risk Reduction (ARR)
If an intervention decreases risk of an event occurring from 2/100 to 1/100 1% absolute risk reduction
Relative Risk Reduction (RRR)If an intervention decreases risk of an event occurring from 2/100 to 1/10050% relative risk reduction
Number Needed to Treat (NNT)For every n patients treated with the intervention, one extra patient would be expected to benefit
i.e. how many people do you need to treat to be certain that one person would benefit
Example: If NNT = 10 then on average for every ten patients treated we would expect one of them to benefit
Number Needed to Treat (NNT)
Extreme examples:
For the use of antibiotics Cimetidine, Ranitidine & Omeprazole to eradicate Heliobacter Pylori from the stomach NNT = 1.3
For the use of cholesterol lowering Statins in the primary prevention of vascular events like stroke and heart diseaseNNT = 1000
Useful Calculations
Absolute Risk Reduction (ARR) = Control Rate – Experimental Rate
Relative Risk Reduction (RRR) = Absolute Risk Reduction / Control Rate
Number Needed to Treat (NNT) = 100 / Absolute Risk Reduction
Control Rate = Chance of outcome in control group (%)
Experimental Rate= Chance of outcome in experimental group (%)
Example Exercise
Infected with flu? Yes No Total
New Trial Drug 38 73 111
Placebo (control) 79 35 114
Total 117 108 225
Outcome: Preventing influenza
Control Rate = Chance of outcome in control group (%)
(79/114) x 100 = 69%
Experimental Rate= Chance of outcome in experimental group (%)
(38/111) x 100 = 34%
Example Exercise (continued)Outcome: Preventing influenza
Absolute Risk Reduction (ARR) = Control Rate – Experimental Rate
69 - 34 = 35%
Relative Risk Reduction (RRR) = Absolute Risk Reduction /Control Rate
35 / 69 = 0.5%
Number Needed to Treat (NNT) = 100 / Absolute Risk Reduction
100/35 = 2.85 round up to 3
Control Rate = 69%
Experimental Rate=34%
Practice Paper Part 2Look at the checklist and try to respond to points 7 - 11
• What is the sample size and do they achieve it?• Are there any P values given in the results and what do these suggest?
Looking at the primary outcome measure
• What is the absolute risk reduction ?• What is the relative risk reduction ?• What is the number needed to treat?
Useful WebsitesCritical appraisal
http://www.casp-uk.net/ http://jamaevidence.com/
Statshttp://www.medicine.ox.ac.uk/bandolier/glossary.html
Further ReadingGreenhalgh, Trisha (2014) How to read a paper: the basics of evidence-based medicine. Oxford: Wiley Blackwell. 5th ed.
Crombie, Iain K (1996) Pocket guide to critical appraisal.London: BMJ.
Straus, Sharon (ed.) (2005) Evidence-based medicine: how to practice and teach EBM. Edinburgh: Elsevier Churchill Livingstone. 3rd ed.
Ajetunmobi, Olajide (2002) Making sense of critical appraisal.London: Arnold.
ConclusionWe have examined:
• Why we need to evaluate literature
• Critical appraisal of the evidence – using appraisal tools / checklists
• Statistics introduction
Contact Details
Dominic GilroyLibrary & Knowledge Service ManagerLeeds & York Partnerships NHS [email protected] 85 55658
Leeds Libraries for Healthwww.leedslibraries.nhs.uk