Critical Appraisal: An Introduction

Melanie Browne HBHSc, MLIS Information Specialist

Agenda

• Introduction• Define Evidence-based clinical practice• How to read a research article• Search for best evidence• Apply critical appraisal in your library• Our new role• Case presentation• Resources

Evidence-Based Medicine

"...the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.“from Sackett, DL, et al. "Evidence based medicine: What it is and what it isn't." (BMJ 1996; 312: 71-2)

• "Evidence based medicine (EBM) is an approach to health care that promotes the collection, interpretation, and integration of valid, important and applicable patient-reported, clinician-observed and research-derived evidence. The best available evidence, moderated by patient circumstances and preferences, is applied to improve the quality of clinical judgements" (McKibbon et al 1995).

Why EBM?

• A clinician needs to read 17 peer reviewed articles per day, every day of the year, to stay current (Haynes 1993). Haynes, R. (1993) Where's the meat in clinical journals? ACP Journal Club, 119: A23-4.

There are 20 million pieces of 'evidence' of varying quality and sometime of contradictory conclusions.

How Can You Access, Distil and Apply Research/Evidence?

• Develop summaries • Train clinicians and other health care

decision makers to find and appraise relevant evidence

• Decision support systems

Summaries With Rigorous Methodologies

• Cochrane Collaboration • National Centre for Clinical

Excellence • Centre for Clinical Effectiveness at

Monash University

The Argument for Evidence-Based Medicine

• Stay up to date with the current literature

• Communicate effectively• Make the best use of information

from the history, physical examination, and diagnostic testing

• Avoid common pitfalls of clinical decision-making

Critical Appraisal

• The assessment of evidence by systematically reviewing its relevance, validity and results to specific situations. - Chambers, R. (1998).

What is ‘best’ evidence?

• Using critical appraisal skills you can understand the methods and results of the research and then be able to assess the quality of the research.

Critical appraisal Exact science

Critical appraisal “right” answer

Information Mastery

Slawson and Shaughnessy Formula

Usefulness of Medical Information =

Relevance x Validity____________________

Work to Access

• Smith R. What clinical information do doctors need? Br Med J 1996; 313: 1062-8.

• When doctors see patients they usually generate at least one question

• Most of the questions concern treatment

• Many of the questions are highly complex, simultaneously asking about individual patients and particular areas of medical knowledge

• Often doctors are asking not simply for information but for support, guidance, affirmation, and feedback

• Doctors are most likely to seek answers to these questions from other doctors

• The best information sources provide relevant, valid material that can be accessed quickly and with minimal effort

Basic elements of clinical decision making

Haynes RB. Loose connections between peer-reviewed clinical journals and clinical practice. Ann Intern Med 1990;113:724-8.

Why Should we Critically Appraise?

• Published research is not always reliable

• Published research is not always relevant

• To improve clinical effectiveness, we need a systematic framework to interpret research

Difficulties with Critical Appraisal • Can be time consuming initially• Doesn’t provide an “easy” answer• It could show a lack of good evidence

in a particular topic

Key Steps to Effective Critical Appraisal

1. Are the results valid?2. What are the results?3. How will these results be

relevant to the patient?

Validity and Reliability

• A test is valid when it measures what it’s supposed to.

• If a test is reliable, it yields consistent results.

• A test can be both reliable and valid, one or the other, or neither.

• Reliability is a prerequisite for measurement validity. 

Reliable, but Not Valid!

Not Reliable, Not Valid!

Reliable and Valid

Mark Newman - Middlesex University updated 04/2001

What a Physician can be Faced With on a Daily Basis?• ~1 - 2 questions per patient (clinics)• 15/patient/day (wards)

• ~30% of questions are followed – People – Books– Electronic resources

• 40% easy to answer• 30% tough to answer• 30% cannot answer

Source: Dawes M, Sampson U. Knowledge management in clinical practice: a systematic review of information seeking behaviour in physicians. Int J Med Inform. 2003 Aug; 71(1):9-15. Review. PMID: 12909153

Steps to EBM

1. Formulate a clear, focused clinical question 2. PICO model3. Search the literature for the best external evidence4. Critically appraise the evidence for its

validity and usefulness5. Implement the useful evidence in clinical practice6. Evaluate the results

Question Formulation

Not easy but EXTREMELY important

Good questions will

– Focus/clarify your information need – Give you some idea of where to look for

information– Give you searching concepts and terms

Background Questions

…Often broad in nature…Often not patient-specific but fact

based…May not need to integrate

knowledge…More common early in training/new

situations

 

Foreground Questions

…Often for a specific patient or clinical situation

…Narrow in focus…Need to integrate external

information with clinical/situational data

PICO Structure

Patients P

Intervention IComparison COutcomeO

Clinical Scenario

What therapeutic agents can be used for rate control of atrial fibrillation (AF) in a patient with congestive heart failure (CHF)?

Starting Point

Department: EmergencyPopulation: Patients with atrial

fibrillation and congestive heart failure

Intervention: Rate controlComparison: N/AOutcome: Mortality, effectiveness of

rate control

Patient w/ Atrial Fibrillation in congestive heart failure

Comparison

PICO Schematic Model

Rate control

Effectivenessof ratecontrol

Rhythm control /No treatment

Clinical problem

Define the search question

Choose a resource/database

Create a search strategy

Summarize the evidence

Apply the evidence

Try another relevant resource

Create a search strategy

Summarize the evidence

Try another relevant resource

General Search Strategy

Adapted from: Sackett, D. et al. 2000. Evidence-Based Medicine: How to Practice and Teach EBM. 2nd Edition. Toronto: Churchill Livingstone.

Pooryield

Study Types for Question Types

Diagnosis

Prospective cohort study with good quality validation against “Gold Standard”

Therapy Randomized controlled clinical trial (RCT)

Etiology/Harm

RCT, cohort or case-control study (probably retrospective)

Prognosis

Prospective cohort study

Does this treatment work? systematic review, RCT

How good is a diagnostic test? (prospective) cohort study

Should we screen? RCT

What causes this disease?RCT, prospective cohort study, case control study (rare diseases)

What did people think or do?cohort study, cross-sectional survey, qualitative study

How can we do Critical Appraisal?

• Use common sense• Use simple checklists• Use different checklists depending on

the different types of studies (i.e., RCTS, systematic reviews etc)

• Checklists help you focus on the important parts of the article

Research Methodology

• Who were the participants of the study?• How were they recruited?• Was there bias in the recruiting

methods?• How was the data collected?• What statistical tests were used?• Where the data collection methods

accurate?

Critical appraisal questions

• What is the paper about? • Why was the study done? • What type of study was done? • Was it primary research

(experiment, RCT, cohort, case-control, cross-sectional, longitudinal, case report/series)?

Critical appraisal questions

• Was it secondary research (overview, systematic review, meta-analysis, decision analysis, guidelines development, economic analysis)?

• Was the design appropriate (for study on treatment, diagnosis, screening, prognosis, or causation)?

Critical appraisal questions

• Was the study ethical? • Is the design right?

(BMJ Editor's checklists)

How to Read a Research Article? – First Glance.

• Purpose of reading the paper.• Do not read the abstract.• Read the title, find out who the authors are and where they work,

look for sources of funding and conflicts of interest• Look at the tables and figures• Is there a diagram to show the flow of participants through each

stage of the study • Why was the study done and what hypothesis were the authors

testing?• What is broadly the topic of research?• Therapy, Diagnosis, Screening, Prognosis, Causation?• What type of study was done?• Is this a primary (experimental, clinical trial, survey) or a

secondary paper (review, meta-analysis, guideline, economic analysis)?

• Was the study design appropriate?• Now proceed to a critical appraisal of the paper

Case Presentation

• Read the Ray et al(2008) paper and divide the group into 2 – for a debate on the paper “Breast size and risk of type 2 diabetes mellitus.” CMAJ

• Ray and colleagues studied data from 92 106 women in conjunction with the Nurses' Health Study II and found that breast size at age 20, assessed by recall of bra cup size, correlated positively with the incidence of type 2 diabetes

• After adjustment for relevant factors, such as body mass index, waist circumference and family history of diabetes, the hazard ratio dropped to 1.58 but remained significant.

• Nurses' Health Study II population

data emanates from women who were mainly of white ancestry, and that their analysis is based on recall and self-report.

New Role of Health Sciences Librarians

• Teaching access to the literature and other information resources

• Teaching use of technology as a means to access and manage information

• Teaching skills in information organization and critical appraisal

Role of Librarians as Information Clinicians

• Information Clinician• Medical Informatics Tutor • EBM Educator

Informationist

• A discipline requiring a combination of the skills of a librarian, a clinical epidemiologist and a medical scientist

Davidoff, F. & Florance,V. (2000) The Informationist: A New Health Profession? Annals of Internal Medicine, 132: 996-998.

Effective INFORMAtician

• Knows categories of information resources…therapy, diagnosis, prognosis…synthesized resources vs original studies

• Knows strengths/weaknesses of information resources

• Knows when to use each category• Knows that some resources are better

than others in certain situations…Cochrane diagnostic accuracy data?

• Knows what to do with the results

• Practical solutions - Librarian as a partner on the teaching team; teachable moments

• Formal hospital library educational programs have focused on orientation and instruction in the use of bibliographic tools. i.e.- LATCH (Literature Attached to Charts) and clinical medical librarianship

Critical Appraisal Skills Training

• Research methodology • Statistical techniques

Courses for Librarians Out There

CILIP: the Chartered Institute of Library and Information Professionals

• Critical appraisal skills for healthcare librarians: building on the basicshttp://www.cilip.org.uk/training/training/2008/libinfo/criticalappraisalskillsforhealthcarelibrariansbuildingonthebasics.htm

General Resources:• CASP (Critical Appraisal Skills Programme) - part of

the Public Health Resource Unit based at Oxford, CASP runs training workshops on critical appraisal skills.  

• Evidence-Based Medicine Toolkit - hosted by the University of Alberta, this is an online "box" of handy tools to help you find, appraise, and apply in practice, evidence-based research

• How to read a paper - a set of ten guides from the BMJ (individual links given in the relevant section below).

• Levels of Evidence - a ranking system used to rank various study designs in order of evidence-based merit: systematic reviews/meta-analyses and well conducted randomised controlled trials (RCTs) are usually seen as the best form of "evidence", with research based on the outcome of a case series placed somewhere near the bottom

• Netting the Evidence - search for the keyword "appraisal" to find a quality assessed list of appraisal resources

• User's guides to evidence based practice - based on a series of articles published in JAMA, these guides give comprehensive advice on how to find, appraise and apply research in practice

Keeping Current

• Daily InfoPOEMs http://www.infopoems.com/

• bmjupdates+ http://bmjupdates.mcmaster.ca/index.asp

• Tables of contents of journals (e.g. JAMA)– From journal itself– My NCBI from PUBMED

Absolute

Risk D

iffere

nce

Absolute Risk Reduction

Absolute Benefit Increase

Confidence Intervals

Intention to treat

Likelihood Ratio

Numbers Needed to Treat (NNT)

Odds Ratio

P Value

Predictive Value of tests

Relative R

isk

Relative Risk Reduction

Sensitivity

Specificity