CRITICAL APPRAISAL OF THE MEDICAL LITERATURE

Partini Pudjiastuti TChild Health Department

Faculty of Medicine University of Indonesia

Critical appraisal (Making Reading More Worthwhile)

What is Critical Appraisal?1. Critical appraisal = quality assessment2. ….process of weighing up evidence to see

how useful it is in decision making3. .…a process of assessing the validity,

reliability and usefulness of evidence4. Critical appraisal is about considering,

evaluating and interpreting information in a systematic and objective way

Critically Appraise What You Read.

• Separating the wheat from the chaff.• Time is limited – you should aim to quickly

stop reading the dross.• Others contain useful information mixed

with rubbish.• Simple checklists enable the useful

information to be identified.

Critical Appraisal – Critical Thinking

• Appraising (evaluating/reviewing) the available evidence to construct clinical reasoning strategies and to make decisions

• Finding strengths and limitations of written ‘evidence’

• You need to decide what evidence to pay attention to (what is “worthy” of your attention) versus what to ignore

Why critically appraise?

• Supports sound decision making based on best available evidence

• Helps us determine (three R’s):• How rigorous a piece of research is• What the results are telling us• How relevant it is to our patient

What is “Evidence”?

• People disagree on what constitutes “evidence”

• Evidence - what is generally regarded as a scientific fact

• Evidence - a combination of information obtained from 3 sources: research, clinical experience, and client preferences (Kitson, Harvey, & McCormack, 1998)

Why do we need evidence?

– Resources should be allocated to things that are EFFECTIVE

– The only way of judging effectiveness is EVIDENCE

– “In God we trust – all others bring data”

Why do we need evidence?

• Move towards: EVIDENCE-BASED MEDICINE

• Move away from: EMINENCE- BASED MEDICINE

What we really, really want is

EVIDENCE-INFORMED MEDICINE

Sources of Evidence

• Primary sources– Based on experiments and published

research• Secondary sources

– Systematic reviews– Clinical guidelines– Journals of secondary publication e.g.

Evidence Based Medicine

“5S” Pyramid of Evidence Resources

Levels of evidence 1. Systematic reviews of RCTs and high quality

RCTs2. Systematic reviews of Cohort studies, lower

quality RCTs, Outcomes research3. Systematic reviews of case controls, case

control studies4. Case Series5. Expert opinion

See http://www.cebm.net/levels_of_evidence.asp for full descriptions

Types of EvidenceQuestion Types

Type of Question Best EvidenceHealth care interventions: treatment, prevention

Quantitative: Systematic Review of RCTs or RCT

Harm or Etiology Quantitative: Observational Study - Cohort or Case Control

Prognosis Quantitative: Observational Study - Cohort, Case Control

Diagnosis or Assessment Quantitative: Comparison to Gold Standard

Economics Quantitative: Cost-effectiveness Study

Meaning Qualitative: case study, ethnography, grounded theory, phenomenologic approach

KEY QUALITY PARAMETERS

• VALIDITY

• RELIABILITY

• IMPORTANCE

VALIDITY• INTERNALIs the study designed in such a way that I can trust the findings?

• EXTERNALIs the study designed in such a way that Ican generalize the findings?

RELIABILITY

If the study was conducted again,would the results be the same?

Usually interpreted as the accuracyof measurement.

IMPORTANCE

What was the effect sizeor magnitude of effect?

Clinical vs. statistical significance.

Tools for Critical Appraisal

• What are the results?

• Are the results valid?

• Will the results help me in patient care?

EBM “simplified” approach:

I

V

A

Evidence based medicine5 steps

Formulate question

Efficiently track down bestavailableevidence

Critically review thevalidity and usefulnessof the evidence

Implement changes in clinical practice

Evaluate performance

CHECK LIST FOR MEDICAL LITERATURE (COMPLETENESS)

1. Title2. Authors3. Abstract: structured? Informative? Abbreviation?4. Introduction: length? Relevant references? Target

population?5. Methods:

– Design– Inclusion criteria – Exclusion criteria– Sample size, sampling method– Randomization technique– Intervention: masking?– Outcome measurement: blinding?

• Primary outcome: type of variable • Secondary outcome: type of variable

– Analysis

6. Results– Baseline characteristics– Main outcome– Secondary outcome

7. Discussion– General– Strength and weakness– Conclusions

8. References– Vancouver style– Constant

9. Acknowledgment10.Ethics approval11.Conflict interest

CHECK LIST FOR MEDICAL LITERATURE (COMPLETENESS)

What to assess?(in study of cause-effect relationship)

A. General description• Type of design• Target population, source population,

sample• Sampling method• Dependent and independent variables• Main results?

B. Internal validity, non-causal relationship– Influence of bias– Influence of chance– Influence of confounders

What to assess?(in study of cause-effect relationship)

BiasWhat is a bias? A process that tends to produce

results that depart systematically from the true values existing in the study population

Types of bias1.Sample (subject selection) biases, which may result

in the subjects in the sample being unrepresentative of the population which you are interested in

2.Measurement (detection) biases, which include issues related to how the outcome of interest was measured

3. Intervention (performance) biases, which involve how the treatment itself was carried out.

C. Internal validity, causal relationship• Temporality (cause precedes effect)• Strength of association (large difference, RR, OR, etc) or

small p value or narrow confidence interval• Biological gradient (dose dependence)• Consistency among studies (diff. populations or designs)• Specificity (certain factor results in certain effect)• Coherence (does not conflict with current knowledge)• Biological plausibility: can be explained with current

knowledge (at least in part)

What to assess?(in study of cause-effect relationship)

D. External validity– Applicable to study subjects– Applicable to source population– Applicable to target population

What to assess?(in study of cause-effect relationship)

11 items, each with 3 sections

1. Can you find this information in the paper?

2. Is the way this was done a problem?3. Does this problem threaten the validity of

the study?

11 items1. What is the research question?2. What is the study type?3. What are the outcome factors and how are they measured?4. What are the study factors and how are they measured?5. What important confounders are considered?6. What are the sampling frame and sampling method?7. In an experimental study, how were the subjects assigned to

groups? In a longitudinal study, how many reached final follow-up? In a case control study, are the controls appropriate? (Etc)

8. Are statistical tests considered?9. Are the results clinically/socially significant?10. Is the study ethical? 11. What conclusions did the authors reach about the study question?

1.What is the research question?

• (Is the way this was done a problem?)– Is it concerned with the impact of an

intervention, causality or determining the magnitude of a health problem?

• (Does this problem threaten the validity of the study?)– Is it a well stated research

question/hypothesis?

2. What is the study type?• (Is the way this was done a problem?)

– Is the study type appropriate to the research question?

• (Does this problem threaten the validity of the study?)– If not, how useful are the results produced by

this type of study?

3. What are the outcome factors and how are they measured?

• (Is the way this was done a problem?)– a) are all relevant outcomes assessed– b) is there measurement error?

• (Does this problem threaten the validity of the study?)– a) how important are omitted outcomes– b) is measurement error an important source

of bias?

4. What are the study factors and how are the measured?

• (Is the way this was done a problem?)– Is there measurement error?

• (Does this problem threaten the validity of the study?)– Is measurement error an important source of

bias?

5. What important potential confounders are considered?

• (Is the way this was done a problem?)– Are potential confounders examined and

controlled for?

• (Does this problem threaten the validity of the study?)– Is confounding an important source of bias?

6. What are the sampling frame and sampling method?

• (Is the way this was done a problem?)– Is there selection bias?

• (Does this problem threaten the validity of the study?)– Does this threaten the external validity of the

study?

7. Questions of internal validity• (Is the way this was done a problem?)

– In an experimental study, how were the subjects assigned to groups?

– In a longitudinal study, how many reached follow-up?– In a case control study, are the controls appropriate?

• Note: other issues of relevance to internal validity are considered under the other headings in this critical appraisal system. You can add your own questions, and also design your own questions for other study types such as cross sectional studies and systematic reviews

• (Does this problem threaten the validity of the study?)– Does this threaten the internal validity of the study?

8. Are statistical tests considered?

• (Is the way this was done a problem?)– Were the tests appropriate for the data?– Are confidence intervals given?– Is the power given if a null result?– In a trial, are results presented as absolute

risk reduction as well as relative risk reduction?

• (Does this problem threaten the validity of the study?)– If not, how useful are the results?

9. Are the results clinically/socially significant?

• (Is the way this was done a problem?)– Was the sample size adequate to detect a

clinically/socially significant result?– Are the results presented in a way to help in

health policy decisions?

• (Does this problem threaten the validity of the study?)– Is the study useful?

10. Are ethical issues considered?

• (Is the way this was done a problem?)– Does the paper indicate ethics approval?– Can you identify potential ethical issues?

• (Does this problem threaten the validity of the study?)– Are the results or their application

compromised?

11. What conclusions did the authors reach about the study

question?• (Is the way this was done a problem?)

– Do the results apply to the population in which you are interested?

• (Does this problem threaten the validity of the study?)– Will you use the results of the study?

Appraisal Tools• Tools from the Critical Appraisal Skills

Programme (CASP)– Systematic Reviews– Randomised Controlled Trials– Qualitative Research Studies– Cohort Studies– Case-Control Studies– Diagnostic Test Studies– Economic Evaluation Studies

Available at: http://www.phru.nhs.uk/casp/critical_appraisal_tools.htm

Study Designs Recap

Effectiveness of Therapy

Risk Factors / Prognosis

Diagnosis

Attitudes & Beliefs

Randomised Controlled Trial

Cohort Study

Survey using Gold Standard

Qualitative (Interviews, Observations, etc)

CRITICAL APPRAISAL

- VALID- IMPORTANT- APPLICABLE

METHODS RESULTS DISCUSSION

THANKS

From Data to Wisdom

• Data are what researchers collect• Information results when data is analyzed

and interpreted (EVIDENCE)• Knowledge results when information is

shared, acquired, and used• Wisdom is the ability to make the right

use of knowledge

Types of evaluations

Efficacy– treatment does more good than harm when

offered to those who adhere to treatment recommendations

– Does it work under ideal conditions?

Types of evaluations

Effectiveness– treatment does more good than harm in those

to whom it is offered, under ordinary (clinical) circumstances

– Can it work in the real world?

Critical review protocol - quantitative research

• Study purpose• Application to occupational therapy• Study design• Bias• Sampling issues/ sample size/ drop outs• Outcome measurement (reliability, validity)• Intervention description & implementation• Results - statistical & clinical significance• Implications