實証醫學Critical Appraisal In

Evidence-Based Medicine

柯德鑫 , MD, PhD.奇美醫學中心 神經內科

Evidence-Based Medicine (EBM)

The consistent use of current best evidence derived from published clinical and epidemiologic research in management of patients, with attention to the balance of risks and benefits of diagnostic tests and alternative treatment regimens, taking account of each patient’s unique circumstances, including baseline risk, comorbid conditions and personal preferences.

Evidence-Based Medicine (EBM)

Meta-analysis is an overview in which uses quantitative methods to summarise the results from different clinical trials.

Odds Ratio describes the odds of an experimental patient suffering an adverse event relative to a control patient.

What evidence-based medicine is:

The practice of EBM is the integration of individual clinical expertise

with the best available external clinical evidence

from systematic research.and

patient’s values and expectations

What is Evidence-Based Medicine?

See a patient Ask a question Seek the best evidence for that

question Appraise that evidence Apply the evidence Monitor the change

健康狀態

病人

感覺不適

形成最可能的假設診斷

證實假設診斷

病史詢問理學檢查臨時診斷思考

診斷條件：特異性高的發現統計上相關性相近的病態生理 / 病理機制

相關的檢驗與檢查

病態生理變化

疾病

醫師

Evidence-Based Medicine

Ask a question Acquire some articles Appraise the evidence Apply the findings Assess your performance

證據等級之定義

第一級：最高級證據 (Level Ⅰ) 來源 (1) 隨機雙盲試驗，並有適當病人數所原發性終點分析； (2) 適當執行的優質隨機分組試驗之巨集分析第二級：中級證據 (Level Ⅱ)來源 (1) 隨機非雙盲試驗 (2) 小規模隨機試驗 (3) 大規模隨機試驗並有事先界定的次發性終點分析第三級：次級證據 (Level Ⅲ)來源 (1) 前瞻性病例系列，並有同時或前後對照 (2) 隨機試驗之事後分析第四級：證據未明 (Level Ⅳ)來源 (1) 小規模病例系例無對照或病例報告 (2) 雖然沒有對照試驗之科學證據，但專家普遍同意

Critical appraisal is the process of systematically examining research evidence to assess its validity, results and relevance before using it to inform a decision.

What is critical appraisal?

Critical Appraisal

The process of deciding whether a piece of research can help you in answering your clinical question. Three questions you need to ask about any kind of research:

1. Is it valid? 2.  Is it important? 3. Is it applicable to the patient?

Critical Appraisal Skills

VIP Valid: 研究方法的探討 Important: 結論的分析 Practice: 如何運用來照顧病人 RCT/Cohort study/Case-control

study

· By Expertise -Secondary journal -CAT(critical appraisal topics) -Cochrane database -JAMA & BMJ Guide -Basic Clinical Statistics·Have an appraisal course·EBM style journal meeting

How to Appraise Evidence?

Evaluation of A Diagnostic Test

Sensitivity is the proportion of people with the disease who have a positive test.

Specificity is the proportion of people free of the disease who have a negative test.

Positive Predictive Value (+PV) is the proportion of people with a positive test who truly have the disease.

Negative Predictive Value (-PV) is the proportion of people with a negative test who are free of the disease.

Evaluation of A Diagnostic Test

SnNout when a sign/test has a high sensitivity, a negative result rules out the diagnosis; e.g. the sensitivity of a history of ankle swelling for diagnosing ascites is 92%, therefore if a person does not have a history of ankle swelling, it is highly unlikely that the person has ascites.

SpPin when a sign/test has a high specificity, a Positive result rules in the diagnosis; e.g. the specificity of fluid wave for diagnosing ascites is 92%. Therefore, if a person has a fluid wave, it is highly likely that the person has ascites.

Critical Appraisal for Diagnosis

Are the results of this diagnostic study valid?

Are the valid results of this diagnostic study important?

Can you apply this valid, important evidence about a diagnostic test in caring for your patient?

Are the results of this diagnostic study valid?

Was there an independent, blind comparison with a reference ("gold") standard of diagnosis?

Was the diagnostic test evaluated in an appropriate spectrum of patients (like those in whom we would use it in practice)?

Was the reference standard applied regardless of the test result?

Was the test (or cluster of tests) validated in a second, independent group of patients?

Are the valid results of this diagnostic study important?

Are the valid results of this diagnostic study important?

Sensitivity = a / (a+c) = 371/809 = 90 %

Specificity = d / (b+d) = 1500/1770 = 85 %LR+ = sens / (1-spec) = 90/15 = 6LR- = (1-sens) / (spec) = 10/85 = 0.12Positive Predictive Value = a / (a+b) = 731/1001 = 73 % Negative Predictive value = d / (c+d) = 1500/1578 = 95 % Prevalence = (a+c) / (a+b+c+d) = 809/2579 = 32 %Pre-test odds = prevalence / (1-prevalence) = 31/69 = 0.45Post-test odds = pre-test odds * LRPost-test Probability = post-test odds / (post-test odds + 1)

Can you apply this valid, important evidence about a diagnostic test in caring

for your patient?

Is the diagnostic test available, affordable, accurate and precise in your setting?

Can you generate a clinically sensible

estimate of your patient's pre-test probability (from practical data, from personal experience, from the report itself, or from clinical speculation)?

Can you apply this valid, important evidence about a diagnostic test in caring

for your patient?

Will the resulting post-test probabilities affect your management and help your patient? (Could it move you across a test-treatment threshold? Would your patient be a willing partner in carrying it out?)

Would the consequences of the test help your patient?

Evidence-Based Medicine (EBM) in Clinical Trial

Number Needed to Treat (NNT) is the number of patients who need to be treated to prevent one bad outcome.

Number Needed to Harm (NNH) is the number of patients under treatment that is need to develop one bad outcome or adverse effect.

Evidence-Based Medicine (EBM) in Clinical Trial

Intention-to-treat analysis: including all patients in the analysis who are randomized to treatment, regardless of what occurs during the clinical trial.

Adherence-to-protocal (On treatment) analysis: analyzing patients based on the treatment they received finally.

Therapies

To treat or not to treat? Validity Importance Applicability

Therapies

Validity• Was it randomised?• Was the allocation concealed?• Were the all the subjects analysed

correctly?• Was it blinded?• Were the groups similar?

Therapies

Importance• What were the results?• Over what time period?• With what precision?

Therapies

Number needed to treat Relative risk reduction Absolute risk reduction Event rates

Therapies

Event rates• n with event / total

Control event rate (CER) Experimental event rate (EER)

Therapies

Absolute risk reduction• difference in two event rates• CER - EER = ARR

Relative risk reduction• proportion of control rate• CER-EER / CER = RRR

Therapies

Number needed to treat• number of extra patients you need to

treat to prevent one bad outcome• 1 / ARR = NNT

Therapies

95% confidence interval• range within which the true value falls

with 95% confidence

• use computer (e.g. CATMaker)

Occurrence of death, stroke, or other major complications

Adverse eventsNumber

needed to treat NNT

Patient status at entry

Placebo P

Active A

RRR ARR 1/ARR=NNT

Prior target organ damage

.22 .08 64% .14 1/.14=7

No prior organ damage

.10 .04 60% .06 1/.06=17

THERAPY WORKSHEETCitation:

Are the results of this single preventive or therapeutic trial valid?

Was the assignment of patients to treatments randomized?-and was the randomization list concealed?

Were all patients who entered the trial accounted for at its conclusion? -and were they analyzed in the groups to which they were randomized?

Were patients and clinicians kept “blind” to which treat-ment was being received?

Aside from the experimental treatment, were the groups treated equally?

Were the groups similar at the start of the trial?

THERAPY WORKSHEET Are the valid results of this randomized trial

important?

Occurrence of diabeticneuropathy

Relative RiskReduction

RRR

Absolute RiskReduction

ARR

Number Neededto Treat

NNT

Usual InsulinControl Event

RateCER

IntensiveInsulin

ExperimentalEvent Rate

EER

CER - EERCER

CER - EER 1/ARR

9.6% 2.8% 9.6% - 2.8% = 71%

9.6% 9.6% - 2.8% = 6.8%

(4.3% to 9.3%) 1/6.8% = 15 pts,

(11 to 23)

SAMPLE CALCULATIONS:

THERAPY WORKSHEET SAMPLE CALCULATIONS:

YOUR CALCULATIONS:

Relative RiskReduction

RRR

Absolute RiskReduction

ARR

Number Neededto Treat

NNT

CER EER CER - EER

CER CER - EER 1/ARR

95% Confidence Interval (CI) on an NNT = 1 / (limits on the CI of its ARR) =

%4.2/711

972.0028.0

730

904.0096.096.1/

.exp#

)1(

.#

)1(96.1/

ptserof

EEREER

ptscontrolof

CERCER

Therapies

Application• Can it be applied to my patient?• Can it be done here?• How do patient values affect the

decision?

Therapies

Is it valid? Is it important?

• NNT for whatover how longwith what precision

Does it apply?

Six guides to distinguish useful from useless or even harmful therapy

1. Was the assignment of patients to treatments really randomized ？

2. Was all clinically relevant outcomes reported ？3. Were the study patients recognizably similar to your own ？4. Were both clinical and statistical significance considered ？5. Is the therapeutic maneuver feasible in your practice ？6. Were all patients who entered the study accounted for its

conclusion ？

The likelihood of help vs. harm (LHH)

In applying a SR or RCT to an individual patient, we need to consider:

•our patient's risk, relative to patients in the trial, of the event we hope to prevent with the treatment: ft

•our patient's risk, relative to patients in the trial, of the side-effect we might cause from the treatment: fh

•our patient's perception of the severity of the event we're trying to prevent relative to the side-effect we might cause: s

The likelihood of help vs. harm is(1 /NNT) x ft x s vs. (1/NNH)x fh

For example, suppose we're applying a trial with an NNT of 9 and an NNH of 12 and we think our patient is at just half the risk of the event but at twice the risk of the side-effect, then the "raw" LHH before we adjust it for our patient's perception of relative severity is 1/9 x 0.5 vs. 1/12 x 2 = 1/18 vs. 1/6, or three times as likely to harm vs. help the patient. However, if our patient regards the severity of the event that the treatment might prevent to be six times worse than the side-effect it might cause, then the final LHH = 1/18 x 6 vs. 1/6, or two times as likely to help vs. harm

Should these valid, potentially important results change the treatment of our patient?

1. Is our patient so different from those included in the study that its results don’t apply?

2. What is our patient’s risks of the adverse event? What is our patient’s potential benefit from the therapy?

3. What are our patient’s preferences, concerns and expectations from this treatment?

4. What alternative treatments are available?

Are the results of this systematic review of therapy valid?

1. Is this a systematic review of randomized trials?

2. Does it include a methods section that describes:

(a) finding and including all the relevant trials?

(b) assessing their individual validity?

3. Were the results consistent from study to study?

(4. Were individual patient data used in the analysis (or aggregate data)? )

Is the systematic review important?

Are the valid results of this systematic review important?

Translating odds ratios to NNTs

Are the valid, important results of this systematic review applicable to our

patient?

1.Is our patient so different from those in the study that its results cannot apply?

2.Is the treatment feasible in our setting?

3.What are our patient's potential benefits and harms from the therapy?

4.What are our patient's values and preferences for both the outcome we are trying to prevent and the side-effects we may cause?

Are the recommendations in this guideline valid?

1. Did its developers carry out a comprehensive, reproducible literature review within the past 12 months?

2. Is each of its recommendations both tagged by the level of evidence upon which it is based and linked to a specific citation?

Are the results of this prognosis study valid?

1. Was a defined, representative sample of patients assembled at a common (usually early) point in the course of their disease?

2. Was patient follow-up sufficiently long and complete?

3. Were objective outcome criteria applied in a “blind” fashion?

4. If subgroups with different prognoses are identified, was there adjustment for important prognostic factors?

5. Was there validation in an independent group (“test set”) of patients?

Are the valid results of this prognosis study important?

1. How likely are the outcomes over time?

2. How precise are the prognostic estimates?

Can we apply this valid, important evidence about prognosis in caring

for our patient?

1. Were the study patients similar to our own?

2. Will this evidence make a clinically important impact on our conclusions about what to offer or tell our patient?

Are the results of this harm/etiology study valid?

1. Were there clearly defined groups of patients, similar in all important ways other than exposure to the treatment or other cause?

2. Were treatments/exposures and clinical outcomes measured in the same ways in both groups (was the assessment of outcomes either objective or blinded to exposure)?

3. Was the follow-up of study patients complete and long enough?

Are the results of this harm/etiology study valid?

4. Do the results satisfy some “diagnostic tests for causation”?

• Is it clear that the exposure preceded the onset of the outcome?

• Is there a dose-response gradient?• Is there positive evidence from a “dechallenge-

rechallenge” study?• Is the association consistent from study to

study?• Does the association make biological sense?

Are the valid results from this harm/etiology study important?

Adverse outcome Totals

Present (case)

Absent (control)

Exposed to the

treatment

Yes (cohort) a b a+bNo (cohort) c d c+d

Totals a+c b+d a+b+c+d

In a randomized trial or cohort study:Relative risk (RR) = [a/(a+b)]/[c/(c+d)]In a case-control study: relative odds (RO) = ad/bcTo calculate the NNH for any OR and PEER:

Are the results of this clinical decision analysis valid?

1. Were all the important therapeutic alternatives (including no treatment) and outcomes included?

2. Are the probabilities of the outcomes valid and credible?

3. Are the utilities of the outcomes valid and credible?

4. Was the robustness of the conclusion tested?

Are the valid results from this decision analysis important?

1. Did one course of action lead to clinically important gains?

2. Was the same course of action preferred despite clinically sensible changes in probabilities and utilities?

Are the valid, important results of this decision analysis applicable to

our patient?

1. Do the probabilities apply to our patient?

2. Can our patient state his/her utilities in a stable, usable form?