critical appraisal of systematic review & meta-analysis ... · 1 critical appraisal of...

1

Critical appraisal of Systematic review &

Meta-analysis

: Practical points for oncologists

Kullathorn ThephamongkholJiraporn Setakornnukul

Radiation Oncologist, Siriraj hospital

2

ทําไมตอ้งสนใจsystematic review

meta-analysis

ฝันวา่ไดย้นิแพทยป์ระจําบา้นคยุกนั

• ปี 2: สอบตอ้งอา่น/อา้งผลการศกึษาแตล่ะ paper ไหม• ปี 3: ไมต่อ้งเลย ทอ่งเฉพาะ NCCN ไดเ้ลย• ปี 2: ทําไมละ่• ปี 3: NCCN อา้งไดเ้ลย เชือ่พี่• ปี 2: ตอ้งรูเ้หตผุลไหมวา่ จากหลกัฐานอะไร เพราะอะไร• ปี 3: สว่นใหญจ่าก RCTทกุอนัแหละ อยา่สนใจ อยา่เสยีเวลาจํา บางเรือ่ง RCT ไมไ่ดผ้ล แตเ่ขามเีหตผุลของเขาแหละ ถา้ NCCN บอก จะไมรั่กษาเหรอ

•ผมตืน่ขึน้มา อนจิจา….ภาวนาขอใหไ้มเ่ป็นเรือ่งจรงิ!!!3

Your motivation to look for meta-analysis ?

1. NEVER 2. Examination only3. Personal interest (e.g. for fun) 4. Problem in clinical practice5. Find reasons in the guideline

4

5

Clinical problem in practice

What is the best one you believe ?

1. Randomized controlled trial 2. Literature-based meta-analysis3. Individual-patient data (IPD) meta-analysis4. NCCN Guideline5. ASCO Guideline

6

VOTE: Do you give chemotherapy for Postop R0 NSCLC stage IB ? First round

1. YES, in Majority of patients2. NO, in minority of patients3. Y/N, in only special group T >=4 cm 4. Y/N, in at least one of high risk features5. Y/N, in only significant or many high risk

features 7

National Comprehensive Cancer Network

8

9

Cancer Care OntarioMcMaster University, the birthplace of evidence-based health-care

10

11

12

13

14

15

16

ASCO-CCO

17

Table summary: Adjuvant chemo after surgery (R0, -RT)

NCCN guideline 1.2018

CCO 2016

IPD pooled analysis LACE

JCO 2008

IPD metaLancet 2010

Cochrane 2015

ASCO-CCO guideline 2017

IA observe -

Clear benefit , not specify in which patients

Not recommended

IB Recommended in high risk group (poorly diff, vascular invasion, visceral pleura,T >= 4 cm, Nx

T >= 4 cm (would be reasonable to consider)

Official conclusion : Sig improve survival Last paragraph: Benefit didn’tvary except perhaps IAUndertaking validation

Benefit vs risk

IIA/B Recommended Recommended Recommended IIIA Recommended Recommended Recommended

18

19

VOTE: Do you give chemotherapy for Postop R0 NSCLC stage IB ? First round

1. YES, in Majority of patients2. NO, in minority of patients3. Y/N, in only special group T >=4 cm 4. Y/N, in at least one of high risk features5. Y/N, in only significant or many high risk

features 20

21

ในชวีติจรงิบางเรือ่งมีขดัแยง้กนัชดักวา่นีอ้กี

22

NCCN 1.2017 breast screening

•Women should be encouraged to be aware of their breast science this may facilitate detection of interval cancer between routine screening

23

Cochrane 2008

• Data from two large trials do NOT suggest a beneficial effect of screening by breast self-examination but do suggest increased harm in terms of increased numbers of benign lesions identified and an increased number of biopsies performed.

• At present, screening by breast self-examination or physical examination cannot be recommended.

24

•The USPSTF recommends againstclinicians teaching women howto perform breast self-examination. (Grade D recommendation)

25

• The Task Force’s charge is to provide evidence-based,population-level guidance.

• Yet, the media and politicians presented the breast cancer screening recommendations as a major departure from existing guidelines that ……

26

Important message Recommendation # Evidence

•PURE EVIDENCE

•Interpretation and consensus

•Recommendation

27

ทําไมตอ้ง critically appraise systematic reviewเชือ่ไปเลยไดไ้หม

28

Important message

•Suboptimal systematic review: harmful •Recommendation # PURE EVIDENCE

•Critical appraisal is very important

•How in practical/ easy/ clinician’s way?29

Objectives

•Understand critical process in sys review

•Practice/discussion of Critical appraisal

30

Discover (outline)

•Steps from question to synthesis

•Practical points in each step

•Solve examples of clinical problems

Big idea / Key concepts

•See difference in recommendation and find evidence/reasons behind this

Theory: CHECKLIST

33

Key steps

34

Critical appraisal tools

•McMaster & Duke checklist•ROBIS checklist•AMSTAR checklist•PRISMA reporting standard

35

1.Formulate question

2.Conduct searching

3.Studies collection

4.Studies appraisal

5.Synthesis & finding

36

FOCUS, PRE-SPECIFIED

COMPREHENSIVE, SYSTEMATIC, REPLICABLE

CLEAR CRITERIA, JUSTIFIED , TRANSPARENCY

RELIABLE TOOLS OF ASSESSMENT

MAGNITUDE (ABSOLUTE), CI, EXPLAIN DIFFERENCE, RATING

Theory: Statistical reasoning

37

38

Test of heterogeneity I2, p value (cut-off at 0.1)

39

Subgroup benefit of group 2 only ?(treatment effect difference between group 1 and 2)

40

Subgroup benefit of group 2 only ?

•LACE meta-analysis• IA HR 1.4,95%CI = 0.95-2.06• IB HR 0.93,95%CI =0.78-1.10• II HR 0.83,95%CI =0.73-0.95• III HR 0.83,95%CI =0.72-0.94

41

•Probability of interaction

•Trend test= more reliable than p value within subgroup

42

Subgroup benefit of stage II, III?No subgroup benefit of stage IB?

43

44

No subgroup benefit of stage III?(treatment effect difference between stage I,II and III?)

45

Only subgroup benefit of stage II?

46

Practical :

Find evidence/reason behind conclusion= synthesis

57

Table summary: Adjuvant chemo after surgery (R0, -RT)

NCCN guideline 1.2018

CCO 2016

IPD pooled analysis LACE

JCO 2008

IPD metaLancet 2010

Cochrane 2015

ASCO-CCO guideline 2017

IA observe -

Clear benefit , not specify in which patients

Not recommended

IB Recommended in high risk group (poorly diff, vascular invasion, visceral pleura,T >= 4 cm, Nx

T >= 4 cm (would be reasonable to consider)

Official conclusion : Sig improve survival Last paragraph: Benefit didn’tvary except perhaps IAUndertaking validation

Benefit vs risk

IIA/B Recommended Recommended Recommended IIIA Recommended Recommended Recommended

58

NCCN 1.2018NCCN guideline

1.2018CCO 2016

IPD pooled analysis LACE

JCO 2008

IPD metaLancet 2010

Cochrane 2015

ASCO-CCO guideline 2017

IA observe -

Clear benefit , not specify in which patients

Not recommended

IB Recommended in high risk group (poorly diff, vascular invasion, visceral pleura,T >= 4 cm, Nx

T >= 4 cm (would be reasonable to consider)

Official conclusion : Sig improve survival

Last paragraph: Benefit did not vary except perhaps stage IA

Benefit vs risk

IIA/B Recommended Recommended Recommended IIIA Recommended Recommended Recommended

59

NCCN: not seen reasons or reference behind this conclusion!

60

61

62

CCO 2016NCCN guideline

1.2018CCO 2016

IPD pooled analysis LACE

JCO 2008

IPD metaLancet 2010

Cochrane 2015

ASCO-CCO guideline 2017

IA observe -

Clear benefit , not specify in which patients

Not recommended

IB Recommended in high risk group (poorly diff, vascular invasion, visceral pleura,T >= 4 cm, Nx

T >= 4 cm (would be reasonable to consider)

Official conclusion : Sig improve survival

Last paragraph: Benefit did not vary except perhaps stage IA

Benefit vs risk

IIA/B Recommended Recommended Recommended IIIA Recommended Recommended Recommended

63

Synthesis CCO 2016Synthesis

CCO 2016 No meta-analyses by themselvesLACE pooled analysis, a difference in the effect oftreatment by stage was found (stage IA : HR, 1.40; 95% CI, 0.95-2.06; stage IB : HR, 0.93; 95% CI, 0.78-1.10; stage II : HR, 0.83; 95% CI, 0.73-0.95; stage III : HR, 0.83; 95% CI, 0.72-0.94; test for trend, P=0.043)

T>=4 cm 1) CALGB 9633 trial exploratory analysis HR 0.69, 90% CI 0.48-0.99, p=0.043 in T>=4 cm2) JBR.10 trial exploratory analysis HR non sig but positive interaction between tumor

size & chemotherapy, p=0.02T>=4 cm (HR, 0.66; 95% CI, 0.39-1.14)T< 4 cm (HR, 1.73; 95% CI, 0.98-3.04)

64

LACE pooled analysisNCCN guideline

1.2018CCO 2016

IPD pooled analysis LACE

JCO 2008

IPD metaLancet 2010

Cochrane 2015

ASCO-CCO guideline 2017

IA observe -

Clear benefit , not specify in which patients

Not recommended

IB Recommended in high risk group (poorly diff, vascular invasion, visceral pleura,T >= 4 cm, Nx

T >= 4 cm (would be reasonable to consider)

Official conclusion : Sig improve survival Last paragraph: Benefit didn’tvary except perhaps IAUndertaking validation

Benefit vs risk

IIA/B Recommended Recommended Recommended IIIA Recommended Recommended Recommended

65

Synthesis IPD LACE

66

• Different effect by stage was found • stage IA : HR, 1.40; 95% CI, 0.95-2.06; • stage IB : HR, 0.93; 95% CI, 0.78-1.10; • stage II : HR, 0.83; 95% CI, 0.73-0.95; • stage III : HR, 0.83; 95% CI, 0.72-0.94; • test for trend, P=0.04• test for interaction, P=0.06

Synthesis : IPD pooled analysis LACE

67

IPD meta Lancet 2010, Cochrane 2015NCCN guideline

1.2018CCO 2016

IPD pooled analysis LACE

JCO 2008

IPD metaLancet 2010

Cochrane 2015

ASCO-CCO guideline 2017

IA observe -

Clear benefit , not specify in which patients

Not recommended

IB Recommended in high risk group (poorly diff, vascular invasion, visceral pleura,T >= 4 cm, Nx

T >= 4 cm (would be reasonable to consider)

Official conclusion : Sig improve survival

Last paragraph: Benefit did not vary except perhaps stage IA

Benefit vs risk

IIA/B Recommended Recommended Recommended IIIA Recommended Recommended Recommended

68

Synthesis IPD meta 2015

69

70

Subgroup stage for all chemo

71

72

Subgroup stage in platinum only

• Absolute benefit 5yr OS

• IA 70% to 73%*• IB 55% to 60%** • II 40% to 45% • IIIA 30% to 35% • * contradictory with CI

73

Larger IB to stage II in 7th edition

Synthesis: IPD meta 2015

74

• *** All chemotherapy HR 0.86 (0.81-0.92);p<0.0001• *** Test for subgroup difference p=0.06, I2 = 51%

ASCO-CCO 2017NCCN guideline

1.2018CCO 2016

IPD pooled analysis LACE

JCO 2008

IPD metaLancet 2010

Cochrane 2015

ASCO-CCO guideline 2017

IA observe -

Clear benefit , not specify in which patients

Not recommended

IB Recommended in high risk group (poorly diff, vascular invasion, visceral pleura,T >= 4 cm, Nx

T >= 4 cm (would be reasonable to consider)

Official conclusion : Sig improve survival

Last paragraph: Benefit did not vary except perhaps stage IA

Benefit vs risk

IIA/B Recommended Recommended Recommended IIIA Recommended Recommended Recommended

75

Synthesis: ASCO-CCO guideline 2017

76

ASCO-CCO guideline 2017: IB•A pooled exploratory analysis based on two RCTs found a non-significant trend for increased chemotherapy effect on OS with larger tumor size

•Exploratory subgroup analysis of the NSCLCCG meta-analysis found no significant difference in the effect of adjuvant chemotherapy on survival by stage

77

ASCO-CCO 2017 Other Factors (outside trial)

•Tumor size : 5 year survival rate•3-4 cm : 74%, 4-5 cm : 65%

•Histo : PNI, tumor necrosis, LVI, visceral pleural invasion, mitotic index >10 mitoses/10 HPF

•High risk subtype of adenoCA: micropapillary or solid

78

Synthesis: ASCO-CCO guideline 2017• Using the LACE data• Increase 5 year OS• IB 64% to 67% • II 39% to 49% • IIIA 26% to 39% • blue= die either chemo or not

• Yellow =live without chemo

• Grey = live due to chemo

• Red = die due to chemo

79

So should we give chemo for IB?

• LACE IPD pooled analysis (pre-specified, positive interaction test, subgroup effect)stage IA/IB vs II, III or IA vs IB, II, III ?

• CALGB 6933 IB stage prelim data significant effect• CALGB 6933 IB stage mature data negative trial• CALGB 6933 & JBR.10 exploratory subgroup T>=4 cm (later

abstract negative), positive interaction test • NSCLCCG IPD meta (pre-specified, negative interaction test)

failed to prove difference of treatment in I, II, III• NSCLCCG IPD meta (exploratory, negative interaction test)

failed to prove this difference except IA?• Consider chemo in stage IB ?

81

VOTE: Do you give chemotherapy for Postop R0 NSCLC stage IB ? Second Round

1. YES, in Majority of patients2. NO, in minority of patients3. Y/N, in only special group T >=4 cm 4. Y/N, in at least one of high risk features5. Y/N, in only significant or many high risk

features 82

REASONS for VOTEYES, Majority of patients

1. From CALGB9633 abstract 2011: there is a POWERFUL TREND OS in all pts (HR=0.80; 90% CI 0.63-1.02; p=0.062, 1-tail), and 7% advantage (51% vs. 44%; p=0.087)

2. From NSCLCCG shows no treatment difference (negative interaction test) from stage II-III, absolute benefit is 5%, the same as II-III

3. Other personal reasons83

REASONS for VOTENO, Minority of patients

1. LACE IPD meta •Toxicity G3-4 =66% , toxicity G4 = 32% (G4 neutropenia 28%)

•Whereas survival benefit in IB only 3%2. Negative CALGB 6933 trial, so there is no

strong data support. We should play safe. Better safe than sorry

3. Other person reasons84

REASONS for VOTEYES, in special group T >=4 cm (page 1/2)1. CALGB 9633 JCO 2008:

Survival difference in T>=4 cm (HR, 0.69; CI, 0.48 to 0.99;P=0.043) and

CALGB 9633 abstract 2011:STRONG OS SIGNAL for those with large tumors. • T ≥4 cm, HR = 0.77 (90% CI: 0.57-1.04; p=0.079); • 10% improve in 8 yr OS (53% vs.43%; p=0.059); • T<4cm little OS diff (HR=0.89; p=0.31).

85

REASONS for VOTEYES, in special group T >=4 cm (page 2/2)

2. Positive interaction test (p=0.02) in JBR.10 trial with reversed effect•T>=4 cm (HR, 0.66; 95% CI, 0.39-1.14;p 0.13)•T< 4 cm (HR, 1.73; 95% CI, 0.98-3.04)3. Other personal reasons

86

REASONS for VOTEYES, in at least one of high risk features 1/2

1. Believe NCCN, • Even no explicit reasons but these high risk factors are reasonable and likely from reliable source

• NCCN doesn’t tell us weighting/ how many risk factors for considering chemotherapy, so at least one should be enough

87

REASONS for VOTEYES, in at least one of high risk features 2/2

2. Believe ASCO-CCO 2017, • Most update one• From NSCLCCG shows no treatment difference (negative interaction test) IB from stage II-III, absolute benefit is 5%, the same as II-III

• Give detail of literature review of these factors (size, high risk histo features, histo subtypes)

88

REASONS for VOTEYES, in only significant or many high risk features

With the ceiling of benefit around 3% (LACE), 5% (NSCLCCG) and 8% (CALGB at 4 year) and high toxicity G3-4 =66%, G4 = 32% (G4 neutropenia 28%), so it is worthwhile to pursue chemotherapy only in very high risk group

89

VOTE: Do you give chemotherapy for Postop R0 NSCLC stage IB ? FINALROUND

1. YES, in Majority of patients2. NO, in minority of patients3. Y/N, in only special group T >=4 cm 4. Y/N, in at least one of high risk features5. Y/N, in only significant or many high risk

features 90

My feeling (not conclusion) 1/3• IPD meta Cochrane 2015, including CALGB 9633 total =1000+/arm

“We split stage I disease into IA and IB for all but five trial comparisons (A04 Park1; A19 SGACLC ACTLC1; A21 SGACLC ACTLC2; A26 OLCSG2b; A32 OLCSG2a) which we had to exclude since this information was not available”• CALGB 9633 abstract 2011 : underpowered

• HR=0.80; 90% CI 0.63-1.02; p=0.062, 1-tail

91

My feeling (not conclusion) 2/3

•CALGB 9633 and IPD meta Cochrane 2015 are conflicting?

•Which one is more reliable? •Non statistical significance of interaction test = the same treatment benefit of stage IB, II, III?

92

My feeling (not conclusion) 2/3

•CALGB 9633 and IPD meta Cochrane 2015 are conflicting? NO

•Which one is more reliable? RCT vs IPD meta (stage is stratification factor?, diff. chemo in diff. stage from diff. trial?

•Non statistical significance of interaction test = the same treatment benefit of stage IB, II, III? Not necessary

93

My feeling (not conclusion) 3/3

• CALGB and IPD meta could reach the same conclusion but• Minute benefit but inadequate sample size?

Survival benefit 3% LACE, 5% NSCLCCG, 8% CALGB at 4 year • It’s not therapeutic question but prognostic/predictive question

because of heterogeneity? • subgroup question in IB: exploratory analysis in CALGB needed to

be validated before use• Prediction model to find direct evidence from high-risk patients from RCTs

(not from outside trial) • Size ? Chemotherapy regimens ? (Carbo+ Pac ?/ Cis ?)

• Decision analysis model (from RCTs data + cost) 94

Discussion & Conclusion from medical oncologists, please

95

We could learn BETTER from..•Different recommendations are challenging•Find specific evidence (RCTs, meta-analysis) /other reasons for each recommendation

•Use skill of statistical/clinical reasoning •Discuss with your friends & supervisors

97

อาจารย์ : ทําไมคณุเลอืกใหเ้คมบํีาบดัในรายนี้

• (แพทยป์ระจําบา้นปี 2) ผมอา่นจาก NCCN เดมิไม่ recommend แต่ version ใหมใ่ชคํ้าวา่ “consider” แลว้ครับ และ Category 1 ดว้ย ผมคยุกบัเพือ่นแลว้ตคีวาม “CONSIDER” คอืวา่สามารถใหไ้ดค้รับ

• (แพทยป์ระจําบา้นปี 3) หนูไมไ่ดอ้า่น NCCN อยา่งเดยีวคะ่ แตก่ลบัไปดดูว้ยวา่เขาอา้ง evidence อะไร พบวา่เขาอา้ง benefit ของบาง subgroup ของ RCT ตพีมิพใ์น JCO คะ่ และเป็นหลกัฐานทีเ่ชือ่ไดว้า่ควรใหค้ะ่

• (FELLOW) ถงึแม ้NCCN “consider” และมี RCT evidence แตเ่ป็นexploratory/unplanned subgroup ครับ ผมดู paper แลว้ RCT ที่ NCCN นํามาอา้ง มขีอ้น่าสงสยัหลายประการครับ และยงัมี subgroup จาก IPD meta-analysis ทีผ่ลขดัแยง้กบั NCCN ครับ ขออนุญาตนํามาอภปิรายกนักอ่นตดัสนิใจครับ

98

Credits:

99