![Page 1: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/1.jpg)
Issues of Simultaneous Tests for Non-Inferiority and Superiority
Tie-Hua Ng*, Ph. D.U.S. Food and Drug Administration
Presented atMCP 2002
August 5-7, 2002Bethesda, Maryland
_______
* The views expressed in this presentation are not necessarily of the U.S. Food and Drug Administration.
![Page 2: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/2.jpg)
2
Simultaneous Tests for Non-Inferiority and Superiority
• Multiplicity adjustment is not necessary– Intersection-union principle (IU)
• Dunnett and Gent (1996)
– Closed testing procedure (CTP)• Morikawa and Yoshida (1995)
• Indisputable
![Page 3: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/3.jpg)
3
A Big Question
Is Multiplicity Adjustment Necessary?
![Page 4: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/4.jpg)
4
Is Is MultiplicityMultiplicityAdjustment Adjustment Necessary?Necessary?
![Page 5: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/5.jpg)
5
Outline• Assumptions and Notations
• Switching between Superiority and Non-Inferiority
• Is Simultaneous Testing Acceptable?
• Use of Confidence Interval in Hypothesis Testing --- Pitfall
• Problems of Simultaneous Testing
• Conclusion
![Page 6: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/6.jpg)
6
Assumptions/Notations• Normality and larger is better• T: Test/Experimental treatment (t)
• S: Standard therapy/Active control (s): Non-Inferiority Margin (> 0)• For a given d (real number), define
– Null: H0(d): T S - d – Alternative: H1(d): T > S - d
• Non-Inferiority: d = • Superiority: d = 0
![Page 7: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/7.jpg)
7
Non-Inferiority (d = )
H0(): T S - against H1(): T > S -
H0() H1()
•°
TBoundaryWorse
Better
Mean Response
S
![Page 8: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/8.jpg)
8
Superiority (d = 0)
H0 (0): T S against H1 (0): T > S
H0(0)H1(0)
•°
TBoundary
Worse
Better
Mean Response
S
![Page 9: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/9.jpg)
9
Switching between Superiority and Non-Inferiority
CPMP (Committee for Proprietary Medicinal Products), European Agency for the Evaluation of Medicinal Products
Points to Consider on Switching Between Superiority and Non-Inferiority, 2000.
http://www.emea.eu.int/htms/human/ewp/ewpptc.htm
![Page 10: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/10.jpg)
10
Switching between Superiority and Non-Inferiority (2)
• Non-Inferiority Trial– If H0() is rejected, proceed to test H0(0)
– No multiplicity issue, closed testing procedure
• Superiority Trial– Fail to reject H0(0), proceed to test H0()
– No multiplicity issue
– Post hoc specification of
![Page 11: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/11.jpg)
11
Switching between Superiority and Non-Inferiority (3)
• Non-inferiority Trial– Intention-to-treat (ITT)– Per protocol (PP)
• Superiority Trial– Primary: Intention-to-treat (ITT)– Supportive: Per protocol (PP)
• Assume ITT = PP
![Page 12: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/12.jpg)
12
Simultaneous Testing
One-sided 100(1 - )% lower Confidence Interval for T - S
Test is worse
Test is betterMean Difference (T – S)
0-
Superiority
Non-inferiority
Neither
![Page 13: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/13.jpg)
13
Simultaneous Testing (2)
• Multiplicity adjustment is not necessary– Dunnett and Gent (1996)
• Intersection-Union (IU):
Superiority: Both H0() and H0(0) are rejected
– Morikawa and Yoshida (1995)• Closed Testing Procedure (CTP):
Test H0(0) when H0()H0(0) is rejected
![Page 14: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/14.jpg)
14
Simultaneous Testing (3)
• Discussion Forum (October 1998)– London
– PSI (Statisticians in Pharmaceutical Industry)
• Is Simultaneous Testing of Equivalence [Non-Inferiority] and Superiority Acceptable?– Superiority trial:
• Fail to reject H0 (0)
• No equivalence/non-inferiority claim
– Ok: Morikawa and Yoshida (1995)
• Ref: Phillips et al (2000), DIJ
![Page 15: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/15.jpg)
15
Is Simultaneous
TestingAcceptable?
![Page 16: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/16.jpg)
16
Use of Confidence Interval in Hypothesis Testing
H0(d): T S - d (at significance level )
One-sided 100(1-)% lower CI for T-S
Reject H0(d) if and only if the CI excludes -d
Test is worse
Test is better
Mean Difference (T – S)
-d
Reject H0(d)
Do not reject H0(d)
![Page 17: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/17.jpg)
17
Use of Confidence Interval in Hypothesis Testing (2)
• If CI = (L, ), then H0(d) will be rejected for all -d < L.
• A Tricky Question– Suppose CI = (-1.999, ), L = -1.999
• H0(2): T S - 2 is rejected (d=2) since -d < L• Can we conclude that T > S - 2?
• Yes, if H0(2) is prespecified.• No, otherwise.
![Page 18: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/18.jpg)
18
Use of Confidence Interval in Hypothesis Testing (3)
Post hoc specification of
H0(d) is a
No No
![Page 19: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/19.jpg)
19
Simultaneous Testing: Problems
H0(d1) and H0(d2), for d1 > d2
One-sided (1 - )100% lower CI for T - S
Test is worse
Test is betterMean Difference (T – S)
-d2-d1
Reject H0(d2)
Reject H0(d1)
Neither
![Page 20: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/20.jpg)
20
Simultaneous Testing: Problems (2)
H0(d1), H0(d2) and H0(d3), for d1 > d2 > d3
One-sided (1 - )100% lower CI for T - S
Test is worse
Test is betterMean Difference (T – S)
-d3-d1
Reject H0(d3)Reject H0(d2)
None
-d2
Reject H0(d1)
![Page 21: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/21.jpg)
21
Simultaneous Testing: Problems (3)
H0(d1), H0(d2),…, H0(dk), for d1 > d2 > … > dk
One-sided (1 - )100% lower CI for T - S
Test is worse
Test is betterMean Difference (T – S)
-dk-d1
Reject H0(dk)
Reject H0(d2)
None
-d2
Reject H0(d1)
…
…
.
.
.
-d3
![Page 22: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/22.jpg)
22
Simultaneous Testing: Problems (4)
• Choose k large enough Pr[-d1 < Lower limit < -dk] close to 1
Max |dk - dk-1| < a given small number
Simultaneous testing of H0(di), i = 1,…, k
Post hoc specification of H0(d)
![Page 23: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/23.jpg)
23
Simultaneous Testing: Problems (5)
Number of Nested hypotheses
Exploratory (many H0(d))
Confirmatory (one H0(d))
1 2 3 4 …………. k …………
Simultaneous H0() and H0(0)
![Page 24: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/24.jpg)
24
Simultaneous Testing: Problems (6)
• What is wrong with IU and CTP?
• Nothing
• Pr[Rejecting at least one true null] • What kind of problems?
![Page 25: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/25.jpg)
25
Simultaneous Testing: Problems (7)
• Post hoc specification of H0(d)
Let -d0 = 100(1 - )% lower limit -
Reject H0(d0), since -d0 < lower limit
Repeat the same trial independently
Pr[Rejecting H0(d0)] = 0.5 +
![Page 26: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/26.jpg)
26
Simultaneous Testing: Problems (8)
• Simultaneous testing of many H0(d)
– Repeat the same trial independently
– Low probability of confirming the finding
• 1st trial: Reject H0(dj) but not H0(dj+1)
• 2nd trial: Pr[Rejecting H0(dj)] is low (e.g., 0.5+)
![Page 27: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/27.jpg)
27
Simultaneous Testing: Problems (9)
• Simultaneous testing of H0() and H0(0)?
• Confirm the finding = 2 Known variance Let T - S Significance level = 0.025 80% power for H0() (at = 0)
![Page 28: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/28.jpg)
28
Simultaneous Testing: Problems (10)
f() = Pr[Rejecting H0() | ]
f0() = Pr[Rejecting H0(0) | ]
![Page 29: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/29.jpg)
29
Simultaneous Testing: Problems (11)
• Test one null hypothesis H0()
• Suppose that H0() is rejected
• Repeat the same trial independently
• Pr[Rejecting H0() again] = f()
![Page 30: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/30.jpg)
30
Simultaneous Testing: Problems (12)
• Test H0() and H0(0) simultaneously
• Suppose that H0() or H0(0) is rejected
• Repeat the same trial independently• Pr[Rejecting the same null hypothesis again] =
[1 - w()] · f() + w() · f0()
= f() - f0() [1 – f0()/f()],
where w() = f0()/f()
![Page 31: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/31.jpg)
31
Simultaneous Testing: Problems (13)
[1 - w()] · f() + w() · f0()
where w() = f0()/f()
Simultaneous tests in the 2nd trial
![Page 32: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/32.jpg)
32
Simultaneous Testing: Problems (14)
• Ratio: 1 – [f0()/f()] [1 – f0()/f()]
• Ratio may be as low as 0.75
![Page 33: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/33.jpg)
33
Conclusion
• Many H0(d): Problematic
• Not type I error rate
• H0() and H0(0): Acceptable?
• If “zero tolerance policy”: No
• If 25% reduction cannot be tolerated: No
• If 25% reduction can be tolerated: Yes
![Page 34: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/34.jpg)
34
Is Simultaneous Testing of
H0() and H0(0) Acceptable?
![Page 35: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/35.jpg)
35
You be the judge
![Page 36: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/36.jpg)
36
References• Dunnett and Gent (1976), Statistics in Medicine, 15,
1729-1738.
• Committee for Proprietary Medicinal Products (CPMP; 2002). Points to Consider on Switching Between Superiority and Non-Inferiority. http://www.emea.eu.int/htms/human/ewp/ewpptc.htm
• Morikawa T, Yoshida M. (1995), Journal of Biopharmaceutical Statistics, 5:297-306.
• Phillips et al., (2000), Drug Information Journal, 34:337-348.
![Page 37: Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Ng@cber.fda.gov Presented at MCP](https://reader034.vdocuments.site/reader034/viewer/2022042814/551b8bc1550346167e8b4fd1/html5/thumbnails/37.jpg)
37