Two bootstrapping routines for obtaining uncertainty measurement around the nonparametric distribution obtained

in NONMEM VIPaul G. Baverel 1, Radojka M. Savic 2 and Mats O. Karlsson 1

1 Division of Pharmacokinetics & Drug Therapy, Uppsala University, Sweden2 INSERM U738, University Paris Diderot Paris 7, Paris, France

Two novel bootstrapping routines intended for nonparametric estimation methods are proposed. Their evaluation with a simple PK model in the case of informative sampling design was performed when applying FOCE- NONP in NONMEM VI but it is easily transposable to other nonparametric applications.

These tools can be used for diagnostic purpose to help detecting misspecifications with respect to the distribution of random effects.

From the sampling distribution obtained, standard uncertainty metrics, such as standard errors and correlation matrix can be derived in case reporting uncertainty is intended. References: [1]. Perl-speaks-NONMEM (PsN software): L. Lindbom, M. Karlsson, N. Jonsson.

http://psn.sourceforge.net[2]. Savic RM, Baverel PG, Karlsson MO. A novel bootstrap method for obtaining uncertainty around the nonparametric distribution. PAGE 18 (2008) Abstract 1390.

[3]. Efron B. Bootstrap methods: another look at the jackknife. Ann Stat 1979; 7:1-26.

Overall, the trend and the magnitude of the 95% CI derived with the full and the simplified nonparametric bootstrapping methods (N=100 and N=500) matched the true 95% CI in all distributional cases and regardless of individual numbers in original data.

The simplified version induced slightly less bias in quantifying uncertainty (prediction errors of 95% CI width) than the full version. This is expected as the former methodology derives uncertainty from the original data.

Quantifying uncertainty in parameter estimates is essential to support

decision making throughout model building process.

Despite providing enhanced estimates of parameter distribution,

nonparametric algorithms do not yet supply uncertainty metrics.

Two different permutation methods automated in PsN [1] were developed to quantify uncertainty around NPD (95% confidence interval) and nonparametric estimates (SEs and variance-covariance matrix):

o The full method [2] relies on N bootstraps of the original data and a re-analysis of both the preceding parametric as well as the nonparametric stepo The simplified method relies on N bootstrap samples of the vectors of individual probabilities associated with each unique support point of the NPD

Six informative datasets of 50 or 200 individuals were simulated from an

IV bolus PK model in which CL and V conformed to various underlying

distributional shapes (log-normal, bimodal, heavy-tailed). Residual

variability was set to 10% CV.

Re-estimation was conducted assuming normality under FOCE, and NPDs were estimated by applying FOCE-NONP method in NONMEM VI.

Figure 1: Sequential steps of the operating procedure of both the full and simplified nonparametric bootstrapping methods intended for nonparametric estimation methods.

Figure 2. On the left : 95% confidence intervals obtained based on the full and simplified nonparametric bootstrapping methodologies in case of various underlying distributions of CL. The true 95% CI around the true parameter distribution is also represented for comparison, as well as the parametric cumulative density function.On the right: Prediction errors of the 95% CI width are displayed for each quartile of parameter distribution, the true uncertainty being taken as reference.

UNDERLYING BIMODAL CLEARANCE DISTRIBUTION

UNDERLYING HEAVY-TAILED CLEARANCE DISTRIBUTION (200 IDs)

Simplified nonparametric bootstrapping method: 5-step procedure (ca 2 mn.)

Full nonparametric bootstrapping method: 7-step procedure (CPU time: ca 4 hr.)

BOOTSTRAP* N times

original data DJ

Bootstrapped

data B1...BN

PARAMETRIC ESTIMATION:

B1... BN

N sets (θ,Ω,σ) each defined at

<J support points

NONPARAMETRIC

ESTIMATION:

DJ given (θ,Ω,σ)

N sets NPDN

defined at J support points for <J

individuals in B1...BN

PARTITIONING

NPDN into J

individual probability densities IPDN

BOOTSTRAP*

IPDN according to

sample scheme B1...BN

NxN matrices MbootN of

bootstrapped IPD<J

RE-ASSEMBLING

bootstrapped IPD<J From NPDnewN construct

nonparametric 95% CI around NPD Derive SEs and correlation

matrix of nonparametric estimates

1 2 3 4 5

6

7 5

PARTITIONING

NPD into J

individual probability densities IPDJ

NONPARAMETRIC ESTIMATION:

Original data DJ

Matrice M (JxJ) of

individual probabilities: Row entries J individuals

Column entries J support points

NPD defined at J support points

BOOTSTRAP

IPDJ

N times

RE-ASSEMBLING

bootstrapped IPDJ

N matrices MbootN

of bootstrapped IPDJ

1 2 3 4

50 IDs200 IDs

The true uncertainty was derived by standard nonparametric bootstrapping (N=1000) [3] of the true individual parameters and used as reference for qualitative and quantitative assessment of the uncertainty measurements derived from both techniques.

CL distributionC

um

ula

tive d

ensi

ty f

unct

ion

10 20 30 40 50 60

0.0

0.2

0.4

0.6

0.8

1.0

Q1 Q2 Q3 Q4

TrueParametric95% CI true95% CI simplified (N=100)95% CI full (N=100)

CL distribution

Cum

ula

tive d

ensi

ty f

unct

ion

5 10 15 20 25 30 35

0.0

0.2

0.4

0.6

0.8

1.0

Q1 Q2 Q3 Q4

TrueParametric95% CI true95% CI simplified (N=100)95% CI full (N=100)

N sets NPDnewN defined at J support

points of NPDN

N matrices MN (JxJ) of individual

probabilities: Row entries J individuals

Column entries J support points

A single set of NPDnewN defined at J support points of NPD

Prediction errors

-0.05

0.00

0.05

CL

200 IDs

-0.10

-0.05

0.00

0.05

CL

50 IDs

Simplified Full

Q1 Q2 Q3 Q4

CL distribution

10 20 30 40

0.0

0.2

0.4

0.6

0.8

1.0

Q1 Q2 Q3 Q4

TrueParametric95% CI true95% CI simplified (N=100)95% CI full (N=100)

N=100 Prediction errorsN=500

Table 1. SEs of parameter estimates obtained from 100 stochastic simulations followed by estimations given the true model under FOCE, FOCE-NONP and the analytical solution under FOCE ($COVARIANCE) in NONMEM VI. The simplified methodology was applied to each simulated dataset; SEs were computed and average SEs were reported for comparison with SSE.

SE100 Stochastic Simulations followed by Estimations (SSE)

Simplified nonparametric

bootstrap version

True empirical FOCE

Asymptotic ($COV)FOCE

True empirical FOCE-NONP

(N=100)FOCE-NONP

Θ CL 0.022 0.022 0.021 0.021Θ V 0.024 0.022 0.024 0.02

Ω CL 0.01 0.009 0.01 0.008 Ω CL,V 0.007 0.006

Ω V 0.011 0.01 0.011 0.008

SEs obtained with the simplified methodology matched the ones obtained by SSE.

CL distribution

5 10 15 20 25 30 35

0.0

0.2

0.4

0.6

0.8

1.0

Q1 Q2 Q3 Q4

TrueParametric95% CI true95% CI simplified (N=500)95% CI full (N=500)

-0.05

0.00

0.05

CL

N=100

-0.05

0.00

0.05

CL

Simplified Full

N=500

Q1 Q2 Q3 Q4