uncertainty and sensitivity analysis in performance assessment for
TRANSCRIPT
ELSEVIER Computer Physics Communications 117 (1999) 156-180
Computer Physics Communications
Uncertainty and sensitivity analysis in performance assessment for the Waste Isolation Pilot Plant
J.C. H e l t o n 1 Department of Mathematics, Arizona State University, Tempe, AZ 85287-1804, USA
Received 22 April 1998
Abstract
The Waste Isolation Pilot Plant (WIPP) is under development by the U.S. Department of Energy (DOE) for the geologic (deep underground) disposal of transuranic (TRU) waste. This development has been supported by a sequence of performance assessments (PAs) carried out by Sandia National Laboratories (SNL) to assess what is known about the WIPP and to provide guidance for future DOE research and development activities. Uncertainty and sensitivity analysis procedures based on Latin hypercube sampling and regression techniques play a major role in these PAs by providing an assessment of the uncertainty in important analysis outcomes and identifying the sources of this uncertainty. Performance assessments for the WIPP are conceptually and computationally interesting due to regulatory requirements to assess and display the effects of both stochastic (i.e., aleatory) and subjective (i.e., epistemic) uncertainty, where stochastic uncertainty arises from the possible disruptions that could occur over the 10000 yr regulatory period associated with the WIPP and subjective uncertainty arises from an inability to unambiguously characterize the many models and associated parameters required in a PA for the WIPE The interplay between uncertainty analysis, sensitivity analysis, stochastic uncertainty and subjective uncertainty is discussed and illustrated in the context of a recent PA carried out by SNL to support an application by the DOE to the U.S. Environmental Protection Agency for the certification of the WIPP for the disposal of TRU waste. L~ 1999 Elsevier Science B.V.
Keywords: Aleatory uncertainty; Epistemic uncertainty; Performance assessment; Radioactive waste disposal; Sensitivity analysis; Stochastic uncertainty; Subjective uncertainty; Uncertainty analysis; Waste Isolation Pilot Plant
1. Introduction
The Waste Isolation Pilot Plant (WIPP) is under development by the U.S. Department of Energy (DOE) for the geologic (deep underground) disposal of transuranic (TRU) waste [ 1-3]. The WIPP is located in southeastern New Mexico, with waste disposal planned to take place in excavated chambers (i.e., waste panels) in a salt formation approximately 2000 ft below the land surface (Fig. l ) .
i Dept. 6848, MS 0799, Sandia National Laboratories, Albuquerque, NM 87185-0799, USA; e-mail: [email protected].
0010-4655/99/$ - see front matter (~) 1999 Elsevier Science B.V. All rights reserved. PII S001 0-4655 (98)00 17 1-4
J.C. Helton / Computer Physics Communications 117 (1999) 156-180 157
Salt Storage Waste Handling Building Area (Elevation: 3409 It AMSL)
~n~lin~Sh~It Sh~It
EleVa,[l~, '.
Exper imen ta l A r e ~
operation Area
Shaft Seal
~ , ~ Drift Seal
Current Excavation
~- ~ Planned Excavaliofl ~' ~ ~
Fig. 1. Cross-sectional view of the W1PP 14, Fig. 1-9, Vol. 11; see 15, Section 2.2, Vol. 2] for detailed slratigraphy.
An important part of the development process for the WIPP has been a series of uncertainty and sensitivity analyses carried out by Sandia National Laboratories (SNL) to assess the current state of knowledge with respect to the WIPP and to provide guidance for future model development and research activities [4-7] , with these analyses having been extensively reported in the journal literature [8-13]. The most recent uncertainty and sensitivity analyses [14,15] have been carried out in support of an application by the DOE to the U.S. Environmental Protection Agency (EPA) for the certification of the WIPP for the disposal of TRU waste (i.e., the compliance certification application or CCA) [16].
In response to the CCA, the EPA certified the WIPP for the disposal of TRU waste in May 1998 [17]. At present (August 1998), several legal actions are pending that seek to block the opening of the WIPE If these actions are unsuccessful, the WIPP will be the first facility in the United States to begin operation for the geologic disposal of radioactive waste.
Regulations promulgated by the EPA (i.e., 40 CFR 191, Subpart B [18,19]) determined the nature of the calculations performed by SNL to support the CCA and also the uncertainty and sensitivity analyses implemented as part of these calculations. The following is the central requirement of 40 CFR t91, Subpart B, and the primary determinant of the structure of the analysis (i.e., performance assessmenl or PA) carried out to support the CCA:
§ 19 I. 13 Containment requirements. (a) Disposal systems for spent nuclear fuel or high-level or transuranic radioactive wastes shall be designed
to provide a reasonable expectation, based upon performance assessments, that cumulative releases of radionuclides to the accessible environment for 10000 years after disposal from all significant processes and events that may affect the disposal system shall: (1) Have a likelihood of less than one chance in 10 of exceeding the quantities calculated according to Table 1; and (2) Have a likelihood of less than one chance in 1000 of exceeding ten times the quantities calculated according to Table 1.
(b) Performance assessments need not provide complete assurance that the requirements of 191.13(a) will be met. Because of the long time period involved and the nature of the events and processes of interest,
158 J.C. Helton/Computer Physics Communications 117 (1999) 156-180
there will inevitably be substantial uncertainties in projecting disposal system performance. Proof of the future performance of a disposal system is not to be had in the ordinary sense of the word in situations that deal with much shorter time frames. Instead, what is required is a reasonable expectation, on the basis of the record before the implementing agency, that compliance with 191.13(a) will be achieved.
Containment Requirement 191.13 (a) refers to "quantities calculated according to Table I", which means a normalized radionuclide release to the accessible environment based on the type of waste being disposed of, the initial waste inventory, and the release that takes place [ 18, Appendix A]. Table 1 (Appendix A) of [18] specifies allowable releases (i.e., release limits) for individual radionuclides. The WIPP is intended for TRU waste, which is defined to be "waste containing more than 100 nanocuries of alpha-emitting transuranic isotopes, with half-lives greater than twenty years, per gram of waste" [ 18, p. 38084]. Specifically, the normalized release R for transuranic waste is defined by
R = ~ ' ~ ( Q i / L i ) ( I x 106 C i / C ) , ( I )
where Qi is the cumulative release of radionuclide i to the accessible environment during the 10000-yr period following closure of the repository (Ci), Li is the release limit (Ci) for radionuclide i (Table 1, [ 18] ) and C is the amount of TRU waste emplaced in the repository (Ci). For the 1996 WIPP PA (i.e., the PA carried out in support of the CCA), C = 3.44 x 106 Ci.
To help clarify the intent of 40 CFR 191, the EPA also published 40 CFR 194 [20]. There, the following elaboration on the intent of 40 CFR 191.13 appears [20, pp. 5242-5243]:
§ 194.34 Results of performance assessments. (a) The results of performance assessments shall be assembled into "complementary, cumulative distribution
functions" (CCDFs) that represent the probability of exceeding various levels of cumulative release caused by all significant processes and events.
(b) Probability distributions for uncertain disposal system parameter values used in performance assessments shall be developed and documented in any compliance application.
(c) Computational techniques, which draw random samples from across the entire range of the probability distributions developed pursuant to paragraph (b) of this section, shall be used in generating CCDFs and shall be documented in any compliance application.
(d) The number of CCDFs generated shall be large enough such that, at cumulative releases of 1 and 10, the maximum CCDF generated exceeds the 991h percentile of the population of CCDFs with at least a 0.95 probability.
(e) Any compliance application shall display the full range of CCDFs generated. (f) Any compliance application shall provide information which demonstrates that there is at least a 95
percent level of statistical confidence that the mean of the population of CCDFs meets the containment requirements of §191.13 of this chapter.
An interesting feature of analyses to assess compliance with 191.13(a) and (b) is the requirement to incorporate two distinct treatments of uncertainty [21-24]. First, there is the uncertainty that leads to the CCDF specified in 191.13(a). This uncertainty results from an assumed randomness in what will occur at the WIPP site over the next 10000 yr. Numerous designations for such uncertainty have been used in the literature, including stochastic, aleatory, variability, irreducible and type A. Second, there is the uncertainty characterized by the distributions called for in 194.34(b). This uncertainty results from a lack of knowledge about models and parameter values required in the construction of the CCDF specified in 191.13(a) and leads to the distribution of CCDFs called for in 194.34(e). Numerous designations for this characterization of uncertainty have also been used, including subjective, epistemic, state of knowledge, reducible and type B. In this presentation, stochastic and subjective will be used as the designations for these two types of uncertainty. Decompositions
J.C. Helton/Computer Physics Communications 117 (1999) 156-180 159
. + ,0, +++uo+ ^ 1o-1 ~ -...~.;1+113(+1
-,~ 10+2 (10, o.001)
1o-3 ""+ O- ,~st ~ - Density F u n c t l o n ~
10-4 - \
,, where \
1o-+ { /x, x
0 J . ~ l I I I 1 I I I
0 10 - 5 10-4 10 -3 10 -2 10 -1 10 0 101 10 2
R : Release to Accessible Environment
Fig. 2. Boundary line and associated CCDF specified in 191.13(a) 115, Fig. 4].
of uncertainty in large analyses similar to that implemented in the WIPP PA have been widely discussed from various perspectives [25-30].
The terms uncertainty analysis and sensitivity analysis as used in analyses lbr the WIPP refer to assessments of the effects of subjective uncertainty. Specifically, uncertainty analysis designates an investigation of the uncertainty in model predictions that results from uncertainty in model inputs, and sensitivity analysis designates an investigation to determine the effects of individual variables on model predictions. Stochastic uncertainty enters into uncertainty and sensitivity analyses for the WIPP because the CCDF specified in 191.13(a), which derives from stochastic uncertainty, is the single most important result considered in such analyses.
2. Overview of PA for the WIPP (adapted from [15,33])
When viewed at a high level, three basic entities underlie the results required in 191.13 and 194.34 and ultimately determine the conceptual and computational structure of the 1996 WIPP PA: ENI, a probabilistic characterization of the likelihood of different futures occurring at the WIPP site over the next 10000 yr; EN2, a procedure for estimating the radionuclide releases to the accessible environment associated with each of the possible futures that could occur at the WIPP site over the next 10000 yr; and EN3, a probabilistic characterization of the uncertainty in the parameters used in the definitions of EN1 and EN2. The term parameter employed in the description of EN3 is used in a sense that is sufficiently broad to include designators for alternative models or model structures. In the terminology of Draper [31] and Draper et al. [32], ENI characterizes scenario uncertainty; EN2 involves models and their structure; and EN3 characterizes uncertainty in both model structure and parameters used within models. Together, ENI and EN2 give rise to the CCDF specified in 191.13(a) (Fig. 2), and EN3 corresponds to the distributions indicated in 194.34(b). In the context of PA for the WIPE uncertainty and sensitivity analysis involves the determination of the effects of the uncertainty characterized by EN3 on results that derive from ENI and EN2.
The entity EN1 is the outcome of the scenario development process for the WIPP and provides a probabilistic characterization of the likelihood of different futures that could occur at the WIPP over the next 10000 yr as specified in 191.13(a). When viewed formally, EN1 is defined by a probability space (S~t, ,S~t,p~t), with the sample space S,~t given by
S~t -- {xst : x~t is a possible 10000 yr sequence of occurrences at the WIPP}. (2)
Further, in consistency with the usual representation for a probability space, ,S~t is a suitably restricted collection
160 J.C. Helton/Computer Physicw Communications 117 (1999) 156-180
c.~'- m c
E ~ m c
. ~ - [ =
(.)
CU'I-FINGS_S ~ (Release of Cuttings, Cavings, Spellings, Brine to BRAGFLO_DBR ~ E n v i r o n m e n t )
GRASP-INV ransmissivi Fields I
~ ~ SECOFL2D/SECOTP2D ~ ,
-~ I ~'~--Upper ShaltSe I System I
~1 ystem FL /2-Phase Flow/'~ I' ~ ~ V ~ ,, MB138
p~nel~~ An~hychite Layers A and B
/P [ BRAGFLO~'~. MB,39 Panel Seal Bnne Flow PANEL/NUTS ( )
V (Radionuclide Concentration)
Subsurface /" Brine ~ Boundary
Reservoir ~ of Accessible ~ Not to Scale Environment
Fig. 3. Computer programs (models) used in 1996 WIPP PA[ 15, Fig. 5].
of subsets of Sst that have defined probabilities, and Pst is the function (i.e., a probability measure) that actually defines the probabilities of the individual elements of Sst [34, p. 116]. The subscript 'st ' refers to stochastic uncertainty and is used because (S~t, Sst, pst) is providing a probabilistic characterization of random occurrences that may take place in the future.
Guidance by the EPA [20] and an extensive review of possible disruptions at the WIPP led to drilling intrusions and potash mining being the only occurrences incorporated into the definition of Sst. Specifically, the elements xst of S~t are vectors of the form
xst = [ tl, Ii, el, bl, p], a], t2, 12, e2, b2, P2, a2 . . . . . tn, l,, e, , b,, p,,, a,,, train ]
1 st in~-rusion 2nd in~l'usion nth intrusion
(3)
in the 1996 WIPP PA, where n is the number of drilling intrusions, ti is the time (yr) of the ith intrusion, li
designates the location of the ith intrusion, ei designates the penetration of an excavated or nonexcavated area by the ith intrusion, bi designates whether or not the ith intrusion penetrates pressurized brine in the Castile Formation, pi designates the plugging procedure used with the ith intrusion (i.e., continuous plug, two discrete plugs, three discrete plugs), ai designates the type of waste penetrated by the ith intrusion (i.e., no waste, contact-handled (CH) waste, remote-handled (RH) waste), and tmin is the time (yr) at which potash mining occurs. A full characterization of (S~t, S~t, Pst) then results from the definition of a probability distribution for each element of Xst [ 14, ch. 3].
The entity EN2 is the outcome of the model development process for the WIPP and provides a way to estimate radionuclide releases to the accessible environment (i.e., values for Qi, and hence R, in Eq. (1 ) ) for the different futures (i.e., elements x~t of Sst) that could occur at the WIPE Estimation of environmental releases corresponds to evaluation of the function f in Fig. 2. Release mechanisms associated with f include direct removal to the surface at the time of a drilling intrusion (i.e., cuttings, spallings, direct brine flow) and release subsequent to a drilling intrusion due to brine flow up a borehole with a degraded plug (i.e., groundwater transport). The primary computational models in the 1996 WIPP PA are illustrated in Fig. 3 and described in more detail in Table 1. Most of these models involve the numerical solution of partial differential equations used to represent material deformation, fluid flow or radionuclide transport. It is the models in Fig. 3 that actually define the function f in Fig. 2; a detailed description of these models is given in ch. 3 of [ 14].
As indicated in Fig. 2, the CCDF specified in 191.13(a) can be formally defined by an integral of f over Sst. In practice, this CCDF is never obtained by direct evaluation of an integral due to the complexity of f
J.C. Helton / Computer Physics Communications 117 (1999) 156 180
Table 1 Summary of computer models used in the 1996 WIPP PAI 15, Table I I
161
BRAGFLO
BRAGFLO_DBR
CUTTINGS_S
GRASP-INV
NUTS
PANEL
SECOFL2D
SECOTP2D
SANTOS
Calculates multiphase flow of gas and brine through a porous, heterogeneous reservoir. Uses finite difference proce- dures to solve system of nonlinear partial differential equations that describes the mass conservation of gas and brine along with appropriate constraint equations, initial conditions and boundary conditions. Additi~mal information: 135 I: [14, Section 4.2]. Special configuration of BRAGFLO model used in calculation of dissolved radionuclide releases to the surface (i.e., direct brine releases) at the time of a drilling intrusion. Uses initial value conditions obtained from calculations performed with BRAGFLO. Additional information: 1361; 114, Section 4.71. Calculates the quantity of radioactive material brought to the surface in cuttings and cavings and also in spallings generated by an exploratory borehole that penetrates a waste panel, where cuttings designates material removed by the drillbit, cavings designates material eroded into the borehole due to shear stresses resulting fi'om the circular flow of the drilling fluid (i.e., mud), and spallings designates material carried to the borehole at the tilne of an intrusion due to the flow of gas fiom the repository to the borehole. Spallings calculation uses initial value conditions obtained from calculations performed with BRAGFLO. Outside reviewers expressed concern over the spallings model incorporated into CUTTINGS_S but additional study indicated that the model was unlikely to have overestimated the size of the spallings release 137 ]. Additional information: 138,391 ; [ 14, Sections 4.5, 4.6 ]. Generates transmissivity fields (estimates of transmissivity values) conditioned on measured transmissivity values and calibrated to steady-state and transient pressure data at well locations using an adjoinl sensitivity and pilot-point technique. Additional informafion: 140,411. Solves system of partial differential equations for radionuclide transport in vicinity of repository. Uses brine volumes and flows calculated by BRAGFLO as input, Additional information: [42l: [14, Section 4.3I. Calcnlates rate of discharge and cumulative discharge of radionuclides from a waste panel through an intruding borehole. Discharge is a function of fluid flow rate, elemental solubility and radionuclidc inventory. Uses brine volumes and flows calculated by BRAGFLO as input. Based on solution of system of linear ordinary differential equations. Additional information: [ 421 ; [ 14, Section 4.4 I. Calculates single-phase Darcy flow for groundwater flow in two dimensions. The formulation is based on a single partial differential equation for hydraulic head using fully implicit time differencing. Uses transtnissivity fields generated by GRASP-INV. Additional information: 143,44l; [14, Section 4.8I. Simulates transport of radionuclides in fractured porous media. Solves two partial differential equations: one provides two-dimensional representation for convective and diffusive radionuclide transport in fracture,; and the other provides one-dimensional representation for diffusion of radionuclides into rock matrix surrounding the fi+actures. Uses flow fields calculated by SECOFL2D. Additional information: 143,44I; 114, Section 4.91. Solves quasistatic, large deformation, inelastic response of two-dimensional solids with finite element techniques. Used to determine porosity of waste as a function of time and cumulative gas generation, which is an input to calculations performed with BRAGFLO. Additional inlormation: [35,45+46[; ]14, Section 4.2.3 I.
and S,t. Rather, an approximation procedure based on importance sampling I47] or Monte Carlo (random) sampling [48] is used. The 1996 WIPP PA uses a Monte Carlo procedure. Specifically, elements Xst,i,i =- 1 ,2 . . . . . nS, are randomly sampled from Sst in consistency with the definition of (S~t, S~t, p,t), where nS is the sample size. Then, the integral in Fig. 2, and hence the associated CCDK is approximated by
nS
prob (Re l > R) = / 6R [ f ( x s t ) ] ds t (Xsl ) d ~ t - Z 6R [ f ( x m . i ) ] / i tS ,
s t i=l
(4)
where prob(Rel > R) is the probability that a release greater than R will occur, d~t is the density function associated with (Sst, Sst, pst), and the indicator function ~R is defined by ~R[f(x~t)] = I if f ( x s t ) > R and 0 if f ( x s t ) ~ R. The models in Fig. 3 are too computationally intensive to permit their evaluation for every element 3¢st,i o f Sst in Eq. (4 ) . Due to this constraint, the models in Fig. 3 are evaluated for representative elements o f Sst and then the results o f these evaluations are used in conjunction with various interpolation and algebraic procedures to construct values of f for the large number of xst,i in Eq. (4) [ 14. chs. 9 - 1 3 ] .
The entity EN3 is the outcome of the data development effort for the WIPP and provides a probabilistic
162 J.C. Helwn/Computer Physics Communication,~ 117 (1999) 156-180
characterization of the uncertainty in the parameters that underlie the WIPP PA. When viewed formally, EN3 is defined by a probability space (S~, ,$~, ,p~), with the sample space Ssu given by
S~,~ = {x~, : x,~u is possibly the correct vector of parameter values to use in the WIPP PA}. (5)
The subscript 'su' refers to subjective uncertainty and is used because (Ss,, $, . , P~u) is providing a probabilistic characterization of where the appropriate inputs to use in the WIPP PA are believed to be located.
The vector x~u has the form
x~u : l x l , x 2 . . . . . x , , v ] , (6)
where nV is the number of uncertain variables under consideration. Further, the probability space (S~., $, , , p~,) is obtained by specifying a distribution
D i, j : 1,2 . . . . . nV, (7)
['or each element x i of x~u. The preceding distributions correspond to the distributions in 194.34(b). In concept, some elements of x~u can affect the definition of (S~t, S~t, Pst) (e.g., the rate constant A~/used to define a Poisson process that characterizes the distribution of drilling intrusion times) and other elements can relate to the models in Fig. 3 that determine the function f in Fig. 2 and Eq. (4) (e.g., radionuclide solubilities in Castile brine or alternative models for physical processes). However, all elements of xs, in the 1996 WIPP PA relate to the models in Fig. 3 (Table 2).
The role of uncertainty analysis in PA for the WIPP is to determine the uncertainty in the predictions of the individual models in Fig. 3 and also the uncertainty in the location of the CCDF in Fig. 2 that results fi'om the uncertainty characterized by (Ssu, S~u,p~.). Similarly, the role of sensitivity analysis is to determine the contribution of the individual variables contained in xsu to this uncertainty.
3. Approach to uncertainty and sensitivity analysis
The WIPP PA uses an approach to uncertainty and sensitivity analysis based on Monte Carlo procedures [49]. Specifically, a sample
x~u,k = [.rlk, x2k . . . . . X,,V,k], k= 1,2 . . . . . nK, (8)
is generated from S~u in consistency with the definition of (Ssu,,S~u,psu), where nK is the sample size. Both the models in Fig. 3 and the CCDF in Fig. 2 are evaluated for the individual elements x~,,k of this sample.
The indicated model and CCDF evaluations for the sample elements X~u,k in Eq. (8) create a mapping fi'om analysis input to analysis results. When an appropriate probabilistic procedure has been used to select the sample elements x~u,k, distributions can be constructed for individual model results and also for the CCDF specified in 191.13 (a). Given that (S~u, ,S~,, p~u) is providing a characterization of the subjective, or state of knowledge, uncertainty in the inputs to the analysis, the resultant distributions are providing a characterization for the corresponding uncertainty in the results of the analysis. Further, the pairing
[x~u,k,g(x~u,k)], k = 1,2 . . . . . nK, (9)
of sample elements x~,,,k and analysis results of interest g(x~u,k) creates a mapping from analysis input to analysis results that can be explored with various sensitivity analysis procedures such as examination of scatterplots, stepwise regression analysis and partial correlation analysis [49].
As already indicated, the 1996 WIPP PA uses random sampling from (Sst,,5'sl,pst) in the generation of the CCDF specified in 191.13(a). The use of random sampling is possible because a relatively small number of rnechanistic results is used in the construction of the releases represented by f (x~ , i ) in Eq. (4). However, the
J.C. Helton /Computer Physics Communications 117 (1999) 156-180 163
Table 2
Example elements of xsu in the 1996 WIPP PA (see [ 14, Table 5.11 and 116, Appendix PAR] for a complete listing of the nV = 57 elements of xsu and sources of additional information)
ANHBCVGP
,4 NHPRM
BHPRM
BPCOMP
BPINTPRS
CFRCSP
CMA TRD U
C VEL
HALPOR
HALPRM
WASTWICK
WGRCOR
WMICDFLG
WPRTDIAM
WRBRNSAT
WTA UFA IL
Pointer variable for selection of relative permeability model for use in anhydrite. Used in BRAGFLO. Distribution: Discrete with 60% 0. 40% 1. Value of 0 implies Brooks-Corey model; value of 1 implies van Genuchten-Parker model.
Logarithm of anhydrite permeability (m2). Used in BRAGFLO. Distribution: Student's I distribution with 5 degrees of freedom. Range: -21 .0 to -17.1 (i.e., permeability range is 1 x 10 -2t to I x 10 -171 m2). Mean, Median:
-18.9 , -18.9. Correlation : -0 .99 rank correlation with ANHCOMP (Bulk compressibility of anhydrite, Pa I).
Logarithm of borehole permeability (m2). Used in BRAGFLO. Distribution: Uniform. Range: - 1 4 to - I t (i.e.. permeability range is 1 x 10 -14 to I x 10 - t l m2). Mean, median: -12.5, -12.5.
Logarithm of bulk compressibility of brine pocket (Pa - j ). Used in BRAGFLO. Distribution: Triangular. Range: -11 .3 to -8 .00 (i.e., bulk compressibility range is I x 10 - j l ' 3 to I x 10 -~ Pa -1 ) . Mean, mode: -9.80. -10.0.
Correlation: -0 .75 rank correlation with BPPRM (Logarithm of brine pocket permeability, m 2).
Initial pressure in brine pocket (Pal. Used in BRAGFLO. Distribution: Triangular. Range: I.I1 x 107 to 1.70x 107 Pa. Mean, mode: 1.36 x 107 Pa, 1.27 x 107 Pa.
Culebra fracture spacing (m). Used in SECOTP2D. Equal to half the distance between fractures. Distribution: Uniform. Range: 0.05 to 0.5 m. Mean, median: 0.275 m, 0.275 m.
Culebra matrix retardation for uranium (dimensionless). Defined as function of other uncertain variables. Used in SECOTP2D. Not a sampled variable.
Norm of Culebra fluid velocity vector calculated by SECOFL2D (m/s) . Defined as function of other nncerlain variables. Representative of fluid velocity used in SECOTP2D. Not a sampled variable.
Initial value for halite porosity (dimensionless). Used in BRAGFLO. Distribution: Piecewise uniform. Range: 1.0 x 10 -3 to 3 × 10 -2 . Mean, median: 1.28 x 10 -2, 1.00 x 10 -2 .
Logarithm of halite permeability (m2). Used in BRAGFLO. Distribution: Uniform. Range: - 2 4 to -21 (i.e.,
permeability range is I x 10 -24 to 1 x 10 -2] m2). Mean, median: -22.5, -22.5. Correlalion: -0 .99 rank correlation with HALCOMP (Bulk compressibility of halite, Pa-1 ).
Increase in brine saturation of waste due to capillary forces (dimensionless). Used in BRAGFLO. Distribution: Uniform. Range: 0 to I. Mean, median: 0.5, 0.5.
Corrosion rate for steel under inundated conditions in the absence of CO2 (m/s) . Used in BRAGFLO. Distribution: Uniform. Range: 0 to 1.58 x 10 - i4 m/s. Mean, median: 7.94 x 10 -15 m/s, 7.94 x 10 I~ m/s.
Pointer variable for microbial degradation of cellulose. Used in BRAGFLO. Distribution: Discrete, with 50% 0,
25% 1, 25% 2. WMICDFLG = 0, l, 2 implies no microbial degradation of cellulose, microbial degradation of only cellulose, microbial degradation of cellulose, plastic and rubber.
Waste particle diameter (m). Used in CUTTINGS_S. Distribution: Loguniform. Range: 4.0 x 10 -5 to 2.0 x 10-I m. Mean, median: 2.35 × 10 -2 m, 2.80 x 10 -2 m.
Residual brine saturation in waste (dimensionless). Used in BRAGFLO and BRAGFLO_DBR. Distribution: Uniform. Range: 0 to 0.552. Mean, median: 0.276, 0.276.
Shear strength of waste (Pal. Used in CUTTINGS_S. Distribution: Uniform. Range: 0.05 to 10 Pa. Mean. median: 5.03 Pa, 5.03 Pa.
sample elements x~u,k in Eq. (8) are used directly as input to these mechanistic models; further, a complete CCDF construction is carried out for each Xsu,k. Thus, it is important to use an efficient sampling technique in the generation of the x.~u,k. As a result, Latin hypercube sampling [50] was selected for use because of its efficient stratification properties and the observed stability of results obtained with relatively small sample sizes in past analyses that involved a separation of stochastic and subjective uncertainty [51,52]. Further, because Latin hypercube sampling is a probabilitically-based sampling technique, direct estimates of distribution functions are obtained. Specifically, each sample element can be given a weight equal to the reciprocal of the sample size, l/nK, for use in the estimation of distribution functions [50]; in addition, procedures exist to determine the effects of changed distribution assumptions without rerunning the models under consideration [53]. Various theoretical results exist that indicate that Latin hypercube sampling is superior to simple random sampling in numerical integration problems [50,54,55]. However, there is no way to assess the relative performance of the two sampling methods for a specific, complex analysis problem except by repeatedly performing the analysis
164 J.C. He#on/Computer Physics Communications 117 (1999) 156 180
4.0
3.6
3.2
2.8 ~ J
Latin Hypercube Sample I ' I I ' I '
2.4 m -5 '1" ~' 20 ~ '~
- r _
I - -
1.8- • G 1.2 ~1,
0.8 ~-
~D 0.4
o 0'-1.0 -0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0
U: Normal
Fig. 4. Example o f an LHS o f size nLHS = 10 frorn variables U and V with U normal on J - I,I ] (mean = 0, 0.01 quantile = - I . 0.99 quantile = 1 ) and V triangular on 10,41 (mode= l ) .
with both methods and then comparing the stability of the results [51,52,56]. Latin hypercube sampling operates in the following manner to generate a sample of size nLHS from nV
variables. The range of each variable (i.e., the xj in Eq. (6)) is divided into nLHS intervals of equal probability and one value is selected at random from each interval. The nLHS values thus obtained for xl are paired at random with the nLHS values obtained for x2. These nLHS pairs are combined in a random manner with the nLHS values of x3 to form nLHS triples. This process is continued until a set of nLHS nV-tuples is formed. These nV-tuples are of the form
xi= [xli,x2, . . . . . x,,v,i], i= 1 . . . . . nLHS, (10)
and constitute the Latin hypercube sample (LHS). The individual xj must be independent for the preceding construction procedure to work; however, a method for generating LHSs from correlated variables has been developed by Iman and Conover [57,58].
An LHS from two variables is illustrated in Fig. 4. As examination of this figure shows, Latin hypercube sampling results in a dense stratification across the range of each variable (i.e., there is one value from each equal probability interval), which is consistent with the requirement in 194.34(c) for sampling across the entire range of each variable.
Given that Latin hypercube sampling is to be used, the confidence intervals required in 194.34(f) can be obtained with a replicated sampling technique proposed by R.L. Iman ([59]; [15, Section 2.4]). In this technique, the LHS_indicated in Eq. (10) is repeatedly generated with different random seeds. These samples lead to a sequence Pr(R) , r = I, 2 . . . . . nR, of estimated mean exceedence probabilities, where Pr(R) defines the mean CCDF obtained for sample r (i.e., Pr(R) is the mean probability that a normalized release of size R will be exceeded, with this mean taken over the nLHS exceedance probabilities for R obtained from the rth replicated sample) and nR is the number of independent LHSs generated with different random seeds. Then,
nR
-fi( R) = Z - f i r ( g) / ng r=l
( l l )
and
Z C. Helton / Computer Physics Communications 117 (1999) 156-180
Table 3 Example rank correlations in the LHS that constitutes replicate RI
165
WGRCOR WMICDFLG HALCOMP HALPRM ANHCOMP ANHPRM BPCOMP BPPRM
WGRCOR 1.0000 WMICDFLG 0.0198 1.0000 HALCOMP 0.0011 0.0235 1.0000 HALPRM -0.0068 -0.0212 -0.9879 ANHCOMP 0.0080 0,0336 -0.0123 ANHPRM 0.0049 -0.0183 0.0037 BPCOMP 0,0242 0.1071 -0.012 I BPPRM --0.0514 -0,0342 0.0035
1.0000 0.0025 1.0000 0.0113 -0.9827 1.0000 0.0057 -0.0184 0.0078 1,0000 0.0097 0.0283 -0.0202 -0,7401 1.0000
nR
P ( R ) ]2/nR(nR -- 1)
I/~,
(12)
provide an additional estimate of the mean CCDF and an estimate of the standard error associated with the mean exceedance probabilities. The t-distribution with nR - 1 degrees of freedom can be used to place confidence intervals around the mean exceedance probabilities for individual R values (i.e., around P r ( R ) ) . Specifically, the I - cr confidence interval is given by Pr(R) ± tI_~/2SE(R), where tl ..... /2 is the 1 - or/2 quantile of the t-distribution with nR - 1 degrees of freedom (e.g., t1_~,/2 = 4.303 for of = 0.05 and nR = 3).
The requirement in 194.34(d) implies the use of a sample with at least 298 elements (i.e., 1 - 0.99" > 0.95 yields n = 298). Thus, to satisfy both 194.34(f) and 194.34(d), the 1996 WIPP PA used nR = 3 replicated LHSs of size nLHS = 100 each, with these replicated samples denoted by R 1, R2 and R3. This produced a total of 300 observations, which is approximately the same as the sample size of 298 indicated above. Each sample was generated with the restricted pairing technique developed by Iman and Conover [57,58] to induce specitied rank correlations between correlated variables and also to assure that uncorrelated variables had correlations close to zero (Table 3).
Once the indicated LHSs were generated, calculations were performed with the models in Fig. 2 for the individual sample elements (Table 4). The guiding principle in the selection of the calculations to be performed was to avoid the unnecessary proliferation of computationally demanding calculations by identifying situations where (1) a single computationally demanding calculation could be used to supply input to several less demanding calculations, (2) mathematical properties of the models could be used to extend the results of a single calculation to many different situations, or (3) a relatively inexpensive screening calculation could be used to determine if a more detailed, and hence more expensive, calculation was needed ([15, Section 2.4]: [ 14, chs. 9 - 1 2 ] ) .
4. Uncertainty and sensitivity analysis for individual models
As indicated in Table 4, many distinct cases exist for which uncertainty and sensitivity analysis results can be obtained for individual models. For example, 6 cases exist for BRAGFLO, with calculations having been performed with the same 300 LHS elements for each case. In each of these cases, the uncertainty is due solely to subjective uncertainty as characterized by (S~u,,S~u,P~u). In particular, the effects of stochastic uncertainty as characterized by (S~t, Sst, P~t) do not enter into the analysis until the results of individual models are used in the construction of CCDFs. Example uncertainty and sensitivity analysis results for individual models follow; more detailed results are available in Helton et al. [14].
166 J.C. He#on / Computer Physics Communications 117 (1999) 156-180
Table 4 Mechanistic calculations performed as part of the uncertainty and sensitivity analyses in the 1996 WIPP PA
BRAGFLO
CUTTINGS_S
B RAGFLO_DBR NUTS
PANEL
SECOFL2D
SECOTP2D
hzdividual Calculations (6 cases): E0 (i.e., undisturbed conditions); El at 350, 1000 yr (i.e., drilling intrusion through repository that penetrates pressurized brine in the Castile Fin); E2 at 350, 1000 yr (i.e., drilling intrusion through repository that does not penetrate pressurized brine in the Castile Ftn); E2EI with E2 intrusion at 800 yr and El intrusion at 2000 yr. Total calculations: 6 nR nLHS = 6 × 3 × 100 = 1800. Individual Cahulations (52 cases): Intrusion into lower waste panel in previously unintruded (i.e., E0 conditions) repository at 100, 350, 1000, 3000, 5000, 10000 yr; Intrusion into upper waste panel in previously unintruded repository at 100, 350. 1000+ 3000, 5000+ 10000 yr; Initial El intrusion at 350 yr followed by a second intrusion into the same waste panel at 550, 750, 2000. 4000 or 10000 yr; Initial El intrusion at 350 yr followed by a second intrusion into a different waste panel at 550, 750, 2000, 4000 or 10000 yr; Initial El intrusion at 1000 yr followed by a second intrusion into the same waste panel at 1200, 1400, 3000, 5000 or 10000 yr; Initial El intrusion at 1000 yr followed by a second intrusion into a different waste panel at 1200, 1400, 3000, 5000 or 10000 yr; same 23 cases for initial E2 intrusions as for initial El intrusions. Total calculations: 52 nR nLHS = 52 × 3 × 100 = 15 600. Same computational cases as for CUTTINGS_S. Individual Calculations ( 15 cases): E0:E1 at 100+ 350, 1000, 3000, 5000, 7000, 9000 yr; E2 at 100, 350, 1000, 3000, 5000, 7000, 9000 yr. Screening ~alculations: 5 nR nLHS = [500. Total NUTS calculations: 594. Note: Screening calculations were initially performed for each LHS element (i.e., E0, El at 350 and 1000 yr, E2 at 350 and 1000 yr, which produces the multiplier of 5 in the calculation of the number of screening calculations) to determine if the potential for a radionuclide release existed, with a full NUTS calculation only being performed when such a potential existed. hzdividual Calculations (7 cases): E2EI at [00+ 350, 1000, 2000, 4000, 6000, 9000 yr. Total calculations: 7 nR nLHS = 7 × 3 × 100 = 2100. Note: Additional PANEL calculations were also performed at 100, 125, 175, 350, 1000, 3000, 5000, 7500 and 10000 yr for Salado-dominated brines and also for Castile-dominated brines Io determine dissolved radionuclide concentrations for use in the determination of direct brine releases. hldividual Cah:ulations (2 cases): Partially mined conditions in vicinity of repository; Fully mined conditions in vicinity of repository. Total calculations: 2 nR nLHS = 2 × 3 × 100 = 600. hzdividual Calculations (2 cases): Partially mined conditions in vicinity of repository; Fully mined conditions in vicinity of repository. Total calculations: 2 nR nLHS = 2 × 3 × 100 = 600. Note: Each calculation is for four radionuclides: Am-241, Pu-239, Th-230, U-234+ Further, calculations are done for unit releases at time 0 yr, which can then be used to construct transport results for the Culebra for arbitrary time-dependent release rates into the Culebra [14, Section 12.2]).
The B R A G F L O mode l (Fig . 3, Table l ) is used to r ep resen t t w o - p h a s e (i .e. , gas and b r i n e ) flow in the
v ic in i ty o f the reposi tory . An i m p o r t a n t resul t ca lcu la ted by B R A G F L O is the p ressure in a was te pane l as a
func t ion o f t ime (Fig . 5 a ) , wi th this p ressure in f luenc ing spa l l ings and d i rec t b r ine re leases , w h i c h are re leases
d i rec t ly to the sur face at the t ime o f a dr i l l ing in t rus ion, and also r ad ionuc l ide t r anspor t away f rom the r epos i to ry
in a n h y d r i t e m a r k e r beds. T he spread in the cu rves in Fig. 5a is due to the effects o f i m p r e c i s e l y - k n o w n inpu ts to
the ana lys i s (i .e. , sub jec t ive u n c e r t a i n t y ) . One way to ident i fy the var iab les tha t are d o m i n a t i n g the unce r t a in ty
is to ca lcu la te part ial r ank co r re la t ion coeff ic ients ( P R C C s ) b e t w e e n p ressures at ind iv idua l t imes and the
var iab les in the L H S (Fig . 5 b ) , wi th W M I C D F L G ( p o i n t e r var iab le for mic rob ia l deg rada t i on o f c e l l u lo se ) ,
W G R C O R ( c o r r o s i o n rate for steel unde r i nunda t ed cond i t ions in the absence o f C O 2 ) , W A S T W I C K ( i n c r e a s e
in br ine sa tu ra t ion o f was te due to capi l la ry forces) and H A L P O R ( in i t ia l va lue for hal i te po ros i t y ) be ing
ident i f ied as the d o m i n a n t va r iab les ( see Table 2 for var iab le de sc r ip t i ons ) . The pos i t ive ef fec ts ind ica ted
for these va r i ab les resu l t becaus e inc reas ing W M I C D F L G increases gas gene ra t ion by mic rob ia l p rocesses ,
i nc reas ing W G R C O R inc reases the rate at w h i c h gas is gene ra t ed by the co r ros ion o f steel, and inc reas ing
W A S T W I C K and H A L P O R inc reases the a m o u n t of b r ine ava i lab le for c o n s u m p t i o n in the co r ros ion process .
The ana lys i s was p e r f o r m e d with 3 rep l ica ted LHSs (i.e. , R l , R2, R3 ) to p rov ide a basis for tes t ing the
s tabi l i ty o f resul t s ob t a ined by s a m p l i n g f rom (Ssu, $su, P s , ) . For t i m e - d e p e n d e n t resul t s such as the p ressu re
cu rves in Fig. 5a, such tests can be based on the s imi lar i ty o f m e a n and pe rcen t i l e cu rves ob t a ined f rom the
ind iv idua l rep l ica tes and cons t ruc t ed in the fo l lowing manner . A t each po in t on the absc i s sa in Fig. 5a, a ver t ical
l ine is d r a w n t h r o u g h the n L H S = 100 curves above this point , The po in t s at w h i c h the ind ica ted ver t ical l ine
J.C. Helton / Computer Physics Comrnunications 117 (1999) 156-180 167
[l. %
t r
¢/)
#_
o
o~ (3-
BRAGFLO ( E0, Replicate R1) P r e s s u r e in L o w e r Waste Panel (WAS_PRES)
1.5
1.2
I
0.9
0.6
0 . 3
.£
8 8
o 0
n -
g_
BRAGFLO ( E0, Replicates R1, R2, R3) P r e s s u r e in Lower Waste Panel (WAS_PRES)
l OOl . . : :
0.75 I...F;/. : . : : I I . - - . - - . . . . . . . . . . . . . . . . . . . . :__
o.oo H >-: . . . . . . . . . . . . . . .
0.251-1 ,," ""------ . . . . 0.00
43.25
-0 .50
-0.75 Frame 5b
0.0 -1.00 i I
0.0 2.0 4.0 6.0 8.0 10.0 0.0 7.5 9.0
Time (103 yr)
Dependent Variable WAS_PRES
WMICDFLG . . . . . WGRCOR
WASTVMICK HALPOR
I I I i
1.5 3.0 4.5 6.0
Time (10:3 yr)
Fig. 5. Uncertainty and sensitivity analysis results for pressure (Pa) in waste panel under undisturbed (i.e., E0) conditions: (a) time-dependent pressures for 100 LHS elements in replicate RI, and (b) PRCCs obtained from analysis of all 300 LHS elements associated with replicates RI, R2 and R3.
BRAGFLO (E0, R1, R2, R3) VoI-Averaged Pressure Lower Panel (WAS_PRES)
1 . 5 ~
12r- , ¢ -
03 I ,~ 0.9
I:i[~' ..:.:-~'-'o" <> R2
Q. ~ - - - - - - 901h Pecer ntile 0.0
0.0 2.0 4.0 6.0 8.0 10.0 Time (10 3 yr)
Fig. 6. Mean and percentile curves for three replicated LHSs for pressure in lower waste panel under undisturbed conditions.
crosses the curves yields nLHS pressures from which the mean and various percentiles (e.g., 10%, 50%, 90%) can be determined. The desired mean and percentile curves then result from connecting the mean and percentile values obtained for individual times. In concept, the determination of mean and percentile values can be viewed as an integration problem involving (S.~u, S,u, p.~u) [60]. The resultant mean and percentile curves were quite stable for both the pressure curves in Fig. 5a (Fig. 6) and other analysis outcomes. If desired, the procedures indicated in Eqs. ( l 1) and (12) can be used to provide a formal characterization of the uncertainty associated with the estimates of the individual curves in Fig. 6.
The results in Fig. 5 are for undisturbed conditions. An important event in the 1996 WIPP PA is the occurrence of a drilling intrusion into the repository, which causes a major alteration of the pressure conditions (Fig. 7a). Although PRCCs and also stepwise regression analysis were successful in identifying the variables giving rise to the uncertainty in Fig. 5a, they performed very poorly for the pressure results in Fig. 7a. When this occurs, the examination of scatterplots is often an effective way to identify influential variables, with a strong but highly nonlinear relationship being identified between pressure after a drilling intrusion and BHPRM (logarithm of
168 J.C. Helton/Conqmter Physics Communications 117 (1999) 156-180
1.5
%
~ 1..~ or
COl < 0.9
~_ o.e
o ~ 0.3
n
BRAGFLO ( E2 at 1000 yr, Replicate R1) Pressure in Lower Waste Panel (WAS_PRES)
0.0 0 0 2.0 4.0 6.0 8.0 10.0
Time (103 yr)
%- 13.
9.0 ~9 Cl.I
7.5 CO
6.o
~ 4.5
_9 3.o
~ 1.5
10-14
BRAGFLO ( E2 at 1000 yr, Replicates R1, R2, R3) Pressure in Lower Waste Panel (WAS_PRES)
. . . . . . . . u . . . . . . . . n . . . . . . . .
Frame 7b Time: 10000 yr w_•m
,~. ,,~ ,~4~1~ ~ , . . - : . . - . . - .
m I ! ~ •
10 -13 10 12 10-11
Borehole Permeability (m2): 10 x, x = BHPRM
Fig. 7. Uncertainty and sensitivity analysis results for pressure (Pa) in waste panel after a drilling intrusion at 1000 yr that does not penetrate pressurized brine in the Castile Formation (i.e., an E2 intrusion): (a) time-dependent pressures for 100 LHS elements in replicate RI, and (b) scatterplot for pressure at 10000 yr versus BHPRM for 300 LHS elements in replicates RI, R2 and R3.
borehole permeability) (Fig. 7b). The complex pattern involving pressure and B H P R M results from the role that B H P R M plays in influencing two-phase flow in the borehole, with gas flowing up the borehole and brine typically flowing down the borehole.
The NUTS and PANEL models (Fig. 3, Table 1) are used to represent long-term radionuclide transport away from the repository due to flowing groundwater. The NUTS model was used for undisturbed (i.e., E0) conditions, single drilling intrusions that penetrate pressurized brine in the Castile Formation (i.e., E1 intrusions), and single drilling intrusions that do not penetrate pressurized brine in the Castile Formation (i.e., E2 intrusions). The PANEL model was used for the penetration of a waste panel by two or more drilling intrusions, of which at least one penetrates pressurized brine in the Castile Formation (i.e., an E2E1 intrusion). Most sample elements resulted in little or no radionuclide release being predicted by NUTS and PANEL due to failure of the repository to fill with brine, with the greatest number of releases occurring for the E2E1 intrusions. A single example of uncertainty and sensitivity analysis results for long-term radionuclide release from the repository is given in Fig. 8. The box plots in Fig. 8a provide a compact way to display the uncertainty in a number of variables in a single plot frame. The variable B H P R M dominates radionuclide release, with no releases typically occurring for small values of B H P R M due to a failure of the intruded waste panel to fill with brine (Fig. 8b).
The SECOFL2D and SECOTP2D models (Fig. 3, Table 1) are used to represent brine flow and radionuclide transport, respectively, in the Culebra Dolomite. In the 1996 WIPP PA, only one sample element had the potential fk)r radionuclide transport from the repository to the accessible environment. However, no releases to the Culebra occurred for this sample element. Thus, no release to the accessible environment due to transport through the Culebra occurred in the calculations perlbrmed to support the 1996 WIPP PA. To provide perspective on the processes affecting radionuclide transport in the Culebra, an additional set of calculations was performed for replicate RI for a release of I kg of U-234 at the repository at time 0 yr and the transport of this release in the near vicinity of the repository (i.e., within a few 10's of meters) (Fig. 9), with U-234 selected for consideration because it is the least retarded (i.e., has the lowest ka values) of the radionuclides whose transport in the Culebra was modeled in the 1996 WIPP PA. The release was rapidly attenuated (Fig. 9a), with the dominant variables being C M A T R D U (Culebra matrix retardation for uranium), CVEL (norm of fluid velocity vector calculated by SECOFL2D) and CFRCSP (Culebra fracture spacing) (Fig. 9b). Specifically, C M A T R D U
ZC. He~ton~Computer Physics Communications 117 (1999) 156-180 169
PANEL (E2E1 with E1 at 1000 yr, R1, R2, R3)
Time: 10000 yr AM241 I i ~=,,,,,o=,-.,= ~ ×
PU238 I ~ I •
PU239 I ; '. " , ~ × ×
U234 i ~ . . . .
TH230 I ~ ~ Frame 8a
TOTAL t I --~,.,.,eo=~,,=~ ×
. . . . . ~ . . . . . ~ . . . . . . a . . . . . J . . . . . ~ . , _ , = a . . . . . ~ . . . . . . a . . . . ~ . , _ . , a
10 -9 10 -7 10-5 10 -3 10-1 101
Normalized Release (EPA units)
n
z=
E
PANEL (E2E1 with E1 at 1000 yr, R1, R2, R3) 102
Time: lO000yr • • ..1 , . . !
10 0 • =•. == . • o'k~ • • •• ~ = ===if= •
• % • = • = t • = = ~ . = ~ 10 -2 11, . •,T, • • == • •=• • • • • • ' • "!]
8 • • • • ! • e l o = • •
• ' ~ • 0 • • ' t = d ~ . = - o = ~ P • j
i i o ~ o - - I l l i i
10 -4 " • • " i i i ~ o
10 -6 #
• Frame 8b
10-10_148 10 -13 10 -12 10 -11
Borehole Permeability (m2): 10 x, x = BHPRM
Min {1.5x Box, Largest Obs}
75 lh Extreme Obs Percentile M e d i a n M e a n Percentile
Fig. 8. Uncertainty and sensitivity analysis resuks for an EZEI intrusion with the El ~ntrusion occurring at [000 yr: (a) box plots {br cumulative normalized release of individual radionuclides over 10000 yr, and (b) scanerplot for cumulative normalized release of Am-241 over 10000 yr versus BHPRM.
;g C ~
y_
' t : O
E= L)
SECOTP2D, U-234, Replicate R1 Cumulative Transport to 10 m (MT4U234C)
1.0 ' ' ' ' I ' ' ' ' l ' ' ' ' I ' ' ' '
Frame 9a 0.8
0.6
0.4 I
1.00
0.75
0.50 (.9 .~ 0.25
~- 0.00 0 0 -~ -0.25 cE
-0.50 I E
n ~ -0.75
SECOTP2D, U-234, Replicate R1 Cumulative Transport to 10 m (MT4U234C)
. n n i i
o . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Frame 9b
• . o w = . ° ° o = ° o . . . . . . . . . - - ° ° . . . . . . . . . . . . . . . . . . . . . . . . . . . .
0.2
i 0.0 -1.00 I t I i I i I
0.0 2.0 4.0 6.0 8.0 10.0 0.0 1.5 3.0 4.5 6.0 7.5 9.0
D e p e n d e n t V a r i a b l e MT4U234C
CMATRDU " " " CVEL
RCSP
Time (103 yr) Time (103 yr)
Fig. 9. Uncertainty and sensitivity anafys~s results for transport of a f kg release of U-234 at tile repository across a boundary 10 m from the release point: (a) cumulative releases for individual sample elements, and (b) PRCCs.
characterizes the effects of sorption in the rock matrix; CVEL characterizes fluid velocity in fractures in the vicinity of the release point for the U-234, and CFRCSP affects the amount of diffusion that takes place from fractures into the surrounding rock matrix. A transport distance of only 10 m is used in Fig. 9 because the amount of cumulative transport decreases rapidly as distance increases; indeed, most of the 100 cumulative transport curves in Fig. 9a lie close to or on the abscissa.
170 J.C, Helton/Computer Physics Communican'ons 117 (1999) 156-180
5. Uncertainty and sensitivity analysis for CCDFs
Although a number of different release pathways to the accessible environment were considered in the 1996 WIPP PA, only the direct release pathways (i.e., cuttings and cavings, spallings, direct brine release) produced nonzero releases. As a result, the function f in Eq. (4) has the form
f(Xst) = f c ( x s t ) _a fsp[Xsl, f rmB(Xst) ] _L fDBR{Xsb fsP[ Xst, fB(Xst) ], fB(Xst) }, (13)
where (i) fc(x~t) ~ cuttings and cavings release to accessible environment lbr x~t calculated with CUT- TINGS_S, (ii) fB(X~t) ~ results calculated for x~t with BRAGFLO (in practice, Ji~(x~t) is a vector containing a large amount of information), (iii) fsP[Xst, fB(Xst)] ,-v spallings release to accessible environment for x~t calculated with the spallings model contained in CUTTINGS_S (this calculation requires BRAGFLO results (i.e., fB(X~t)) as input), and (iv) fDBR{Xst, fsP[X.~t, fB(x~t)], fB(x~t)} ~ direct brine release to accessible environment for x~t calculated with a modified version of BRAGFLO designated BRAGFLO_DBR (this calcu- lation requires spallings results obtained from CUTTINGS_S (i.e., fsp[X~t,f~(xs~)]) and BRAGFLO results (i.e., fu(x~t)) as input).
As indicated in Eq. (4), random sampling of Xst from (Sst,$st,Pst) can be used to construct the CCDF in Fig. 2. Further, f c , fsP and fDBR can also be used to obtain the CCDFs for cuttings and cavings, spallings and direct brine releases, respectively. The evaluation of f c , f s p and fDBR, and hence the evaluation of f , also depends on the value of X~u. As a result, Eq. (4) actually has the form
nS f prob( Rel > R]x~u) = / 8e [ f(x~t, xsu) ] d~t(x~t[x~,~) d~t - ~ 6R[f(x~tj, X~, ) ] / n S , (14)
, 7 S~E i= I
although there is no dependence of dst on Xsu in the 1996 WIPP PA (i.e., dst(xst[Xsu ) = ds t (Xs t ) ) . However, such dependence is possible and, indeed, was implemented in certain verification analyses associated with the 1996 WIPP PA carried out by SNL for the EPA [61 ].
The uncertainty associated with X~u, and characterized by (S~, Ss~,P~u), leads to a distribution of CCDFs for f c , fsp, fDBR and f . These distributions can be approximated by constructing CCDFs for each element x~u.t in the three replicated LHSs, with the outcomes of this construction shown in Fig. 10 for replicate RI. Each CCDF in Fig. 10 is constructed from model results (i.e., evaluations of f c , fsp, fDBR) associated with one LHS element in R1. The details of the procedures used to obtain values of f(Xsta, Xsu,k) from the results indicated in Table 4 are described in [ 14].
The distributions of CCDFs in Fig. 10 exhibit the effects of subjective uncertainty. As such, these distributions provide a characterization of where the CCDF for each release mode is believed to be located given the information available for use in the 1996 WIPP PA. The locations of the CCDFs for the spallings release (Fig. 10b) and direct brine release (Fig. 10c) are more uncertain than the location of the CCDF for the cuttings release (Fig. 10a). However, as the cuttings release tends to be larger than the spallings and direcl brine releases, the uncertainty in the total release (Fig. 10d) is dominated by the uncertainty in the cuttings release (i.e., the distribution of CCDFs for total release is similar to, but not identical to, the distribution of CCDFs for cuttings release).
As discussed in conjunction with Fig. 5, the distributions of CCDFs in Fig. 10 can be summarized with mean and percentile curves. As an example, the mean and percentile curves for the distribution of total release CCDFs (Fig. [0d) appears in Fig. [l . The proximity of the distribution of the CCDFs to the boundary line associated with 191.13(a) provides an indication of the confidence that the regulation will indeed be met. The 90th percentile curve, or some other percentile curve, helps provide a numerical indication of where the distribution of CCDFs is located relative to the boundary line.
The mean and percentile curves associated with the three replicated samples RI, R2 and R3 can be used to obtain an indication of how stable the estimates of the CCDFs in Fig. 10 are. In particular, the three estimates
J.C. Helton / Computer Physics Communications 117 (1999) 156-180 171
101
10 0
t.r A
~ 10-1
> ..~ 10 -2
"F,
10 -3 n
10-4
C u t t i n g s N o r m a l i z e d R e l e a s e : R e p l i c a t e R1
1 0 0 O b s e r v a t i o n s , 1 0 0 0 0 F u t u r e s / O b s e r v a t i o n . . . . . . . . i . . . . . . . . i . . . . . . . . i . . . . . . . . i . . . . . . . . i . . . . . . . . i . . . . . . .
i
E P A L i m i t . . . . . i
I
L . . . .
! L . . . .
F r a m e 10a
10 -5 . . . . . . . . ' . . . . . . . . ' . . . . . . . . ' . . . . . . . . ' . . . . . . . . ' . . . . . . . . ~ . . . . .
10 -5 10 -4 10 -3 10 -2 10 -1 100 10 ~ 102
N o r m a l i z e d R e l e a s e ( E P A un i t s ) , R
101 . . . . . . . . r
100
rr"
A -1 10
> ~ . 10-2
~ lO-a
10 -4
S p a l l i n g s N o r m a l i z e d R e l e a s e : R e p l i c a t e R1
100 O b s e r v a t i o n s , 1 0 0 0 0 F u t u r e s / O b s e r v a t i o n , r . . . . . . . . i . . . . . . . . i . . . . . . . . i . . . . . . . . = . . . . . . . . i . . . . . . ,
. . . . . E P A L i m i t
(18 o b s e r v a t i o n s o f f s c a l e )
. . . . . i
!
F r a m e 1 0 b 10 -5 . . . . . . . . i . . . . . . . . i . . . . . . . . i . . . . . . . . i . . . . . . . . i . . . . . . . . L . . . . .
10-5 10-4 10-3 10 -2 10-1 10 o 101 102
N o r m a l i z e d R e l e a s e ( E P A un i t s ) , R
D i r e c t B r i n e R e l e a s e : R e p l i c a t e R1
100 O b s e r v a t i o n s , 1 0 0 0 0 F u t u r e s O b s e r v a t i o n 101 . . . . . . . . , . . . . . . . . , . . . . . . . . , . . . . . . . . , . . . . . . . . i . . . . . . . . , . . . . . . .
. . . . . E P A L i m i t i i
i (51 o b s e r v a t i o n s o f f s c a l e ) I
I I I
L . . . .
F r a m e 10c
10 -5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 -5 10-4 10 --3 10 -2 10 -1 10 0 101 10 2
N o r m a l i z e d R e l e a s e ( E P A un i t s ) , R
100
r r
^ 10-1
~ 10-2
~ 1 0 - 3
10-4
101
100
t,r A ~ 10-1
> . ~ 10 -2
..Q
~ 1 0 - 3 (3_
10 .4
t0-%_;
T o t a l N o r m a l i z e d R e l e a s e s : R1
1 0 0 O b s e r v a t i o n s , 1 0 0 0 0 F u t u r e s / O b s e r v a t i o n . . . . . . . . r . . . . . . . . i . . . . . . . . i . . . . . . . . r . . . . . . . . . . . . . . . J ....
. . . . . E P A L i m i t
L . . . .
F r a m e 10d . . . . . . . . i . . . . . . . . i . . . . . . . . i . . . . . . . . i . . . . . . . . i . . . . . . . . i . . . . .
10-4 10-3 10 -2 10-1 100 101 102
N o r m a l i z e d R e l e a s e ( E P A u n i t s ) , R
Fig. 10. Distributions of CCDFs resulting from subjective uncertainty and obtained with replicate RI Jor the following release modes: (a) cuttings and cavings, (b) spallings, (c) direct brine release, and (d) total (i.e., cuttings, spallings, direct brine) [ 15, Figs. 6, 91. The statements in frames (b) and (c) that observations are off-scale means that the largest normalized release for each of these observations is less than 10 -5 EPA units, with the largest release for each of these observations typically being zero.
of these curves for a particular release mode (i.e., one set of estimates for each replicate) can be plotted on the same plot frame. When the corresponding curves fall close together, an indication is given that the sample size in use (i.e., nLHS = 100 in this analysis) is adequate. For the cuttings, spallings and total releases, the individual mean and percentile curves for the three replicates are so similar that they almost fall on top of each other. Slightly more variation across the replicates occurs for the direct brine release (Fig. 12). However, even here the individual curves are actually quite close together, particularly given that approximately half the sample elements in each replicate have no direct releases and thus produce degenerate CCDFs (i.e., CCDFs lbr which there is a probability of 0 of exceeding a release of 0). Thus, the distributions of CCDFs in the 1996 WIPP are adequately estimated for comparison with the boundary line specified in 191.13(a) with an LHS of size 100.
172 J. C. Helton / Computer Physics Communications 117 (1999) 156-180
10 ~
Total Normalized Releases: R1 100 Observations, 10000 Futures/Observation
10 0
r v
^ 10 -1
to -2
o lO-3
l o 4
10-5 10-5
" ° * % ~ N . i
"ili = i i
: l t , . . . . EPALimi t : • , - - -
Mean : i '~ - - - - - 90th Quanti le : i ' ~ -
- - 50th Quanti le . . . . . . Ouao ! . . . . . . . . . . . . \
1 0 - 4 10-3 10 -'2 1 0 - 1 10 0 101 10 2
Normalized Release (EPA units), R
F i g I I
100 r r
A g
N 1 0 - 1 >
~ 10-2 e
a .
m4
10-4
to - ~ 10-5
Direct Brine Normalized Releases: R1, R2, R3 300 Observations, 10000 Futures/Observation
101 . . . . . . . . , . . . . . . . . ~ . . . . . . . . , . . . . . . . . ~ . . . . . . . . . . . . . . . ~ . . . . . . .
. . . . EPA Limit Mean
O'°,ogS"ua nT =='.-,e=~==~--.. (Offscale) 50th Quantile • "~'Q% ~ (Offscale) 10th Quantile ~ . . . . .
,
10-4 10-3 10-2 10-1 10 0 101 10 2
Normalized Release (EPA units), R
Fig. 12,
Fig. I I. Mean and percentile curves for distribution of total release CCDFs in Fig. 10d [15, Fig. 71.
Fig. 12. Outcome of replicated sampling for distribution of CCDFs for normalized release to the accessible environment over 10000 yr due to direct brine release.
Some readers may be surprised by the mean CCDFs in Figs. I I and 12 that cross the 90th percentile curves. Such behavior occurs for highly skewed distributions in which the mean is dominated by large but low-probability values. Such distributions are actually fairly common in the final results of PAs in which many of the input variables are assigned distributions characterizing epistemic uncertainty that span multiple orders of magnitude. Further, the use of sampling-based procedures to estimate mean CCDFs will always produce a point at which the mean CCDF moves above the 90th percentile curve when 90% of the individual CCDFs have dropped to a value of zero on the abscissa. Due to the distortions caused by large but low-probability results, a mean CCDF may provide a rather poor summary of the information contained in a distribution of CCDFs.
Sensitivity analysis can be used to identify the variables contained in x~u that give rise to the distributions of CCDFs in Fig. 10. One way to do this is to reduce each CCDF to an expected value by the calculation
nS
E( flX~u.~.) = , / f (x~ t , X~u,k)dst(Xst]Xsu.k)d~t - Z f(Xst.i, X~u.k)/nS, (15) S< i= I
where E(flx~u,k) denotes the expected value of f given X~u,k. Then, regression analysis can be used to determine how the elements of X.~u affect E(flXsu). The expected values for f c , fsP and fDBR can also be defined and analyzed in a similar manner.
In regression-based sensitivity analyses, variable importance is indicated by the order in which variables enter the regression model, the changes in R 2 values that occur as variables enter the regression model, and the size of the standardized regression coefficients for the variables in the model [49]. Regression analyses were tried with both raw and rank-transformed [62] data, with rank-transformed data producing more informative results (i.e., regression models that identified more of the physically-relevant variables and had higher R2 values than was the case for raw data). Thus, only results obtained with rank-transformed data will be discussed.
For the cuttings and cavings release CCDFs, the expected release was completely dominated by WTAUFAIL (shear strength of waste), with the corresponding rank regression model having an R 2 value of essentially 1.00. More complex patterns occur for the spallings, direct brine and total release CCDFs (Table 5). The spallings release is sensitive to the pressure in the repository. The variables WMICDFLG, HALPOR, WGRCOR, HALPRM and ANHPRM appear in the regression model with positive regression coefficients because increasing
.L C. Helton / Computer Physics Communications l 17 (1999) 156-180 173
Table 5 Stepwise regression analysis with rank-transformed data for expected normalized release associated with individual CCDFs for spallings release, direct brine release, and total release
Step a Spallings release Direct brine release Total release
Variable h SRRC c R 2 ~ Variable SRRC R 2 Variable SRRC R 2
I WMICDFLG 0.72 0.53 WRBRNSAT -0.46 0.21 WMICDFLG 0.60 0.40 2 HALPOR 0.26 0.60 HALPOR 0.37 0.35 WTAUFAIL -0.39 0.55 3 WGRCOR 0.24 0.66 ANHPRM 0.27 0.42 WGRCOR 0.21 0.59 4 BHPRM -0.20 0.70 BPCOMP 0.20 0.46 WPRTDIAM -0.19 0.63 5 WPRTDIAM 0.19 0.73 BPINTPRS 0.14 0.48 HALPOR 0.17 0.65 6 HALPRM 0.13 0.75 HALPRM 0.14 0.50 BHPRM -0.17 0.68 7 WASTWICK 0.1 I 0.76 HALPRM 0.16 0.7 I 8 A NHPRM 0. I 0 0.77 WA STWICK 0.1 I O. 72 9 ANHPRM 0.09 0.73
~ Sleps in slepwise regression analysis. h Variables listed in order of selection in regression analysis. ~ Standardized rank regression coefficients (SRRCs) in final regression model with ANHCOMP and HALCOMP excluded fi'om entry into
lvgression model because of tile -0.99 rank correlation within the variable pairs (ANHPRM, ANHCOMP) and (HALPRM, HALCOMP). ~l Cumulative R 2 value with entry of each variable into regression model.
their values tends to increase pressure in the repository and thus increase the spallings release. In particular, there is a threshold value of 8 x 106 Pa in the spallings model, with no spallings releases occurring when repository pressure is below this value. The variables B H P R M and W P R T D I A M appear in the regression
model with negative coefficients because increasing B H P R M reduces pressure in the repository and increasing W P R T D I A M increases particle size and thus decreases particle mobility. More detail on the physics underlying these and other results is available in [ 14]. Overall, the regression model is reasonably effective in accounting for the observed uncertainty, with an R 2 value of 0.77.
The analysis for the direct brine release is less effective, with an R z value of only 0.50 (Table 5). When regression analyses have low R 2 values, the examination of scatterplots is often an effective way to gain insights into the variables affecting the analysis outcome under consideration. In this example, many expected values are zero, with the nonzero values tending to be associated with small values for W R B R N S A T (waste residual brine saturation) and large values for H A L P O R (Fig. 13).
For the total release CCDFs, the expected release is dominated by W M I C D F L G and W T A U F A I L (Table 5), which are the dominant variables with respect to the spallings and the cuttings and cavings releases, respectively. The remaining variables selected in the analysis affect the spallings release (i.e., compare the regression results in Table 5 for spallings release and total release), which is reasonable given that the cuttings release is completely dominated by W T A U F A I L and the direct brine release makes little contribution to the total release (Fig. 10).
An alternative way to perform sensitivity analyses for distributions of CCDFs is by calculating regression coefficients or partial correlation coefficients for the exceedance probabilities associated with individual release values and then connecting these coefficients to form curves (Fig. 14). This is analogous to the analysis in Fig. 5 for time-dependent pressures except that exceedance probabilities rather than pressures are now the dependent variables. For the total release CCDFs (Fig. 10d), the two dominant variables identified in this manner are W M I C D F L G and W T A U F A I L (Fig. 14), which is consistent with the regression results in Table 5.
174 J.C. Helton/Computer Physics Ctmunum'cations 117 (1999) 156-180
15.0
< ~ 12.5
% == 10.o
7.5 -m__
5.0 121
u J
2.5
0.0
0.00
Expected Direct Brine Release: R1, R2, R3 • 1 i i i i
,g= j =_ . == • _ •
I I I I Ill I i
0.10 0.20 0.30 0.40 0.50
Waste Residual Brine Saturation: WRBRNSAT
~- 15.0
12.5
% "~ 10.0
n- 7.5
a l
~ 5.0 D
o 2.5
w
0.0
Expected Direct Brine Release: R1, R2, R3 i i u ~ i L I
m l •
4 8 12 16 20 24 28 Halite Porosity: HALPOR (10 -3 )
Fig. 13. Scatterplots for expected (mean) normalized releases associated with individual CCDFs for direct brine release versus WRBRNSAT and HALPOR.
6. Regression analysis as a procedure for sensitivity analysis
Although regression analysis is often an effective sensitivity analysis procedure lbr use in conjunction with a sampling-based uncertainty analysis, considerable care and judgment must be used in the interpretation of results obtained from its application. For example, the presence of two or more correlated variables in a regression model can result in regression coefficients that provide little indication of the actual effects of these variables; further, the changes in R 2 values as correlated variables are added to a regression model do not provide a good representation of the effects of individual variables.
In the present analysis, there were three pairs of correlated variables (i.e., (ANHPRM, ANHCOMP), (HALPRM, HALCOMP) and (BHPRM, BPCOMP), with rank correlations of -0 .99 , - 0 . 9 9 and -0 .75 , respectively; see Table 2), with the remaining variables assumed to be uncorrelated. These correlations were implemented with the Iman/Conover restricted pairing technique [57], with the result that correlated variables have their specified correlations and uncorrelated variables have correlations close to zero (i.e., the uncorre- lated variables are orthogonal) (Table 3). The - 0 . 9 9 rank correlations are quite strong (Fig. 15), with the result that each variable in the pairs (ANHPRM, ANHCOMP) and (HALPRM, HALCOMP) is effectively interchangeable for the other in a rank regression model. As this interchangeability creates instability in the sequence of regression models constructed in the stepwise process when both variables in a pair are included in the analysis (Section 7.2, [ 14] ), only one variable from each pair (i.e., ANHPRM and HALPRM; see footnote c, Table 5) was included as an independent variable in the construction of regression models. For the same reason, only ANHPRM and HALPRM were used in the calculation of PRCCs. Actually, strong correlations between sampled variables can introduce distortions in partial correlation coefficients that are more severe than those that occur in regression analyses because such correlations can result in small partial correlations between sampled variables and predicted variables (i.e., the partial correlations are small because each sampled variable removes the effect of the variable with which it is correlated).
Although the presence of correlations complicates the performance and interpretation of a sensitivity anal- ysis, this is a problem that must be dealt with in a combined uncertainty and sensitivity analysis. In par- ticular, ignoring correlations if they are present and involve influential variables will produce erroneous un-
J.C. He~ton / Computer Physi~'s Communications 117 (1999) 156-180 175
Total Normalized Releases: R1, R2, R3 300 Observations, 10000 Futures/Observation
1 .00 . . . . . . . i . . . . . . . . i . . . . . . . . i . . . . . . . . i . . . . . . . . l . . . . . . . 1 .......
0 . 7 5 . . ~
0.50 =:=
~= 025 I ~ •
0.00 ; .""
~ --0.25 - t oo .,.+J
~. -0.50 '+ . . . ' Dependent Variable ' . . '" " . . . ' TOTREL
-0,75 I- " ~ WMICDFLG / . . . . WTAUFAIL
-1,00 I , j ....... J ........ I ........ I . . . . . . . . I ........ J . . . . . . . . I . . . . . .
10-5 10 -4 10-3 10-2 10-1 100 101 102 Normalized Release (EPA units)
Fig. 14.
Q. 0.175
O O, 0.150 ,< "t-
~ 0.125 0.
~ 0.100
0.075
o E 0,050
0.025 "1-
0.000
i i i i i i i
_ ~ . Rank Correlation • ' ,Ikl~. • Specified:-0.990
~~1~,~4 ~ In Sample: -0,989 , , a= - • ~% .%
'.. . g . ; ~ •
• I I~l l = ii I l d l •11
1 I 1 I I I -24.0 -23.6 -23.2 -22.8 -22.4 -22.0 -21.6 -21.2
Logarithm Halite Permeability (m2): HALPRM
Fig. 15.
Fig. 14. Sensitivity analysis based on PRCCs for CCDFs for total normalized release to the accessible environment over 10 000 yr.
Fig. 15. Scatterplot illustrating correlation within the variable pair (HALPRM, HA LCOMP), with the sample correlation resulting from pooling the 300 observations associated with replicates RI, R2 and R3.
certainty analysis results. In the present analysis, the correlations within the pairs (ANHPRM, ANHCOMP) and (HALPRM, HALCOMP) are so strong that they could be replaced by functional relationships; this easy approach to avoiding the necessity of dealing with the effects of correlations is not possible in the presence of less strong correlations.
An additional complication in a sensitivity analysis based on regression techniques is that such an analysis is predicated on the structure of the candidate model whose coefficients are to be determined. If this candidate model is inappropriate for the patterns that exist between sampled and predicted variables, then important effects may be underestimated or missed entirely. For example, regression analyses with raw and also rank-transformed data completely missed the well-defined relationship between repository pressure and borehole permeability (BHPRM) in Fig. 7b (Section 8.4, [ 141 ). Thus, the analyst must recognize situations where an initial attempt at sensitivity analysis with a regression model or some other procedure has been unsuccessful and then apply additional procedures to determine what is giving rise to the uncertainty in the analysis outcome under study.
A large analysis such as the WIPP PA will consider hundreds of predicted variables, with some of these variables selected for consideration because they are of fundamental importance to the outcome of the analysis and other variables selected for consideration as part of a verification of the correctness of the calculations. Due to the scale of such an analysis, human resources are typically not available to design and implement a custom analysis for each of the hundreds of variables to be considered. Rather, some type of screening procedure is needed that will, with a fairly high likelihood, provide satisfactory sensitivity analysis results for most variables under consideration and also identify those variables that require additional study. Stepwise regression analysis in conjunction with a sampling-based uncertainty analysis provides such a procedure. If a regression model with a reasonably high R 2 value (e.g., _> 0.80) can be produced without overfitting data (e.g., as indicated by PRESS [63] or some other criterion) and the effects indicated by the regression model are consistent with the relationships built into the model under study, then the sensitivity analysis has been successful in identifying the dominant variables with respect to the particular outcome under consideration. In contrast, a regression model with a low R 2 value indicates that the variable under consideration requires additional investigation.
It is important to recognize that the goal of sensitivity analysis is to identify the variables that influence
176 J.C. Helton / Computer Physics Communications 117 (1999) 156-180
individual analysis outcomes. In particular, the goal is usually not to develop simplified replacement models for the original model. As a result, regression models with relatively low predictive capability may adequately serve their role in sensitivity analysis if they lead the analyst to a correct identification of the important variables with respect to the particular analysis outcome under consideration. Often, two- or three-dimensional scatterplots will suffice to understand complex nonlinear relationships or patterns of interaction between two or more variables. For example, once the effect of B H P R M indicated in Fig. 7b was identified and understood, there was no need to develop a regression model or some other type of model that captured this effect, although such a development is certainly possible. Thus, although a more sophisticated replacement models can be developed, this is usually not necessary at this point in a sensitivity analysis. In particular, the goal of sensitivity analysis, which is to identify the dominant variables, will have been satisfied without the construction of such a model.
Ultimately, it is the responsibility of the analyst to explain the uncertainty in a model prediction on the basis of the model under study and the uncertainty in the inputs to that model. Regression analysis, or any other sensitivity analysis procedure, is just a tool to help the analyst develop this explanation.
Regression models with R 2 values close to 1 are certainly desirable in regression-based sensitivity analyses. Then, it will be known that the variables included in the regression model account for most of the uncertainty in the predicted variable under consideration. As seen in the WIPP PA and as just discussed, the construction of such regression models is sometimes difficult. An alternate approach based on the indices developed by Sobol' exists that allows a complete apportioning of the uncertainty in predicted variables to the uncertainty in individual input variables [64-66]. However, it has not been established that this approach can be successfully applied to an analysis as complex and computationally intensive as the WIPP PA.
Complex patterns such as the one illustrated in Fig. 7b can be found without resorting to regression analysis by using the chi-squarc statistic to test scatterplots for deviations from randomness conditional on the marginal distributions of the two variables used in the generation of each scatterplot [67,68]. However, regression models usually, though not always, select the dominant variables even though the R ~ values may be low. Thus, an effective screening procedure is to carry out a stepwise regression analysis and also to require a scatterplot be generated for each variable selected in the regression analysis. If the regression model has a high R 2
value, then it is immediately known what the dominant variables are. If the regression model does not have a high R 2 value, then the examination of the associated scatterplots may reveal the dominant variables. In this situation, the scatterplots would be expected to reveal either a nonlinear relationship (e.g., Fig. 7b) or an interaction involving two or more variables (e.g., Fig. 13). If the scatterplots reveal no discernible patterns, then additional investigation would be required, including checking for errors in the construction of the model or the implementation of the analysis.
The distributions indicated in Eq. (7) and partially listed in Table 2 characterize subjective uncertainty in analysis inputs, lead to a characterization of the subjective uncertainty in predicted results, and determine the mapping between model input and model results that forms the basis for regression-based sensitivity analysis. If a variable has little effect on a particular analysis outcome, then the distribution assigned to it is of little importance (unless an assigned range is so unreasonably large that it excessively magnifies the role of a variable that has a real but small effect). For variables that have substantial effects, the assigned distributions are very important with respect to the assessed uncertainty in predicted results because the uncertainty in such variables transfers through the model to the uncertainty in predicted results. However, in sensitivity analysis, the actual shape of an assigned distribution is typically less important than the range. Although counterexamples can be constructed, in practice important variables will be identified in a sensitivity regardless of the distributions assigned to them (unless the assigned range for a variable is so unreasonably small that it effectively makes the variable a constant) [52].
The author believes that, in most situations, the most effective way to develop a distribution to characterize subjective uncertainty is by defining selected quantiles for the distribution (e.g., 0.00, 0.01, 0.1, 0.25, 0.5, 0.75, 0.90, 0.99, 1.00) (Section 3.1, [49]) and assuming that the variable is uniform or loguniform between the quantiles. With this approach, the reason for defining specific quantiles can be related to information
J.C. Helton/Computer Physics Communications 117 (1999) 156-180 177
possessed by the individual developing the distribution. In contrast, definitions based on specifying parameter values for known distribution types (e.g., normal, lognormal, beta . . . . ) should generally be avoided because most individuals find it difficult to relate the knowledge that they possess to the parameters that define such distributions.
7. Summary
The importance of identifying, characterizing and displaying the uncertainty in the outcomes of analyses for complex systems is now widely recognized [28]. The 1996 WIPP PA provides an example of an analysis for such a system. In particular, this analysis involves explicit treatments for both stochastic (i.e., aleatory) and subjective (i.e., epistemic) uncertainty.
Stochastic uncertainty is described by a probability space (S~t,,S,t, P~t) that characterizes future occurrences at the WIPE An assessment of the effects of stochastic uncertainty requires the evaluation of an integral over S~t, with this integral being evaluated with Monte Carlo procedures based on simple random sampling in the 1996 WIPP PA. The individual CCDFs in Fig. 10 are examples of such evaluations.
Subjective uncertainty is described by a probability space (S~u,S~u,p~u) that characterizes the uncertainty in models and associated parameters required in the calculation of consequences (e.g., radionuclide releases to the accessible environment) associated with different possible future occurrences at the WIPE with the likelihood that these occurrences will actually take place characterized by (S~t, '-~t,Pst). An assessment of the effects of subjective uncertainty (i.e., a determination of the uncertainty in analysis outcomes that results from subjective uncertainty) requires the evaluation of an integral over S~u, with this integral being evaluated with Monte Carlo procedures based on Latin hypercube sampling in the 1996 WIPP PA. For example, the individual curves in Fig. 5a are associated with individual sample elements in an LHS, and the mean and percentile results in Fig. 6 can be expressed formally as integrals involving (S~u, Ssu,p~u). More generally, a combined assessment of the effects of stochastic and subjective uncertainty requires the evaluation of a double integral over S~t and S~,. [ 24,60], with the mean and percentile curves in Fig. 11 arising from such evaluations.
Uncertainty analysis results are provided in the 1996 WIPP PA by the generation of an LHS (actually, 3 replicated LHSs) from Ssu and the propagation of this sample through the analysis. This procedure also provides the basis for sensitivity analysis by providing a mapping between analysis inputs and analysis results as indicated in Eq. (9). Once generated, this mapping can be explored with procedures based on examination of scatterplots, regression analysis and correlation analysis (Figs. 5a, 7b, 8b, 9b, 13, 14; Table 5). In the presence of nonlinear but monotonic relationships, the use of the rank transformation [62] often improves results obtained with regression analysis and correlation analysis. In the presence of more complex patterns, other procedures for identifying deviations from randomness are often effective [68]. In PAs for the WIPP and other complex systems, sensitivity analysis plays an important role by both identifying important variables requiring additional study and providing a powerful tool for model verification (i.e., sensitivity analysis has the capability to identify errors that could be easily overlooked in less extensive examinations of analysis results).
The present article provides a high-level over review of uncertainty and sensitivity analysis in PA for the WIPR with special emphasis on the 1996 WIPP PA. A related review article provides a high-level overview of the conceptual and computational structure of the 1996 WIPP PA [69]. More detailed information about the 1996 WIPP PA than can be presented in either of these reviews is available in a SNL technical report [ 14]. Further, the 1996 WIPP CCA supported by this PA is described in Ref. [ 16].
Acknowledgements
Work performed for Sandia National Laboratories (SNL), which is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy under
178 ZC. Helton/Computer Physics Communications 117 (1999) 156-180
contract D E - A C 0 4 - 9 4 A L 8 5 0 0 0 . M a n y individuals at SNL and its cont rac tors contr ibuted to the results in
this presenta t ion , inc luding D,R. Ande r son , K. Aragon , J.E. Bean, D.G. Bennet , J.W. Berg lund , R. Blaine,
M.B. Crawford , EJ . Davis, K. Economy , D.A. Galson, J.W. Garner, F.D. Hansen , T.W. Hicks, J.D. Johnson ,
H.-N. Jow, M.K. Knowles , K.W. Larson, M.E. Lord, R.J. M a c K i n n o n , M.G. Mariet ta , J. Miller, D.G. O 'Br i en ,
J.L. Ramsey, L.C. Sanchez , J.D. Schreiber , J. Schatz, A. Shinta, L.N. Smith, D.M. Stoelzel , C. S tockman,
RN. Swift , T.W. T h o m p s o n , M.S. Tierney, E Vaughn, M. Wallace, M. Wil l iamson and R.D. Wilmot . Rev iew
provided at SNL by M. Chavez , C. Crawford and M.J. Shortencarier . Editorial suppor t p rov ided by L. Harr ison,
T. Al len and H. Olms tead of Tech Reps, Inc.
References
I1[ U.S. Department of Energy, Final Environmental Impact Statement, Waste Isolation Pilot Plant, DOE/EIS-0026 (U.S. Department of Energy, Washington, DC, 1980).
12[ U.S. Department of Energy, Final Supplement: Environmental Impact Statement, Waste Isolation Pilot Plant, DOE/EIS-0026-FS (U.S. Department of Energy, Office of Environmental Restoration and Waste Management, Washington, DC, 1990).
[ 3 ] National Research Council, The Waste Isolation Pilot Plant: A Potential Solution for the Disposal of Transuranic Waste, Committee on the Waste Isolation Pilot Plant, Board on Radioactive Waste Management, Commission on Geosciences, Environment, and Resources, National Research Council (National Academy Press, Washington, D.C., 19961.
14l WIPP Performance Assessment Division, Preliminary Comparison with 40 CFR Part 191, Subpart B for the Waste Isolation Pilot Plant, December 199 I, Vols. 1-4, SAN D91-0893 / 1-4 (Sandia National Laboratories, Albuquerque, N M, 1991 - 19921.
[5 ] WIPP Performance Assessment Department, Preliminary Performance Assessment for the Waste Isolation Pilot Plant, December 1992, Vols. 1-5, SAND92-0700/I-5 (Sandia National Laboratories, Albuquerque, NM, 1992-1993).
16P J.C. Helton, J.W. Garner, R.D. McCurley, D.K. Rudeen, Sensitivity Analysis Techniques and Results for Performance Assessment at the Waste Isolation Pilot Plant, SAND90-7103 (Sandia National Laboratories, Albuquerque, NM, 19911.
171 J.C+ Helton, J.E. Bean, B.M. Butcher, J.W. Garner, J.D. Schreiber, EN. Swift, E Vaughn, Uncertainty and Sensitivity Analyses for Gas and Brine Migration at the Waste Isolation Pilot Plant, May 1992. SAND92-2013 (Sandia National Laboratories, Albuquerque, NM, 1993).
[ 81 J.C. Helton, J.W. Garner, M.G. Marietta, R.E Rechard, D.K. Rudeen, P.N. Swift, Uncertainty and sensitivity analysis results obtained in a preliminary performance assessment for the Waste Isolation Pilot Plant, Nucl. Sci. & Eng. 114 ( 19931 286-331.
19l J.C. Helton, D.R. Anderson, B.L. Baker, J.E. Bean, J.W. Berglund, W. Beyeler, J.W. Garner, H.J. luzzolino, M.G. Marietta, R.E Rechard, RJ. Roache, D.K. Rudeen, J.D. Schreiber, EN. Swift, M.S. Tierney, E Vaughn, Effect of alternative conceptual models in a preliminary performance assessment for the Waste Isolation Pilot Plant, Nucl. Eng. & Design [54 (1995) 251-344.
0] J.C. Helton. D.R. Anderson, B.L. Baker, J.E. Bean, J.W. Berglund, W. Beyeler, K. Economy, J.W Garner. S.C. Hora, H.J. luzzolino, E Knupp, M.G. Marietta, J. Rath, R.P Rechard, EJ. Roache, D.K. Rudeen, K. Salari, J.D. Schreiber, EN. Swift, M.S. Tierney, P Vaughn, Uncertainty and sensitivity analysis results obtained in the 1992 performance assessment for the Waste Isolation Pilot Plant, Reliability Eng. & System Safety 51 (1996) 53-100.
] J.C. Helton, J.E. Bean, B.M. Butcher. J.W. Garner, J.D. Schreiber, I~N. Swift, P. Vaughn, Uncertainty and sensitivity analysis for gas and brine migration at the Waste Isolation Pilot Plant: fully consolidated shaft, Nucl. Sci. & Eng+ 122 (1996) 1-31.
12] J.C. Helton, J.E Bean, B.M. Butcher, J.W. Garner, J.D. Schreiber, P.N. Swift, E Vaughn, Uncertainty and sensitivity analysis for gas and brine migration at the Waste Isolation Pilot Plant: permeable shaft with panel seals, J. Hazardous Materials 45 (19961 107-139.
13l J.C. Helton, J.E. Bean, B.M. Butcher, J.W. Garner, J.D. Schreiber, EN. Swift, E Vaughn, Uncertainty and sensitivity analysis for gas and brine migration at the Waste Isolation Pilot Plant: permeable shaft without panel seals, Reliability Eng. & System Safety 57 (1997) 299-3[6.
14] J.C. Helton, J.E. Bean. J.W. Berglund, EJ. Davis, K. Economy, J.W. Garner, J.D. Johnson, R.J. MacKinnon, J. Miller, D.G. O'Brien, J.L. Ramsey, J.D. Schreiber, A. Shinta, L.N. Smith, D.M. Stoelzel, C. Stockman, P. Vaughn, Uncertainty and Sensitivity Analysis Results Obtained in the 1996 Performance Assessment for the Waste Isolation Pilot Plant, SAND98-0365 (Sandia National Laboratories, Albuquerque, NM, 1998).
15[ J.C. Helton, J.D. Johnson, H.-N. Jow, R.D. McCurley, L.J. Rahal, Stochastic and subjective uncertainty in the assessment of radiation exposure at the Waste Isolation Pilot Plant, Human & Ecological Risk Assessment 4 (19981 469-526.
161 U.S. Department of Energy, Title 40 CFR Part 19[ Compliance Certification Application for the Waste Isolation Pilot Plant, DOE/CAO-1996-2184 (U.S. Department of Energy, Carlsbad Area Office, Carlsbad, NM, 1996).
17 ] U.S. Environmental Protection Agency, Criteria for the certification and recertification of the Waste Isolation Pilot Plant's compliance with the disposal regulations: certification decision; final rule, Federal Register 63 (1998) 27353-274/16.
[18l U.S. Environmental Protection Agency, Environmental standards for the management and disposal of spent nuclear fuel, high-level and transuranic radioactive wastes; final rule, 40 CFR Part 191, Federal Register 50 (1985) 38066-38089.
J.C. Helton / Computer Physics Commtmica tions 117 (1999) 156-180 179
119 U.S. Environmental Protection Agency, Environmental radiation protection standards for the management and disposal of spent nuclear fuel, high-level and transuranic radioactive wastes; final rule, 40 CFR Part 191, Federal Register 58 (1993) 66398-66416.
[20 U.S. Environmental Protection Agency, 40 CFR Part 194: Criteria for the certification and re-certification of the Waste Isolation Pilot Plant's compliance with the 40 CFR part 191 disposal regulations; final rule, Federal Register 61 (1996) 5224-5245.
[21 J.C. HeRon, Risk, uncertainty in risk, and the EPA release limits for radioactive waste disposal, Nucl. Technol. 101 (1991) 18-39. 122 J.C. Helton, D.R. Anderson, M.G. Marietta, R.P. Rechard, Performance assessment for the Waste Isolation Pilot Plant: from regulation
to calculation for 40 CFR 191.13, Operations Research 45 (1997) 157-177. 1231 J.C. Helton, Treatment of uncertainty in performance assessments for complex systems, Risk Analysis 14 (1994) 483-511. 1241 J.C. Helton, Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty, J. Statist. Comput. & Simul.57
(1997) 3-76. 125] S. Kaplan, B.J. Garrick, On the quantitative definition of risk, Risk Analysis 1 ( 1981 ) 11-27. 126] G. Apostolakis, The concept of probability in safety assessments of technological systems, Science 250 (1990) 1359-1364. 1271 F.O. Hoffman, J.S. Hammonds, Propagation of uncertainty in risk assessments: the need to distinguish between uncertainty due to
lack of knowledge and uncertainty due to variability, Risk Analysis 14 (1994) 707-712. 1281 J.C. Helton, DE. Burmaster, Guest editorial: treatment of aleatory and epistemic uncertainty in performance assessments for complex
systems, Reliability Eng. & System Safety 54 (1996) 91-94. 129] M.E. Pat6-Cornell, Uncertainties in risk analysis: six levels of treatment, Reliability Eng. & System Safety 54 (1996) 95-111. 130] R.L. Winkler, Uncertainty in probabilistic risk assessment. Reliability Eng. & System Safety 54 (1996) 127-132. 131 ] D. Draper, Assessment and propagation of model uncertainty, J. Roy. Statist. Soc. Ser. B 57 (1995) 45-97. 1321 D. Draper, A. Pereira, P. Prado, A. Saltelli, R. Cheal, S. Eguib6r, B. Mendes, S. Tarantola, Scenario and parametric uncertainty in
GESAMAC: A methodological study in nuclear waste disposal risk assessment, Comput. Phys. Conmmn. 117 (1999) 142-155, this issue.
1331 J.C. Helton, Computational structure of a performance assessment involving stochastic and subjective uncertainty, Proc. 1996 Winter Simulation Conference. J.M. Charnes et al., eds. (InstR. Electrical and Electronics Engineers, Piscataway, N J, 1996), pp. 239-247.
1341 W. Feller, An Introduction to Probability Theory and Its Applications, Vol. II, 2nd ed. (Wiley, New York, 1971 ). 135 I .I.E. Bean. M.E. Lord, D.A. McArthur, R.J. MacKinnon, J.D. Miller, J.D. Schreiber, Analysis Package for the Salado Flow Calculations
(Task I) of the Performance Assessment Analysis Supporting the Compliance Certification Application (CCA), Analysis package, SWCF-A: 1.2.07.4.1 :PA:QA:CCA, Sandia WlPP Central Files WPO # 40514 (Sandia National Laboratories, Albuquerque, NM, 1996).
136] DM. Stoelzel, D.G. O'Brien, 1996, Analysis Package for the BRAGFLO Direct Release Calculations (Task 4) of the Performance Assessment Calculations Supporting the Compliance Certification Application (CCA), AP-029, Brine Release Calculations, Analysis package, SWCF-A:I.2.07.4.1:PA:QA, Sandia WlPP Central Files WPO # 40520 (Sandia National Laboratories. Albuquerque, NM, 1996).
1371 ED. Hansen, M.K. Knowles, T.W. Thompson, M. Gross, J.D. McLennan, J.F. Schatz, Description and Evaluation of a Mechanistically Based Conceptual Model for Spall, SAND97-1369 (Sandia National Laboratories, Albuquerque, NM, 1997).
1381 J.W. Berglund, Mechanisms Governing the Direct Removal of Wastes from the Waste Isolation Pilot Plant Repository Caused by Exploratory Drilling, SAND92-7295 (Sandia National Laboratories, Albuquerque, NM, 1992).
1391 J.W. Berglund, Analysis Package for the Cuttings and Spalling Calculations (Tasks 5 and 6) of the Performance Assessment Calculation Supporting the Compliance Certification Application (CCA), AP-015 and AP-016, Analysis package, SWCF- A:I.2.07.4.1:PA:CCA:QA, Sandia Central Files WPO # 40521 (Sandia National Laboratories, Albuquerque, NM, 1996).
1401 AM. LaVenue, B.S. RamaRao, A Modeling Approach to Address Spatial Variability within the Culebra Dolomite Transmissivity Field, SAND92-7306 (Sandia National Laboratories, Albuquerque, NM, 1992).
1411 A.M. LaVenue, AP-018 (Task 3) Analysis of the Generalion of Transmissivity Fields for the Culebra Dolomite Compliance Certification Application (CCA), Analysis package, SWCF-A: 1.2.07.4.1 :PA:QA, Sandia WlPP Central Files WPO # 40517 (Sandia National Laboratories. Albuquerque, NM, 1996).
1421 C. Stockman, A. Shinta, J.W. Garner, Analysis Package for the Salado Transport Calculations (Task 2) of the Performance Assessment Analyses Supporting the Compliance Certification Application (CCA), AP-023, Analysis package, SWCF-A:I.2.07.4.1:PA:QA:CCA, Sandia WlPP Central Files WPO # 40515 (Sandia National Laboratories, Albuquerque, NM, 1996).
1431 l~J. Roache, The SECO suite of codes for site performance assessment, High Level Radioactive Waste Management, Proc. Fourth Annual International Conference, Las Vegas, NV, April 26-30, 1993, Vol. 2 (American Nuclear Society, La Grange Park, IL, 1993) pp. 1586-1594.
[441 J. Ramsey, M.G Wallace, Analysis Package for the Culebra Flow and Transport Calculations (Task 3) of the Performance Assessment Calculations Supporting the Compliance Certification Application (CCA), Analysis Plan 019, Analysis package, SWCF- A:WA:I.2.07.4.1:QA, Sandia WIPP Central Files WPO # 40516 (Sandia National Laboratories, Albuquerque, NM, 1996).
1451 C.M. Stone, SANTOS - A Two-Dimensional Finite Element Program for the Quasistatic, Large Deformation, Inelastic Response of Solids, SAND90-0543 (Sandia National Laboratories, Albuquerque, NM, 1997).
1461 C.M. Stone, Final Disposal Room Structural Response Calculations, SAND97-0795 (Sandia National Laboratories, Albuquerque, NM, 1997).
180 ,I.C. Helton/Computer Physics Communications 117 (1999) 156-180
[471 J.C. Helton, H.J. luzzolino, Construction of complementary cumulative distribution functions lor comparison with the EPA release limits for radioactive waste disposal, Reliability Eng. & System Safety 40 (1993) 277-293.
1481 J.C. Helton, A.W. Shiver, A Monte Carlo procedure for the construction of complementary cumulative distribution functions for comparison with the EPA release limits for radioactive waste disposal, Risk Analysis 16 (1996) 43-55.
149] J.C. Helton, Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal, Reliability Eng. & System Safety 42 (1993) 327-367.
1501 M.D. McKay, R.J. Beckman, WJ. Conover, A comparison of three methods for selecting values of input variables in the analysis of output from a computer code, Technometrics 21 (1979) 239-245.
[511 R.L. lman, J.C. Helton, The repeatability of uncertainty and sensitivity analyses for complex probabilistic risk assessments, Risk Analysis I 1 (1991) 591-606.
152 [ J.C. Helton, J.D. Johnson, M.D. McKay, A.W. Shiver, J.L. Sprung, Robustness of an uncertainty and sensitivity analysis of early exposure results with the MACCS reactor accident consequence model, Reliability Eng. & System Safety 48 (1995) 129-148.
[53] R.L. Iman, W.J. Conover, Small sample sensitivity analysis techniques for computer models, with an application to risk assessment. Commun. Star. A 9 (1980) 1749-1842; rejoinder to Comments, Commun. Stat. A 9 (1980) 1863-1874.
[541 M. Stein, Large sample properties of simulations using Latin hypercube sampling, Technometrics 29 (1987) 143-151. 1551 A.B. ()wen, A central limit theorem for Latin hypercube sampling, J. Roy. Statist. Soc. Ser. B 54 (1992) 541-551. 561 R.L. Imam J.C. Helton, An investigation of uncertainty and sensitivity analysis techniques for computer models, Risk Analysis 8
(1988) 71-90. 57] R.L. lman, W.J. Conover, A distribution-free approach to inducing rank correlation among input variables, Commun. Stat.: Simul. &
Comput. B I1 (1982) 311-334. 581 R.L. lman, J.M. Davenport, Rank correlation plots for use with correlated input variables, Commun. Stat.: Simul. & Comput. B 11
(1982) 335-360. 591 R.L. Iman, Statistical methods for including uncertainties associated with the geologic isolation of radioactive waste which allow
['or a comparison with licensing criteria, Proceedings of the Symposium on Uncertainties Associated with the Regulation of the Geologic Disposal of High Level Radioactive Waste, Gatlinburg, Tennessee, March 9-13, 1981, D.C. Kocher, ed., NUREG/CP-0022, CONF-810372 (Oak Ridge National Laboratory, Oak Ridge, TN, 1982) pp. 145-157.
601 J.C. Helton, Probability, conditional probability and complementary cumulative distribution functions in performance assessment for radioactive waste disposal, Reliability Eng. & System Safety 54 (1996) 145-163.
[611 R.J. MacKinnon, G. Freeze, H.-N. Jow, Summary of EPA-Mandated Performance Assessment Verification Test (Replicate 1) and Comparison with the Compliance Certification Application Calculations, Sandia WIPP Central Files WPO # 46674 (Sandia National Laboratories. Albuquerque, NM, 1997).
162J R.L. hnan, W.J. Conover, The use of the rank transform in regression, Technometrics 21 (1979) 499-509. 163 ] D.M. Allen, The Prediction Sum of Squares as a Criterion for Selecting Predictor Variables, Report No. 23 (Department of Statistics,
University of Kentucky, Lexington, KY, 1971 ). [64J I.M. Sobol', Sensitivity analysis for nonlinear mathematical models, Math. Model. & Comput. Exp. 1 (1993) 407-414; translation
of: I.M. Sobol', Sensitivity estimates for nonlinear mathematical models, Matematicheskoe Modelirovanie 2 (1990) 112-118 [in Russian I.
165 ] T. Homma, A. Saltelli, Importance measures in global sensitivity analysis of nonlinear models, Reliability Eng. & System Safety 52 (1996) 1-17.
1661 A. Saltelli, S. Tarantola, K. Chan, A quantitative model-independent method for global sensitivity analysis of model output, Technometrics (1999), to appear.
[67] H.M. Wagner, Global sensitivity analysis, Operations Research 43 (1995) 948-969. 1681 J.P.C. Kleijnen, J.C. Helton, Statistical Analyses of Scatterplots to Identify Important Factors in Large-Scale Simulations, SAND98-
2202 (Sandia National Laboratories, Albuquerque, NM, 1999). 1691 J.C. Helton, D.R. Anderson, H.-N. Jow, M.G. Marietta, G. Basabilvazo, Performance assessment in support of the 1996 compliance
certification application for the Waste Isolation Pilot Plant, Risk Analysis (1999), to appear.