a wavelet-based spectral procedure for steady-state …stochastics and statistics a wavelet-based...

European Journal of Operational Research 174 (2006) 1769–1801

www.elsevier.com/locate/ejor

Stochastics and Statistics

A wavelet-based spectral procedure for steady-statesimulation analysis q

Emily K. Lada a,1, James R. Wilson b,*

a SAS Institute Inc., 100 SAS Campus Drive, R5413, Cary, NC 27513-8617, USAb Department of Industrial Engineering, North Carolina State University, Campus Box 7906, Raleigh, NC 27695-7906, USA

Received 23 February 2004; accepted 21 April 2005Available online 27 June 2005

Abstract

We develop WASSP, a wavelet-based spectral method for steady-state simulation analysis. First WASSP determinesa batch size and a warm-up period beyond which the computed batch means form an approximately stationary Gauss-ian process. Next WASSP computes the discrete wavelet transform of the bias-corrected log-smoothed-periodogram ofthe batch means, using a soft-thresholding scheme to denoise the estimated wavelet coefficients. Then taking the inversediscrete wavelet transform of the thresholded wavelet coefficients, WASSP computes estimators of the batch means log-spectrum and the steady-state variance parameter (i.e., the sum of covariances at all lags) for the original (unbatched)process. Finally by combining the latter estimator with the batch means grand average, WASSP provides a sequentialprocedure for constructing a confidence interval on the steady-state mean that satisfies user-specified requirements con-cerning absolute or relative precision as well as coverage probability. An experimental performance evaluation demon-strates WASSP�s effectiveness compared with other simulation analysis methods.� 2005 Elsevier B.V. All rights reserved.

Keywords: Simulation; Steady-state analysis; Spectral method; Time series; Wavelet analysis

0377-2217/$ - see front matter � 2005 Elsevier B.V. All rights reserved.doi:10.1016/j.ejor.2005.04.025

q A preliminary, abridged version of some of this work appeared as: Lada, E.K., Wilson, J.R., Steiger, N.M., 2003. A wavelet-basedspectral method for steady-state simulation analysis. In: Chick, S., Sanchez, P.J., Ferrin, D., Morrice, D.J. (Eds.), Proceedings of the2003 Winter Simulation Conference, Institute of Electrical and Electronics Engineers, Piscataway, NJ, pp. 422–430. Available onlinevia <www.informs-sim.org/wsc03papers/052.pdf> [accessed April 12, 2005].

* Corresponding author. Tel.: +1 919 515 6415; fax: +1 919 515 5281.E-mail addresses: [email protected] (E.K. Lada), [email protected] (J.R. Wilson).URLs: http://www4.ncsu.edu/~eklada (E.K. Lada), http://www.ie.ncsu.edu/jwilson (J.R. Wilson).

1 Tel.: +1 919 531 1391; fax: +1 919 677 4444.

http://www.ie.ncsu.edu/jwilson

mailto:[email protected]

mailto:[email protected]

http://www4.ncsu.edu/~eklada


1770 E.K. Lada, J.R. Wilson / European Journal of Operational Research 174 (2006) 1769–1801

1. Introduction

In a nonterminating probabilistic simulation, we are interested in long-run (steady-state) averageperformance measures; and usually we seek to compute point and confidence interval estimators for someparameter, or characteristic, of the steady-state cumulative distribution function (c.d.f.) of a particularsimulation-generated response (performance measure). In this paper, we limit the discussion to estimationof the steady-state mean of the given response.

There are two fundamental problems associated with analyzing stochastic output from a nonterminatingsimulation. The first problem is that we do not usually possess sufficient information to start a simulation insteady-state operation; and thus it is necessary to determine an adequate length for the initial ‘‘warm-up’’period so that for each simulation output generated after the end of the warm-up period, the correspondingexpected value is sufficiently close to the steady-state mean. If observations generated prior to the end of thewarm-up period are included in the analysis, then the resulting point estimator of the steady-state meanmay be biased; and such bias in the point estimator may severely degrade not only the accuracy of the pointestimator but also the probability that the associated confidence interval will cover the steady-state mean(Wilson and Pritsker, 1978a,b). This phenomenon is known as the start-up (or initialization-bias) problem.

Although the start-up problem and considerations of convenience often motivate users to base theiranalysis of a nonterminating stochastic simulation on the output of a single prolonged run (or independentreplication) of the simulation, this approach naturally leads to the second fundamental problem in such ananalysis: namely, that pronounced stochastic dependencies typically occur among successive responses gen-erated within a single simulation run. Sometimes called the correlation problem, this phenomenon compli-cates the construction of a confidence interval for the steady-state mean because standard statisticalmethods require independent and identically distributed (i.i.d.) normal observations to yield a valid confi-dence interval—that is, a confidence interval whose actual coverage probability is equal to the correspond-ing user-specified confidence coefficient.

A number of methods have been developed for steady-state simulation output analysis. In the method ofnonoverlapping batch means (NBM), the sequence of simulation-generated outputs is divided into adjacentnonoverlapping batches of sufficiently large size so that the resulting batch means are approximately i.i.d.observations from a normal distribution centered on the steady-state mean response; and thus a confidenceinterval for the steady-state mean can be based on the classical Student t-ratio involving the grand averageand sample variance of the batch means (Fishman, 1978, 1998; Law and Carson, 1979; Fishman and Yar-berry, 1997; Chen and Kelton, 2003).

Steiger and Wilson (2002) develop an extension of the classical NBM method in which the length of theinitial warm-up period and the size of all subsequent batches are taken separately to be just large enough toyield approximately stationary, normal batch means that may still exhibit substantial levels of correlation;and then an inverse Cornish–Fisher expansion for the classical NBM t-ratio is used to compute a correla-tion-adjusted confidence interval for the steady-state mean. Called ASAP, the procedure of Steiger andWilson is based on the observation that for the purpose of constructing a valid confidence interval on thesteady-state mean, the batch means often achieve approximate joint multivariate normality at a batch size thatis substantially smaller than the batch size required to ensure approximate independence of the batch means.

One advantage of ASAP is that it is completely automated and requires no user intervention. The resultsof an extensive experimental performance evaluation indicate that ASAP significantly outperforms previ-ous NBM procedures in a wide range of test problems. However, there are certain stochastic systems forwhich significant departures from normality of the batch means are observed even for batch sizes suffi-ciently large to ensure negligible dependencies among the batch means. To avoid anomalous behavior insuch cases, Steiger et al. (2002) develop ASAP2, an improved variant of ASAP. Although it is superiorin several respects to its predecessor, ASAP2 is designed primarily for applications involving a user-spec-ified absolute or relative precision requirement on the final delivered confidence interval; and in the absence

E.K. Lada, J.R. Wilson / European Journal of Operational Research 174 (2006) 1769–1801 1771

of such a precision requirement, ASAP2-generated confidence intervals can be highly variable in their half-lengths (Steiger et al., 2002; Lada, 2003; Lada et al., 2003). To further enhance the performance of ASAP2,Steiger et al. (2005) develop the ASAP3 algorithm, which retains the advantage of its predecessors but isspecifically designed to prevent excessive confidence interval variability even in the absence of a precisionrequirement.

Another approach to analyzing stochastic output from a steady-state simulation is to apply a spectralmethod (Fishman and Kiviat, 1967; Law and Kelton, 2000). Heidelberger and Welch (1981a,b, 1983) de-velop a spectral method for steady-state output analysis in which they use standard regression techniques toestimate the power spectrum of the simulation-generated output process at zero frequency. Also called thesteady-state variance parameter (SSVP), this quantity is equal to the sum of the covariances at all lags forthe given output process; and Heidelberger and Welch estimate the SSVP by (a) fitting a quadratic polyno-mial to the logarithm of the smoothed periodogram of the given output process in a small neighborhoodwith zero frequency as its lower boundary; and (b) extrapolating the fitted polynomial to zero frequency.The resulting SSVP estimator is then used to compute a confidence interval for the steady-state mean.

Since the fast Fourier transform of the simulation-generated time series can be used to compute the peri-odogram, Heidelberger and Welch�s spectral method is computationally efficient. Furthermore, their ap-proach involves working with logarithms of averages of pairs of adjacent, asymptotically independentperiodogram values so that the resulting ordinates of the log-smoothed-periodogram are approximatelyindependent and thus should be easier to handle than the original simulation-generated responses, whichare often strongly correlated. However because the smoothing operation is limited to pairs of adjacent peri-odogram values, the log-smoothed-periodogram may still exhibit highly erratic behavior; and the resultingvariability of the SSVP estimator may degrade the performance of the Heidelberger–Welch method in prac-tice. The experimental results obtained by Heidelberger and Welch also indicate that in some cases a pro-cedure based on fitting low-degree polynomials to the log-smoothed-periodogram is not sufficiently flexibleto yield an unbiased estimator of the associated power spectrum at zero frequency. Thus Heidelberger andWelch�s SSVP estimator may possess a large bias (due to lack of fit in estimating the log-spectrum) as wellas a large variance (due to inadequate smoothing of the periodogram).

The primary objective of this article is to develop a completely automated sequential spectral method bywhich an approximately valid confidence interval is constructed for the steady-state mean of a simulationoutput process. This procedure, called WASSP, determines a batch size and a warm-up period beyondwhich the computed batch means form an approximately stationary Gaussian process. For this purposewe use the randomness test of von Neumann (1941) to determine an interbatch spacer preceding each batchthat is sufficiently large to ensure the resulting spaced batch means are approximately i.i.d. We then take thespacer preceding the first batch to be the warm-up period, and we use the univariate normality test of Shap-iro and Wilk (1965) to determine a batch size that is sufficiently large to ensure the spaced batch means areapproximately normal. Using all the adjacent (nonspaced) batch means of the resulting batch size that arecomputed beyond the warm-up period, we obtain a wavelet-based estimator of the batch means log-spec-trum (Vidakovic, 1999; Percival and Walden, 2000) over its full frequency range (that is, from � 1

2to 1

2cycles

per unit of time).To obtain a sufficiently accurate wavelet-based estimator of not only the SSVP but also the batch means

log-spectrum over its full frequency range, first we smooth the periodogram of the batch means by comput-ing a multipoint moving average (consisting of seven points by default); then we apply a logarithmic trans-formation to the smoothed periodogram and correct for the bias induced by this transformation. Next wecompute the discrete wavelet transform (DWT) of the bias-corrected log-smoothed-periodogram of thebatch means and apply a soft-thresholding scheme to obtain a parsimonious, denoised set of wavelet coef-ficient estimators. Finally, we compute the inverse DWT of the thresholded wavelet coefficients to recoverthe wavelet-based approximation to the batch means log-spectrum and ultimately an estimator of the SSVPfor the original (unbatched) process.


The final confidence interval delivered by WASSP is centered on the grand average of the retained batchmeans and has a half-length that accounts for the ‘‘effective’’ degrees of freedom in the wavelet-based esti-mator of the SSVP as determined by the level of smoothing of the batch means periodogram. WASSP ad-dresses both the start-up and correlation problems, delivering a confidence interval for a steady-state meanresponse that satisfies a user-specified absolute or relative precision requirement.

The rest of this article is organized as follows. In Section 2 we introduce the notation required for ourdiscussion of steady-state simulation analysis, and we also describe briefly the fundamental concepts andnotation of wavelet analysis. A high-level overview of the structure and operation of WASSP is presentedin Section 3, and the major steps of WASSP are detailed in Section 4. Some results from our performanceevaluation of WASSP are presented in Section 5. Finally in Section 6 we summarize the main conclusions ofthis research, and we make recommendations for future work. To justify key elements of the design ofWASSP, in the Appendices we sketch the derivations of the bias and variance of the batch means log-smoothed-periodogram. Lada (2003) and Lada and Wilson (2003) provide a complete development ofthe results summarized in this article. Some preliminary results on the formulation of WASSP and its exper-imental performance evaluation are presented in Lada et al. (2003, 2004a). Complete details on the exper-imental performance evaluation of WASSP are presented in Lada et al. (2004b).

2. Preliminaries

2.1. Setup for steady-state simulation analysis

Let {Xi : i = 1, 2, . . .} denote a stochastic process representing the sequence of outputs generated by a sin-gle run of a nonterminating probabilistic simulation. If the simulation is in steady-state operation, then fori = 1, 2, . . . , the random variable Xi will have the steady-state c.d.f. FX (x) = Pr{Xi 6 x} for all real x. Inthis article, we concentrate on estimating the steady-state mean, lX ¼ E½X � ¼

R1�1 x dF X ðxÞ; and we limit

the discussion to output processes for which E½X 2i � <1 so that the process mean lX and process variance

r2X ¼ Var½X i� ¼ E½ðX i � lX Þ

2� are well defined. We let n denote the length of the time series {Xi} of outputsgenerated by a single, prolonged run of the simulation.

The sample mean, X ¼ 1n

Pni¼1X i, is an intuitively appealing point estimator of lX. Furthermore, if {Xi} is

weakly stationary, then the covariance of the process at lag ‘ is cX (‘) = E [(Xi � lX)(Xi+‘ � lX)] for alli P 1 and ‘ = 0, ±1, ±2, . . . ; and the steady-state variance parameter (SSVP) of the process is

cX ¼X1‘¼�1

cX ð‘Þ; ð1Þ

provided that the right-hand side of (1) is absolutely convergent so that cX is well defined. In this article werestrict attention to a simulation-generated process {Xi} for which the SSVP is well defined. Under fairlygeneral conditions on such a process, the variance of the corresponding sample mean is given byVar½X � ¼ cX=nþ Oð1=n2Þ (see, for example, Lemma 6 of Chien et al., 1997); moreover for 0 < b < 1, anasymptotically valid 100(1 � b)% confidence interval for lX is given by

X � z1�b=2

ffiffiffiffiffiffiffiffiffifficX=n

p;

where z1�b/2 is the 1 � b/2 quantile of the standard normal distribution.At the frequency x expressed in cycles per time unit, the power spectrum pX (x) of the output process

{Xi : i = 1, 2, . . . , n} is given by the cosine transform of the covariance function cX (‘),

pX ðxÞ ¼X1‘¼�1

cX ð‘Þ cosð2px‘Þ for � 1

26 x 6

1

2


(Heidelberger and Welch, 1981a). At frequencies of the form ‘n for ‘ = 0, 1, . . . , n � 1, the periodogram asso-

ciated with {Xi : i = 1, 2, . . . , n} is defined as

I‘

n

� �¼ 1

n

Xn

j¼1

X j cos2pðj� 1Þ‘

n

� �" #2

þXn

j¼1

X j sin2pðj� 1Þ‘

n

� �" #28<:

9=; ¼ jðFXÞ‘j2=n for ‘ ¼ 0; 1; . . . ; n� 1;

where FX is the fast Fourier transform (FFT) of the simulation-generated time series X = {X1, . . . , Xn}.Letting fv2

‘ð2Þ : l ¼ 1; 2; . . .g denote i.i.d. chi-squared random variates each with two degrees of freedom,we see that the periodogram has the following asymptotic properties for sufficiently large n:

E I ‘n

� �� pX

‘n

� �; if 0 < ‘ < n

2;

Var I ‘n

� �� p2

X‘n

� �; if 0 < ‘ < n

2;

I ‘n

� �; IðjnÞ are approximately independent; if 0 < ‘ 6¼ j < n

2;

I ‘n

� �� 1

2pX

‘n

� �v2‘ð2Þ; if 0 < ‘ < n

2;

9>>>>=>>>>; ð2Þ

see Theorem 5.2.6 of Brillinger (1981).Instead of working in the time domain with the original output process {Xi}, we are able to work in the

frequency domain if we exploit a spectral approach to steady-state simulation output analysis. At x = 0, wehave

pX ð0Þ ¼X1‘¼�1

cX ð‘Þ ¼ cX ;

and consequently the goal of any spectral method for steady-state simulation output analysis is to estimatepX (0) from the values of the periodogram in the vicinity of zero frequency.

To construct an approximate confidence interval on the steady-state mean lX of a simulation outputprocess {Xi}, WASSP uses a wavelet-based procedure to obtain an estimator bcX of cX; and then WASSPcombines bcX with a version of the overall sample mean X that has been suitably truncated if necessaryto eliminate initialization bias so as to deliver an approximate 100(1 � b)% confidence interval of the form

X � t1�b=2;m

ffiffiffiffiffiffiffiffiffiffiffiffibcX=n0p

; ð3Þ
where n 0 denotes the size of the truncated data set from which X and bcX are computed; m denotes the ‘‘effec-tive’’ degrees of freedom associated with bcX ; and t1�b/2,m denotes the 1 � b/2 quantile of Student�s t-distri-bution with m degrees of freedom. WASSP is a sequential procedure and may request additional dataiteratively before it delivers a final confidence interval of the form (3) that has approximate coverage prob-ability 1 � b and that satisfies a user-specified absolute or relative precision requirement.
2.2. Setup for wavelet analysis

In the following overview of wavelet analysis, Z denotes the set of integers, R denotes the real line, andL2ðRÞ denotes the space of square-integrable real-valued functions defined on R. We start with a scalingfunction /ðtÞ 2 L2ðRÞ having the properties
Z
R

/ðtÞdt 6¼ 0 and

ZR

/2ðtÞdt ¼ 1;

moreover, we require that /(t)! 0 sufficiently fast as jtj ! 1, and the integer-translatesf/ðt � ‘Þ : ‘ 2 Zg must form an orthonormal basis for a certain subspace of L2ðRÞ as detailed in Section3.3 of Vidakovic (1999) and Sections 1.1–1.3 of Percival and Walden (2000). From /(t) we construct awavelet function wðtÞ 2 L2ðRÞ with the properties

ZR
wðtÞdt ¼ 0;

ZR

w2ðtÞdt ¼ 1; and

ZR

wðtÞ/ðt � ‘Þdt ¼ 0 for ‘ 2 Z.

By translating and scaling /(t) and w(t), for any fixed j0 2 Z we can construct an orthonormal basis forL2ðRÞ consisting of the wavelet functions fwj;‘ðtÞ ¼ 2j=2wð2jt � ‘Þ : j P j0; ‘ 2 Zg and the scaling functions

/j0;‘ðtÞ ¼ 2j0=2/ð2j0 t � ‘Þ : ‘ 2 Z

n o. Thus any target function f ðtÞ 2 L2ðRÞ can be expressed as

f ðtÞ ¼X‘2Z

cj0;‘/j0;‘ðtÞ þ

X1j¼j0

X‘2Z

dj;‘wj;‘ðtÞ for almost all t; ð4Þ

where the scaling coefficients,

cj0;‘ ¼Z

R

f ðtÞ/j0;‘ðtÞdt for ‘ 2 Z;

and the wavelet coefficients,

dj;‘ ¼Z

R

f ðtÞwj;‘ðtÞdt for ‘ 2 Z and j P j0;

are defined as the inner product of f (t) and the basis functions /j0;‘ðtÞ and wj,‘ (t), respectively (Vidakovic,

1999). The representation (4) is analogous to the Fourier series representation of a square-integrable real-valued function on the interval [0, 2p] in terms of trigonometric basis functions; see, for example, problem42.1 of Wilcox and Myers (1978).

The discrete wavelet transform (DWT) maps data from the time domain to the wavelet domain. Supposethe data vector Y = (Y1, . . . , Yn)T represents the n sample values of a target stochastic process that is ob-served at equally spaced points of the corresponding index or ‘‘time’’ parameter of the process; and supposewe seek to estimate the process-mean function f (t) = E [Yt] for t = 1, . . . , n using the given data vector. TheDWT of Y is defined by

W ¼ HY ; ð5Þ

where (i) H is the n · n orthogonal matrix that defines the DWT associated with the particular scaling func-tions and wavelet functions used in the representation (4); and (ii) W is the n · 1 vector of estimated scalingand wavelet coefficients. Extending the analogy between wavelet analysis and Fourier analysis, we see thattransforming a data set via the DWT closely resembles the process of computing the FFT of that data set.Because of the orthogonality of H, the inverse DWT is given by

Y ¼ HTW ;

where HT denotes the transpose of H. Mallat (1989) developed an efficient algorithm to compute the DWTand the inverse DWT if the total sample size has the form n = 2J for some positive integer J. For moredetails on this algorithm, see Percival and Walden (2000). In this article, we assume the sample size n isa power of two, and we use Mallat�s algorithm to compute the DWT and the inverse DWT.

In general, after applying the DWT to a data set Y = (Y1, . . . , Yn)T, we obtain the following approxima-tion formula for the associated process-mean function:

f ðtÞ �X2j0�1

‘¼0

bcj0;‘/j0;‘ðtÞ þ

XJ�1

j¼j0

X2j�1

‘¼0

bd j;‘wj;‘ðtÞ;

where the estimated scaling coefficients bcj0;‘ : ‘ ¼ 0; 1; . . . ; 2j0 � 1

and the estimated wavelet coefficientsbd j;‘ : ‘ ¼ 0; 1; . . . ; 2j � 1n o

for the jth level of resolution (j = j0, j0 + 1, . . . , J � 1) all appear as correspond-


ing elements of the vector W defined by (5). In general, at the jth level of resolution, there will be 2j waveletcoefficients, beginning with the coarsest resolution level j = j0 and running up to the finest resolution levelj = J � 1; moreover, there will be 2j0 scaling coefficients at level j0. For a more detailed overview of waveletanalysis, see Lada (2003, pp. 24–27).

3. Overview of WASSP: A wavelet-based analyzer for steady-state simulation processes

Fig. 1 depicts a high-level flow chart of the operation of WASSP. The algorithm begins by dividing theinitial simulation-generated output process of length n = 4096 observations into a set of k = 256 batches ofsize m = 16 observations each, with a spacer of initial size S = 0 observations preceding each batch forwhich we will calculate a batch mean. The randomness test of von Neumann (1941) is applied to the result-ing set of spaced batch means. The randomness test serves two purposes:

• It is used to construct a set of spaced batch means such that the interbatch spacer preceding each batch issufficiently large to ensure all computed batch means are approximately i.i.d. so that subsequently thespaced batch means can be tested for normality.

• It is used to determine an appropriate data-truncation point—that is, the end of the warm-up period aswell as the size of the interbatch spacer preceding the first batch—beyond which all computed batchmeans are approximately independent of the simulation model�s initial conditions.

Each time the randomness test is failed, an additional batch is added to each spacer (up to a limit of ninebatches per spacer); and then the randomness test is reperformed on the new (reduced) set of spaced batchmeans. If the randomness test is failed with a spacer consisting of nine batches so that only 25 spaced batchmeans are used in the test, then the following steps are executed: (a) the batch size m and the total sample size

Start

Randomnesstest passed?

Normality test

Compute new

No

Yes

No

Yes

No

passed?

Stop

requirements?

CI meets precision

Yes

compute spaced batch meansCollect observations;

Increase spacer sizeor batch size m

Fix spacer size

batch size m

compute nonspaced batch meansfrom all remaining observations of the batch means

Compute log of the smoothed periodogram

Compute wavelet–basedestimate of thelog–spectrum

spectral estimate of SSVCCompute wavelet–based

Construct wavelet–basedspectral CI

recompute batch meansCollect extra observations;

Limit batch count,min{4096, k}

Compute batch size,

n/k

compute spaced batch meansCollect observations;

n,Compute total required

sample size

m

k

Skip the first observations;

S

S

S

of formnew batch count k

2J

Fig. 1. High-level flow chart of WASSP.


n are both increased by the factorffiffiffi2p

; (b) the required additional observations are obtained (by restarting thesimulation if necessary); and (c) the process of testing the batch means for randomness is restarted by com-puting adjacent batch means of the new batch size (that is, with interbatch spacers of initial size S = 0).

Once the randomness test is passed, the set of approximately i.i.d. spaced batch means is tested for nor-mality using the method of Shapiro and Wilk (1965). Each time the normality test is failed, the followingsteps are executed: (i) the batch size m is increased by the factor

ffiffiffi2p

; (ii) the required additional observationsare obtained (by restarting the simulation if necessary); (iii) a new set of spaced batch means is computedusing the final spacer size S determined in the randomness test; and (iv) the normality test is repeated for thenew set of spaced batch means.

Once the normality test is passed, all simulation-generated data beyond the warm-up periodfX i : i ¼ 1; . . . ; Sg determined by the randomness test are used to compute adjacent (nonspaced) batchmeans of the batch size m determined by the normality test. The periodogram of the approximately normalbatch means is then computed and smoothed by taking a moving average of A points on the periodogramof the batch means. WASSP allows the user to specify the value of A in the set {5, 7, 9, 11}, with the defaulttaken as A = 7.

To obtain an estimator of the SSVP of the original (unbatched) process, we compute a wavelet-basedestimator of the batch means log-spectrum. This involves taking the logarithmic transformation of thesmoothed periodogram of the batch means; correcting the resulting log-smoothed-periodogram for the biasintroduced by the logarithmic transformation; and then computing the discrete wavelet transform of thebias-corrected log-smoothed-periodogram of the batch means over the full frequency range ½� 1

2; 1

2�. The

estimated wavelet coefficients are thresholded (denoised) using a variant of the soft-thresholding schemeof Gao (1997); and finally the inverse discrete wavelet transform of the thresholded wavelet coefficients pro-vides our approximation to the batch means log-spectrum. From the wavelet-based estimator of the batchmeans log-spectrum, we compute an estimator of the spectrum of the original (unbatched) process at zerofrequency (that is, the SSVP); and we compute a confidence interval of the form (3), where the midpoint ofthe confidence interval is the grand average of all the adjacent (nonspaced) batch means that are computedafter skipping the initial spacer.

The confidence interval (3) is then tested to determine if it satisfies a user-specified absolute or relativeprecision requirement. If the precision requirement is satisfied, then WASSP delivers the latest confidenceinterval and terminates; otherwise, the following steps are executed:

1. The total required sample size is estimated; and on the assumption that the current batch size is main-tained, the estimated batch count is expressed as the largest power of two yielding a total delivered sam-ple size not exceeding the required sample size.

2. If the estimated batch count exceeds 4096, then the batch count is reduced to 4096. Given the batchcount, we adjust the batch size if necessary so that the total delivered sample size is the smallest possiblevalue not less than the total required sample size.

3. The required additional observations are obtained (by restarting the simulation if necessary); and theadjacent (nonspaced) batch means are recomputed using the latest batch size after skipping thewarm-up period.

4. The log-smoothed-periodogram for the new set of batch means is computed.5. A new estimate of the SSVP is obtained from the wavelet-based approximation to the log-smoothed-

periodogram for the latest set of batch means.6. The confidence interval (3) is recomputed and the precision requirement (stopping condition) is retested.

If the confidence interval (3) in step 6 above fails to satisfy the precision requirement, then it is not neces-sary to repeat the randomness test or the normality test; instead we reperform steps 1–6 above until theprecision requirement is satisfied.


4. Detailed operational steps of WASSP

4.1. Formal algorithmic statement of WASSP

WASSP requires the following user-supplied inputs:

1. a simulation-generated output process {Xi : i = 1, 2, . . . , n} from which the steady-state expected responselX is to be estimated;

2. the desired confidence interval coverage probability 1 � b, where 0 < b < 1; and3. an absolute or relative precision requirement specifying the final confidence interval half-length in terms of

(a) a maximum acceptable half-length h* (for an absolute precision requirement); or(b) a maximum acceptable fraction r* of the magnitude of the confidence interval midpoint (for a rel-

ative precision requirement).

WASSP delivers the following outputs:

1. a nominal 100(1 � b)% confidence interval for lX that satisfies the specified absolute or relative precisionrequirement, provided no additional data are required; or

2. a new, larger sample size n to be supplied to WASSP when it is executed again.

If additional observations of the target process must be generated by the user�s simulation model before aconfidence interval with the required precision can be delivered, then WASSP must be executed again withall the observations accumulated so far; and this cycle of simulation followed by automated wavelet-basedspectral output analysis may be repeated before WASSP finally delivers a confidence interval.

A formal algorithmic statement of WASSP is given in Fig. 2. Notice that Eqs. (6)–(12) occur in Fig. 2. InSections 4.2–4.5 we describe the steps of the procedure in more detail. Information on downloading andrunning the WASSP software can be found in Lada and Wilson (2004).

4.2. Eliminating initialization bias

WASSP begins by dividing the initial sample {Xi : i = 1, . . . , n} into k = 256 adjacent (nonspaced)batches of size m = 16. Let

X j ¼ X jðmÞ ¼1

m

Xmj

i¼mðj�1Þþ1

X i ð13Þ

denote the jth batch mean for j = 1, . . . , k; and let

X ðm; kÞ ¼ 1

k

Xk

j¼1

X jðmÞ ð14Þ

denote the grand average of the k batch means.We apply the randomness test of von Neumann (1941) to the batch means fX 1ðmÞ; . . . ;X kðmÞg by com-

puting the ratio of the mean square successive difference of the batch means to the sample variance of thebatch means. In the NBM procedures LBATCH and ABATCH, Fishman and Yarberry (1997) apply thevon Neumann test for randomness iteratively to determine a batch size that is sufficiently large to yieldapproximate independence of adjacent batch means. By contrast, in WASSP we apply the von Neumanntest for randomness iteratively to determine the size of an interbatch spacer that is sufficiently large to yield

Fig. 2. Algorithmic statement of WASSP.


Fig. 2 (continued)


Fig. 2 (continued)


approximate independence of the corresponding spaced batch means. At the level of significance aran = 0.2,we test the null hypothesis of independent, identically distributed batch means,

Hran : X jðmÞ : j ¼ 1; . . . ; k

are i:i:d:; ð15Þ

by computing the test statistic,


Ck ¼ 1�Pk�1

j¼1 X jðmÞ � X jþ1ðmÞ� �2

2Pk

i¼1 X iðmÞ � X ðm; kÞh i2

; ð16Þ

which is a relocated and rescaled version of the ratio of the mean square successive difference of the batchmeans to the sample variance of the batch means. Since WASSP�s test for randomness always involves atleast 25 batch means, we use a normal approximation to the null distribution of the test statistic (16); seeYoung (1941) or Fishman and Yarberry (1997, p. 303). If

jCkj 6 z1�aran=2

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiðk � 2Þ=ðk2 � 1Þ

q; ð17Þ

then the hypothesis (15) is accepted; otherwise (15) is rejected so that WASSP must increase the spacer sizebefore retesting (15). Through extensive experimentation, we found that setting aran = 0.2 works well inpractice and provides an effective balance between errors of type I and II in testing the hypothesis (15).

If the k = 256 adjacent batch means defined by (13) pass the randomness test (15)–(17) at the level ofsignificance aran, then we set the number of batch means to be used in the normality test according tok 0 256; and we proceed to perform the normality test as detailed in Section 4.3 below. On the other hand,if the k = 256 batch means fail the test for randomness, then we insert spacers each consisting of one ig-nored batch between the k 0 128 remaining batches whose corresponding spaced batch means are tobe retested for randomness; and thus the updated spacer size is S m observations. That is, every otherbatch, beginning with the second batch, is retained for assignment as one of the spaced batch means; andthe alternate batches are ignored.

We reapply the randomness test to the k 0 = 128 remaining spaced batch means X 2ðmÞ;

X 4ðmÞ; . . . ;X 256ðmÞg having batch size m and spacers of size S = m. If the randomness retest is passed, thenwe proceed to perform the normality test in Section 4.3 with the current values of S and k 0; otherwise weadd another ignored batch to each spacer so that the spacer size and number of remaining batches are up-dated according to

S S þ m and k0 bn=ðmþ SÞc. ð18Þ

After executing (18), we have k 0 = 85 remaining spaced batch means X 3ðmÞ;X 6ðmÞ; . . . ;X 255ðmÞ

withbatch size m and spacers of size S = 2m; and again the remaining spaced batch means are retested for ran-domness. This process is continued until one of the following conditions occurs: (a) the randomness test ispassed; or (b) the randomness test is failed and in the update step (18), the batch count k 0 drops below thelower limit of 25 batches. If condition (a) occurs, then we proceed to the normality test in Section 4.3 withthe current values of S and k 0. On the other hand if condition (b) occurs, then the batch size m, the batchcount k, the overall sample size n, and the spacer size S are updated according to

m bffiffiffi2p

mc; k 256; n km and S 0;

respectively; the required additional observations are obtained (by restarting the simulation if necessary) tocomplete the overall sample {Xi : i = 1, . . . , n}; and then k adjacent (nonspaced) batch means are computedfrom the overall sample according to (13).

At this point we reperform the entire randomness-testing procedure, starting with the current set ofk = 256 adjacent (nonspaced) batch means of the current batch size m. If the randomness test is passed,then we set k 0 k and proceed to the normality test in Section 4.3; otherwise we repeat the steps outlinedin the two immediately preceding paragraphs. Once the randomness test is passed, we finalize the spacer sizeS so that we have a set of k 0 approximately i.i.d. spaced batch means, where 25 6 k 0 6 256. Moreover, thefirst S observations fX 1; . . . ;X Sg will henceforth be regarded as the warm-up period to be ignored in allsubsequent computations.


We use the following rationale for this approach to handling the simulation start-up problem. In general,the spaced batch means computed from observations comprising the warm-up period will exhibit a signif-icant deterministic trend or a significant degree of stochastic dependence on the simulation�s initial condi-tions. Once the simulation reaches steady-state operation, the corresponding spaced batch means will nolonger exhibit such a trend; and if the spacer size is sufficiently large so that the spaced batch means arealso approximately independent, then the resulting time series will appear to be nearly random in its behav-ior. If the spaced batch means exhibit a practically significant trend, then in our experience the von Neu-mann randomness test with a batch count in the range 25 6 k 0 6 256 has sufficient power to detect thetrend and reject the hypothesis (15), even if the effects of initialization bias are confined to the subseriesof observations consisting of the first spacer and the first batch so that initialization bias contaminates onlythe first observation in the corresponding series of spaced batch means. Each time the test for randomness isfailed, another batch is added to the spacer preceding each retained batch. Once the spaced batch meanspass the randomness test (15)–(17), two conclusions can be deduced.

• First, the observations fX 1; . . . ;X Sg comprising the first spacer can be regarded as the warm-up periodsince the spaced batch means following the first spacer do not exhibit a deterministic trend or any type ofstochastic dependence on the simulation�s initial conditions.

• Second, the spaced batch means computed beyond the warm-up period are randomly sampled from acommon distribution—that is, when they are computed from batches separated by spacers each consist-ing of S successive observations, the resulting spaced batch means are approximately i.i.d.

4.3. Testing batch means for normality

To test the spaced batch means for normality, we used the Shapiro–Wilk test (Shapiro and Wilk, 1965)because it is a well-established, powerful, and widely used test for departures from normality in a data setconsisting of i.i.d. observations (Royston, 1993, 1995). Thus in WASSP we apply the Shapiro–Wilk test tothe k 0 spaced batch means with the final spacer size S determined in the preceding test for randomness. Toassess the normality of the sample X 1ðmÞ; . . . ;X k0 ðmÞ

, we start by sorting the observations in ascending

order to obtain the order statistics X ð1ÞðmÞ 6 X ð2ÞðmÞ 6 � � � 6 X ðk0ÞðmÞ. The Shapiro–Wilk test statistic isthen computed as follows:

W ¼Pbk0=2c

‘¼1 dk0�‘þ1 X ðk0�‘þ1ÞðmÞ � X ð‘ÞðmÞ� �n o2

Pk0

‘¼1 X ‘ðmÞ � X ðm; kÞh i2

; ð19Þ

where the coefficients dk0�‘þ1 : ‘ ¼ 1; . . . ; bk0=2cf g are evaluated using the algorithm of Royston (1982). Thetest statistic W is then compared to the anor quantile wanor of the distribution of W under the null hypothesisof i.i.d. normal batch means,

Hnor : fX jðmÞ : j ¼ 1; . . . ; k0g �i:i:d:N lX ; r2X ðmÞ

h i. ð20Þ

If W 6 wanor , then at the level of significance anor we reject the hypothesis Hnor that the spaced batch meansX jðmÞ : j ¼ 1; . . . ; k0

are i.i.d. normal.For the first iteration of the normality test, the iteration counter is set to i 1 and the level of signif-

icance for the Shapiro–Wilk test is anor(1) 0.05. In general if on the ith iteration of the normality test (19)and (20) the hypothesis (20) is accepted at the level of significance anor(i) given by display (6), then we pro-ceed to estimate the SSVP as outlined in Section 4.4; otherwise, we repeat the following steps until the batchmeans finally pass the normality test (19) and (20):


1. The iteration counter i, the batch size m, and overall sample size are increased according to

TableLevel o

Iteratio

123456P7

i iþ 1; m ffiffiffi2p

mj k

and n k0ðS þ mÞ;

respectively; and the required additional observations are obtained (by restarting the simulation if nec-essary) to complete the overall sample {Xi : i = 1, . . . , n}.

2. The overall data set {X1, . . . , Xn} is reorganized into k 0 spaced batches of size m so that successivebatches are separated from each other by spacers each consisting of S observations.

3. The spaced batch means X jðmÞ : j ¼ 1; . . . ; k0

are recomputed.4. The level of significance anor(i) for the current iteration i of the Shapiro–Wilk test is reset according to (6).5. The k 0 spaced batch means X jðmÞ : j ¼ 1; . . . ; k0

are retested for normality at the level of significance

anor(i).

Remark 1. Table 1 lists the values of anor(i) for i = 1, . . . , 6. We set the value of the constant s = 0.184206 indisplay (6) so that on the first few iterations of the normality test, we use a monotonically decreasingsequence of nearly standard significance levels—specifically, about 5%, 2.5%, 1%, and 0.1% for i = 1, 3, 4,and 6, respectively; moreover for i = 7, 8, . . . , the significance level anor(i) should decrease by at least an orderof magnitude on each additional iteration of the normality test. We found through extensive preliminaryexperimentation that in terms of the confidence interval coverage and final sample size delivered by WASSP,this scheme works well for a wide variety of simulation-generated output processes while preventingexcessive variability of the final sample size in applications involving highly nonnormal data.

Remark 2. The basic idea of steps [1]–[6] of WASSP is to obtain a truncated series of adjacent batch meansthat constitute an approximately stationary Gaussian process by applying to the spaced batch means thevon Neumann test for randomness (with progressively increasing spacer sizes) and then the univariateShapiro–Wilk test for normality (with progressively increasing batch sizes and a fixed spacer size). By con-trast the NBM procedure ASAP (Steiger and Wilson, 2002) involves one the following options: (i) the vonNeumann test for randomness is applied to adjacent batch means as in LBATCH and ABATCH (Fishmanand Yarberry, 1997) so as to deliver the classical NBM-type confidence interval; or (ii) the multivariateShapiro–Wilk test for normality is applied to spaced four-dimensional vectors each consisting of adjacentbatch means so as to deliver a confidence interval that has been corrected for the remaining correlationbetween the truncated series of adjacent batch means. On the other hand in ASAP2 (Steiger et al., 2002)and ASAP3 (Steiger et al., 2005), option (i) is completely eliminated in favor of option (ii).

4.4. Estimating the SSVP via a wavelet-based spectral method

Once the normality test is passed, independence of the batch means is no longer required. Therefore, thefirst spacer consisting of the observations fX 1;X 2; . . . ;X Sg is skipped (to eliminate initialization bias), and

1f significance anor(i) for the ith iteration of the Shapiro–Wilk normality test (19) and (20)

n number i Significance level anor(i)

0.0500.0420.0240.00950.00260.0005<anor(i � 1)/10


the remaining n0 ¼ n� S observations are rebatched into k adjacent (nonspaced) batches of size m. To con-struct the wavelet-based estimate of the batch means log-spectrum in a neighborhood of zero frequency, wesee that the number of points in the log-periodogram (that is, the number of batch means k) must be apower of two as explained in Section 2.2. Therefore, we set k to be the largest power of two less than orequal to n 0/m so that we take k ¼ 2blog2ðn0=mÞc, where m is the final batch size determined by the normalitytest. For j = 1, . . . , k, we compute the jth adjacent (nonspaced) batch mean X jðmÞ. The next step in WASSPis to do the following: (a) smooth the periodogram of the batch means X 1ðmÞ; . . . ;X kðmÞ

by computing a

moving average with a sufficient number of points so as to obtain a reasonably stable estimator of the batchmeans power spectrum; (b) apply the logarithmic transformation to the smoothed periodogram of thebatch means so as to obtain a better-behaved (that is, less positively skewed) estimator of the batch meanslog-spectrum; and (c) correct for the bias introduced by the logarithmic transformation.

4.4.1. Computing the bias-corrected log-smoothed-periodogram of the batch meansThe periodogram of the batch means process is computed by taking the FFT of the adjacent (nonspaced)

batch means X ¼ X 1ðmÞ; . . . ;X kðmÞ

,

ðFXÞ‘ ¼Xk

j¼1

X jðmÞ exp �2pffiffiffiffiffiffiffi�1p� �

ðj� 1Þ‘=kh i

for ‘ ¼ 1; 2; . . . ; k � 1.

Since we will be interested in obtaining an estimate of the batch means log-spectrum in a neighborhood ofzero frequency using the values of the log-smoothed-periodogram of the batch means in that neighborhood,we will use a full set of points of the periodogram on both sides of zero frequency. Like the power spectrum,the periodogram is symmetric about the origin so that we have

IX ðmÞ‘

k

� �¼ IX ðmÞ �

‘

k

� �¼ 1

k

Xk

j¼1

X jðmÞ cos2pðj� 1Þ‘

k

� �" #2

þXk

j¼1

X jðmÞ sin2pðj� 1Þ‘

k

� �" #28<:

9=;¼ FX� �

‘

2.k for ‘ ¼ 1; 2; . . . ;k2� 1: ð21Þ

To compute the smoothed periodogram of the batch means based on a moving average of A = 2a + 1points, first we must determine appropriate values of IX ðmÞð‘kÞ for ‘ = 0 and for ‘ ¼ k

2; k

2þ 1; . . . ; k � 1. Using

the definition (21), we see that the value of the periodogram at ‘ = 0 is simply a scaled sum of the batchmeans and provides no information about the power spectrum of the batch means at zero frequency. Asan alternative, we take the value of the periodogram at ‘ = 0 as follows:

IX ðmÞð0Þ �1

a

Xa

u¼1

IX ðmÞuk

� �; ð22Þ

assuming a is sufficiently small relative to k so that the periodogram ordinates fIX ðmÞðukÞ : u ¼ �1; . . . ;�aghave expected values approximately equal to p X ðmÞð0Þ. Moreover, we assume that for ‘ 2 {1, 2, . . . , k/2} andfor a sufficiently small relative to k, the periodogram ordinates fIX ðmÞð‘þu

k Þ : u ¼ �1; . . . ;�ag have expectedvalues approximately equal to p X ðmÞð‘kÞ.

By the same reasoning that led to (22), we define the periodogram at frequency 12

as follows:

IX ðmÞ1

2

� �� 1

a

Xa

u¼1

IX ðmÞfk=2g � u

k

� �. ð23Þ

The sole purpose of the definition (23) is to facilitate wavelet-based estimation of the log-spectrum of thebatch means as described in Section 4.4.2 below.


For u ¼ 1; 2; . . . ; ðk2� 1Þ, we have

IX ðmÞfk=2g þ u

k

� �¼ IX ðmÞ

fk=2g � uk

� �; ð24Þ

and in view of (22)–(24), we see that for the frequency index set defined by (7) the smoothed periodogram of

the batch means eI X ðmÞð‘kÞ : ‘ 2 Gk

n ocan be computed as a moving average of A = 2a + 1 points,

eI X ðmÞ‘

k

� �¼ 1

A

Xa

u¼�a

IX ðmÞ‘þ u

k

� �for ‘ 2 Gk. ð25Þ

WASSP allows the user to select the value of the smoothing parameter A from the set {5, 7, 9, 11}. Weselected A = 7 as the default value. If A is set too high, then the log-smoothed-periodogram of the batchmeans will be oversmoothed, resulting in an estimator that is flatter in the neighborhood of zero frequencythan the true log-spectrum of the batch means process. If A is too small, then the log-smoothed-periodo-gram of the batch means will not be smoothed sufficiently, resulting in an excessively noisy estimator of thelog-spectrum of the batch means.

We found through extensive experimentation that setting the smoothing parameter to the default value ofA = 7 works well for a wide variety of types of simulation applications. Lada (2003) summarizes the effect ofdifferent values of A on the performance of WASSP as measured by confidence interval coverage, averagesample size, average confidence interval half-length, and the variance of the confidence interval half-length.The following test processes are used in this analysis, which is detailed in Sections 4.2–4.4 of Lada (2003):

1. an M/M/1 queue waiting time process for which the underlying system has i.i.d. exponential interarrivaltimes with mean 10/9, i.i.d. service times with mean 1, long-run server utilization equal to 0.90, and anempty-and-idle initial condition (as reported in Tables 4.1–4.2 of Lada, 2003);

2. the first-order autoregressive (AR(1)) process that has lag-one correlation equal to 0.995, white noisevariance equal to one, steady-state mean equal to 100, and initial condition equal to zero (as reportedin Tables 4.9–4.10 of Lada, 2003); and

3. the ‘‘AR(1)-to-Pareto’’ process that has marginals given by a Pareto distribution with lower limit andshape parameter equal to 1 and 2.1, respectively (implying the marginal mean and variance are bothfinite while the marginal skewness and kurtosis are both infinite), and that is obtained by applying tothe AR(1) process above the composite of the inverse of the specified Pareto c.d.f. and the standard nor-mal c.d.f. (as reported in Tables 4.17–4.18 of Lada, 2003).

These experimental results are also documented in Lada et al. (2004a,b). Since we always have k 6 4096batch means in WASSP and the procedure generally delivers a confidence interval for which k P 256 whenit is applied with a nontrivial precision requirement, the default value of A = 7 is consistent with the rec-ommendation of Chatfield (1984, Section 7.4.4) to use values of A in the region A � k/40 for spectrum esti-mation. (By design WASSP always yields k P 32; and in the absence of a precision requirement, it ispossible for WASSP to deliver a confidence interval based on k = 32 batch means.)

To provide additional justification for taking A = 7 as the default value of the smoothing parameter inWASSP, we start with the observation that at zero frequency the smoothed periodogram (25) has approx-imately a scaled chi-squared distribution with 2a degrees of freedom. This result is based on Eq. (42) inAppendix A of this article; and a more complete justification is given in Appendix A of Lada (2003) andin Section 1 of Lada and Wilson (2003). Therefore if A = 7, then we must take a = 3 so that we have2a = 6 degrees of freedom in the smoothed periodogram estimator (25) evaluated at zero frequency. Thusour approach to setting the default value of A is consistent with the recommendations of Heidelbergerand Welch (1981a), who find that their best-performing estimator of the spectrum at zero frequency has


approximately 7 degrees of freedom. All the foregoing considerations led us to conclude that selectingA = 7 as the default smoothing parameter in WASSP may provide an acceptable balance between (i) thestability of the final estimator of the SSVP of the original (unbatched) process as measured by the associ-ated degrees of freedom; and (ii) the ability of the batch means log-smoothed-periodogram to capture theimportant features of the underlying log-spectrum of the batch means.

Remark 3. The smoothed periodogram should not be computed using a moving average consisting of aneven number of points. If the smoothing parameter A is even, then the smoothed periodogram eI X ðmÞðxÞ willbe defined for frequencies of the form x = ± (2‘ + 1)/(2k), where the frequency index ‘ 2 {1, 2, . . . ,(k/2) � 2}. Unfortunately these frequencies are not equally spaced, specifically in the vicinity of zerofrequency; and incorporating the point at zero frequency based on a definition similar to (22) yields an oddnumber of sample values of the smoothed periodogram evaluated at frequencies that likewise are notequally spaced. Since ultimately we will be computing the discrete wavelet transform of the logarithm of thesmoothed periodogram of the batch means (as described in Section 4.4.2 below), we must evaluate thebatch means log-smoothed-periodogram at equally spaced frequencies, where the number of suchfrequencies is a power of two. These conditions can be satisfied only when the smoothing parameter A isodd.

To use the natural logarithm of the smoothed periodogram of the batch means,

eLX ðmÞ‘

k

� �� ln eI X ðmÞ

‘

k

� �� for ‘ 2 Gk; ð26Þ

as an estimator of the log-spectrum of the corresponding batch means process,

fX ðmÞðxÞ � ln p X ðmÞðxÞh i

for x 2 � 1

2;1

2

� �; ð27Þ

we perform an analysis similar to that given by Wahba and Wold (1975) and Wahba (1980). Upon applyingthe asymptotic properties of the periodogram in (2), we find that the expected value of our estimator (26)is

E ln eI X ðmÞ‘

k

� �� E ln

Xa

u¼�a

p X ðmÞ‘

k

� �wu

v2uð2Þ2

" #( )

¼ fX ðmÞ‘

k

� �þ E ln

1

2

Xa

u¼�a

wuv2uð2Þ

" #( ); ð28Þ

where fv2uð2Þ : u ¼ 0;�1; . . . ;�ag are i.i.d. chi-squared random variables each with two degrees of freedom

and the {wu : u = 0, ±1, . . . , ±a} are nonnegative deterministic weights such thatPa

u¼�awu ¼ 1. Therefore,when we take the logarithmic transformation of the smoothed periodogram of the batch means, we intro-duce bias represented by the second term on the right-hand side of (28); and this bias must be removed toensure the accuracy of our wavelet-based estimator of the batch means log-spectrum.

Table 2 displays general expressions for the bias g‘ ¼ E eLX ðmÞð‘kÞh i

� fX ðmÞð‘kÞ at each frequency ‘k for

‘ 2 Gk. Table 3 shows the numerical values of the bias terms in Table 2 for smoothing parameter valuesA 2 {5, 7, 9, 11} and for frequencies of the form ‘

k, where ‘ 2 Gk. For example if 1 6 ‘ 6 a, then the batchmeans smoothed periodogram estimator eI X ðmÞð‘kÞ has approximately a scaled chi-squared distribution with‘‘effective’’ degrees of freedom given by

m‘ ¼ bm#‘ c; where m#

‘ ¼2aA2

4a2 � 2a‘þ 4a� 2‘þ 1. ð29Þ

Table 3Bias g‘ for smoothing parameter A = 5, 7, 9, and 11 and for frequency ‘

k, where ‘ 2 Gk

‘ A = 5 A = 7 A = 9 A = 11

0 �0.2704 �0.1758 �0.1302 �0.1033±1 �0.2131 �0.1496 �0.1152 �0.0937±2 �0.1496 �0.1302 �0.1033 �0.0856±3 �0.0937 �0.0856 �0.0731±4 �0.0681 �0.0638±5 �0.0536a < j‘j < k

2� a �0.1033 �0.0731 �0.0566 �0.0461

�ðk2� 5Þ �0.0536

�ðk2� 4Þ �0.0681 �0.0638

�ðk2� 3Þ �0.0937 �0.0856 �0.0731

�ðk2� 2Þ �0.1496 �0.1302 �0.1033 �0.0856

�ðk2� 1Þ �0.2131 �0.1496 �0.1152 �0.0937k2 �0.2704 �0.1758 �0.1302 �0.1033

Table 2Bias g‘ of the log-smoothed-periodogram at frequency ‘

k, where ‘ 2 Gk

Frequency index ‘ Bias g‘

‘ ¼ 0; k2 WðaÞ � lnðaÞ

1 6 j‘j 6 a Wðmj‘j=2Þ � lnðmj‘j=2Þa < j‘j < k

2� a WðAÞ � lnðAÞk2� a 6 j‘j 6 k

2� 1 Wðmk=2�j‘j=2Þ � lnðmk=2�j‘j=2Þ


We obtain (29) by computing the reciprocal of half the squared coefficient of variation of (25) for 1 6 ‘ 6 a;and then we apply the approach summarized in Appendix A of this article to obtain the corresponding biasg‘ that appears in Table 2. A similar analysis yields the expression for the bias g‘ in the case thatk2� a 6 j‘j 6 k

2� 1. See Appendix A of Lada (2003) or Section 1 of Lada and Wilson (2003) for complete

derivations of all the bias terms in Table 2. Note that in Table 2, the digamma function W(z) is defined interms of the gamma function C(z) as follows:

WðzÞ � d

dzln CðzÞ½ � ¼ C0ðzÞ

CðzÞ for all z with ReðzÞ > 0; ð30Þ

see Gradshteyn and Ryzhik (2000).An alternative estimator of the batch means log-spectrum fX ðmÞð‘kÞ may be obtained by computing first

the batch means log-periodogram, ln IX ðmÞð‘kÞh i

; then smoothing the result by taking a moving average of

log-periodogram values; and finally taking the bias-corrected smoothed-log-periodogram of the batchmeans,

�LX ðmÞ‘

k

� �� 1

A

Xa

u¼�a

ln IX ðmÞ‘þ u

k

� �� Wð1Þ for ‘ 2 Gk. ð31Þ

(For comparison with the entries in Table 3, note that W(1) = �0.5772; see Gradshteyn and Ryzhik, 2000.)In Appendix B of this article, we explain briefly the approach used to derive expressions for the variances ofthe estimators (26) and (31) of the batch means spectrum fX ðmÞð‘kÞ for ‘ 2 Gk; and the results are displayed in

Table 4Comparison of the variances of the estimators eLX ðmÞð‘kÞ and LX ðmÞð‘kÞ of the log-spectrum of the batch means at frequency ‘

k, where‘ 2 Gk

Frequency index ‘ Var½eLX ðmÞð‘kÞ� Var½LX ðmÞð‘kÞ�‘ ¼ 0; k

2 W 0(a) W0 ð1Þa

1 6 j‘j 6 a W 0(mj‘j/2) W0 ð1Þm#

j‘j=2

a < j‘j < k2� a W 0(A) W0 ð1Þ

A

k2� a 6 j‘j 6 k

2� 1 W 0(mk/2�j‘j/2) W0 ð1Þm#

k=2�j‘j=2

Table 5Comparison of the variance of eLX ðmÞð0Þ, the bias-corrected log-smoothed-periodogram of the batch means at zero frequency, to thevariance of LX ðmÞð0Þ, the bias-corrected smoothed-log-periodogram of the batch means at zero frequency

Smoothing parameter Var½eLX ðmÞð0Þ� Var½LX ðmÞð0Þ�A = 5 (a = 2) 0.6449 0.8225A = 7 (a = 3) 0.3949 0.5483A = 9 (a = 4) 0.2838 0.4112A = 11 (a = 5) 0.2213 0.3290


Table 4. Complete derivations of these results are given in Appendix B of Lada (2003) and in Section 2 ofLada and Wilson (2003).

In Table 5, we show the numerical values of Var eLX ðmÞð0Þh i

and Var LX ðmÞð0Þh i

for A = 5, 7, 9, and 11.

From this table we see that at zero frequency, smoothing the periodogram of the batch means and thentaking the bias-corrected logarithm results in an estimator of fX ðmÞð0Þ with smaller variance than that ofthe estimator obtained by computing the bias-corrected logarithm of the batch means periodogram andthen smoothing by taking a moving average of bias-corrected log-periodogram values.

In Appendix B of Lada (2003) and in Section 2 of Lada and Wilson (2003), we prove the following gen-eral variance reduction results:

W0ðjÞ < W0ð1Þj

for j ¼ 2; 3; . . . ; and W0bxc2

� �<

W0ð1Þx=2

for all real x P 3. ð32Þ

Applying (32) to the variances in Table 4, we conclude that for all frequencies ‘k where ‘ 2 Gk and for any

value of the smoothing parameter A, smoothing the periodogram of the batch means and then taking thebias-corrected logarithm results in an estimator of fX ðmÞð‘kÞ with smaller variance than that of the estimatorobtained by smoothing the bias-corrected log-periodogram of the batch means.

4.4.2. Using wavelets to estimate the spectrum of the batch means

The next step in WASSP is to expand eLX ðmÞð‘kÞ � g‘ : ‘ 2 Gk

n o, the bias-corrected log-smoothed-period-

ogram of the batch means, as a wavelet series so as to obtain a wavelet-based estimate of the batch meanslog-spectrum (27). To apply Mallat�s algorithm for computing the DWT of this data set, we include theendpoint at frequency 1

2to ensure that the data set size has the form k = 2J for some J as explained in Sec-

tion 2.2, Remark 3, and Eq. (23) above. Thus we obtain the DWT

fW ¼ H eL; ð33Þ


where the vector eL has elements eLX ðmÞð‘kÞ � g‘ : ‘ 2 Gk

n o; and the k · k orthogonal matrix H defines the

DWT associated with the s8 symmlet (Bruce and Gao, 1996). The s8 symmlet is an excellent overall choicefor representing many functions since it is orthogonal, smooth, nearly symmetric, and nonzero on a rela-tively short interval.

In practice we have found that a good approximation to eLX ðmÞð‘kÞ � g‘ : ‘ 2 Gk

n ocan be obtained by

taking J � log2ðkÞ and setting the number of resolution levels L in the wavelet estimator as follows:

TableNumbj0, the

k

32641282565124096

L � J=2b c ¼ log2ðkÞ=2b c; ð34Þ
and the coarsest level of resolution is then given by j0 = J � L. This will yield a total of 2j0 scaling coeffi-cients and 2j0 wavelet coefficients at the coarsest level of resolution. Table 6 lists the values of L and j0, aswell as the total number of wavelet coefficients at level j0, for various values of k.
We have found that (34) represents an effective compromise between two extremes:

1. setting L = 1 and j0 = J � 1, so that we have the minimum number of resolution levels and the maxi-mum number of scaling coefficients; and

2. setting L = J and j0 = 0 so that we have the maximum number of resolution levels and the minimumnumber of scaling coefficients.

Since the scaling coefficients typically contain information about important features of the underlying batchmeans log-spectrum and consequently are not usually thresholded, option 1 yields k/2 scaling coefficients,often with substantial noise that cannot be removed. On the other hand, while option 2 ensures that thesingle scaling coefficient contains minimal noise, it is likely that important features of the batch meanslog-spectrum will be lost after thresholding the wavelet coefficients. In our experience, using (34) to setthe number of resolution levels in the wavelet analysis yields a sufficiently accurate representation of theprincipal features of the bias-corrected log-smoothed-periodogram of the batch means while allowingour soft-thresholding scheme (see (35) below) to effectively remove noise from the wavelet coefficient esti-mators. For a more detailed justification of (34), see Lada (2003, pp. 75–79).

After computing the DWT fW of the bias-corrected log-smoothed periodogram of the batch means as

given by (33), we threshold the resulting wavelet coefficients bd j;‘ : j ¼ j0; . . . ; J � 1; ‘ ¼ 0; . . . ; 2j � 1n o

to

remove the remaining noise inherent in these quantities. There are many wavelet thresholding proceduresin the literature that are based on the idea of selecting ‘‘important’’ wavelet coefficients and setting to zerothe ‘‘unimportant’’ coefficients. These methods attempt to find an optimal number of coefficients to accu-rately represent the data, thereby leading to a simplified and smoother (less noisy) data model. When weestimate a function from a set of noisy data using the discrete wavelet transform, each coefficient obtainedfrom the DWT will contain noise. By shrinking the small coefficients to zero, we can remove this noise whileat the same time preserving the original features of the function.

6er of levels of resolution L obtained by computing the DWT of a data set of size k; also shown is the coarsest level of resolutionrange of values for the resolution level j, and the number of wavelet coefficients at the coarsest level j0

J L j0 j # Wavelet coefficients at level j0

5 2 3 3, 4 86 3 3 3–5 87 3 4 4–6 168 4 4 4–7 169 4 5 5–8 32

12 6 6 6–11 64


When we use wavelets to estimate the batch means log-spectrum, we represent the bias-corrected log-smoothed-periodogram of the batch means as a signal plus noise, where the signal is the true batch meanslog-spectrum. After computing the DWT fW , at each level of resolution j we apply the thresholding schemeof Gao (1997) with the soft threshold

kj ¼ maxpffiffiffiffiffi6kp

ffiffiffiffiffiffiffiffiffiffiffiffiffiffi2 lnðkÞ

p; 2�ðJ�j�1Þ=4 lnð2kÞ

� �for j ¼ j0; . . . ; J � 1; ð35Þ

to obtain the thresholded wavelet coefficients fbd j;‘g, where
bd j;‘ ¼ sgn bd j;‘
� �max 0; jbd j;‘j � kj

n ofor j ¼ j0; . . . ; J � 1 and ‘ ¼ 0; . . . ; 2j � 1. ð36Þ

The estimated scaling coefficients bcj0;‘ : ‘ ¼ 0; . . . ; 2j0 � 1

are not thresholded since it is presumed theycontain information about the coarse features of the log-spectrum of the batch means. We used Gao�ssoft-thresholding scheme (35) and (36) in WASSP for three reasons:

1. Gao�s soft-thresholding scheme has been implemented in the S+WAVELETS toolkit, which is available as amodule of the widely used S-PLUS software environment (Bruce and Gao, 1996).

2. With respect to the accuracy criterion of mean squared error, both hard- and soft-thresholding schemesexhibit the same asymptotic performance (Donoho and Johnstone, 1994, Theorems 2 and 4); howeverunder fairly general conditions, soft-thresholding schemes yield function approximations with superiorsmoothness properties (Donoho and Johnstone, 1995, Section 5.3; Donoho, 1995).

3. In our computational experience working with many different types of simulation-generated output pro-cesses, Gao�s soft-thresholding scheme has consistently delivered accurate, parsimonious estimators ofthe log-spectrum of the associated batch means process.

The next step is to compute the inverse DWT,
eL ¼ HTfW ; ð37Þ
where fW is the k · 1 vector containing the scaling coefficients bcj0;‘ : ‘ ¼ 0; . . . ; 2j0 � 1

and thresholded

wavelet coefficients bd j;‘ : j ¼ j0; . . . ; J � 1; ‘ ¼ 0; . . . ; 2j � 1n o

. Thus (37) yields the thresholded waveletapproximation

eL ¼ eL1; . . . ; eLkh iT

¼ eLX ðmÞ � fk=2g � 1

k

� �; . . . ; eLX ðmÞ 1

2

� �� T

to the k · 1 vector eL representing the bias-corrected log-smoothed-periodogram of the batch means.Therefore, our wavelet-based estimator of the log-spectrum of the batch means is

bfX ðmÞ‘

k

� �¼ eLX ðmÞ ‘

k

� �for ‘ 2 Gk. ð38Þ

The wavelet-based estimator of the spectrum of the batch means process is computed from (38) as

bpX ðmÞ‘

k

� �¼ exp bfX ðmÞ

‘

k

� �� for ‘ 2 Gk; ð39Þ

and a wavelet-based estimator of the SSVP for the original (unbatched) process is recovered from (39) asfollows:
bcX ¼ m � bpX ðmÞð0Þ. ð40Þ


Although a justification for (40) can be based on display (29) of Heidelberger and Welch (1981a), a simpleranalysis is given in Lada (2003, pp. 85–86). An approximate 100(1 � b)% confidence interval for lX is thengiven by

Fig. 3.estimaintensi

X ðm; kÞ � t1�b=2;2a


; ð41Þ

where n 0 = mk and X ðm; kÞ is the grand average of the k batch means X 1ðmÞ; . . . ;X kðmÞ

. It should benoted that several other wavelet-based techniques for estimating the spectrum have been developed in addi-tion to the method of Gao (1997). The estimator (39) of the batch means spectrum is also built on some ofthe basic ideas developed by Moulin (1994) and by Walden et al. (1998).

Fig. 3 shows plots of the bias-corrected log-smoothed-periodogram of the batch means and the corre-sponding wavelet-based estimate of the batch means log-spectrum for the waiting times in an M/M/1queueing system with 90% traffic intensity and an empty-and-idle initial condition. Fig. 3 was generatedby first collecting n = 32,768 waiting time observations from a single run of the M/M/1 simulation. Therandomness test and the normality test were performed to obtain a batch means process

X 1ðmÞ; . . . ;X kðmÞ

that is approximately stationary and Gaussian, where we have k = 64. Finally, wecomputed the bias-corrected log-smoothed-periodogram of the batch means and the correspondingwavelet-based estimate of the batch means log-spectrum. From these plots, it is clear that applying thethresholded DWT successfully removed the remaining noise from the bias-corrected log-smoothedperiodogram.

4.5. Fulfilling the precision requirement

The half-length of the confidence interval (41) is given by

H ¼ t1�b=2;2a


¼ t1�b=2;2a

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffibpX ðmÞð0Þ=kq

.

If the confidence interval (41) satisfies the precision requirement H 6 H* of (10), where H* is given by (11),then WASSP terminates, delivering the confidence interval (41).

–0.5 –0.4 –0.3 –0.2 –0.1 0 0.1 0.2 0.3 0.4 0.52

3

4

5

6

7

–0.5 –0.4 –0.3 –0.2 –0.1 0 0.1 0.2 0.3 0.4 0.52

3

4

5

6

7

The bias-corrected smoothed log-periodogram of the batch means (top panel) and the corresponding thresholded wavelette (bottom panel) for k = 64 batch means computed from the waiting times for an M/M/1 queueing system with 90% trafficty.


If the precision requirement (10) is not satisfied, then we estimate the total number of batches of the cur-rent batch size that are needed to satisfy the precision requirement, k* d(H/H*)2ke; and thus k*m is ourlatest estimate of the total sample size needed to satisfy the precision requirement. However, since the num-ber of batches must be a power of two, the batch count k is set for the next iteration of WASSP as follows:k minf2blog2ðkÞc; 4096g, where 4096 is the upper bound on the number of batch means used in WASSP.Then the new batch size m for the next iteration of WASSP is assigned according to m d(k*/k)me, so thatthe total sample size n is increased approximately by the factor (H/H*)2.

On the next iteration of WASSP, the total sample size including the warm-up period is thus given byn S þ km, where the corresponding batch count k and batch size m are given by (12), and the spacer sizeS was finalized in the randomness test. The additional simulation-generated observations are obtained byrestarting the simulation or by retrieving the extra data from storage; and then the next iteration of WASSPis performed.

4.6. Computational complexity of WASSP

The most computationally intensive portion of WASSP is the batching procedure; and for a sample ofsize n, this procedure runs in O(n) time and requires O(n) memory because data are passed to WASSP viaan array. In the algorithmic statement of WASSP displayed in Fig. 2, we see that on each iteration of sub-steps [2.1]–[2.5] with a given value of the batch size m and a new value of the spacer size S, the k 0 spacedbatch means are reassigned (reindexed) rather than being recomputed; and this reassignment procedureruns in O(k 0) = O(1) time since k 0 is always kept in the range 25 6 k 0 6 256. On each occasion that therandomness test is failed with k 0 = 25 so that both m and n = km = 256m are increased by the factorffiffiffi

2p

, step [3] is performed in O(n) time.Each iteration of the normality test in step [4] runs in O(k 0) = O(1) time. On each occasion that the nor-

mality test is failed so that m and n = k 0(S + m) are increased by the factorffiffiffi2p

, step [5] is performed inO(n) = O(m) time.

Each iteration of step [6] is immediately preceded by step [4] or step [12]. If the normality test in step [4] ispassed with k 0 batch means so that n = k 0(S + m) with 25 6 k 0 6 256, then we see that the immediatelyfollowing step [6] runs in O(n) = O(m) time. On the other hand if the user-specified absolute or relative pre-cision stopping rule (10)–(11) in substep [12.1] is not satisfied so that k and m must be recomputed in sub-step [12.3] to yield k 6 4096 and n = S + km, then we see that the immediately following step [6] runs inO(n) = O(m) time.

Because in steps [7] and [8] the number of batch means k is always in the range 25 6 k 6 4096 andthe FFT of the batch means runs in Oðk log kÞ ¼ Oð1Þ time (Percival and Walden, 2000), we see thatcomputing the bias-corrected log-smoothed-periodogram of the batch means runs in O(1) time. FromSection 4.2 of Percival and Walden (2000), we have that the DWT and inverse DWT of the bias-cor-rected log-smoothed-periodogram of the batch means run in O(k) = O(1) time; and thus steps [9] and[10] run in O(1) time. Furthermore, computing the confidence interval (9) in step [11] from an overallsample of size n = S + km runs in O(n) = O(m) time. Finally we observe that step [12] runs in O(1)time.

We applied WASSP to a total of five test processes as described in Lada (2003). By running each testprocess at three or four different levels of precision, two levels of confidence interval coverage, and a varietyof values of the smoothing parameter, we had the opportunity to observe the performance of WASSP in70,400 applications. In the worst case—that is, in the test problem requiring the largest sample sizes—the average sample size for the no precision case was 111,874; and this implies that when WASSP wasapplied to the given test process without a precision requirement, WASSP required about 10 iterationsinvolving 256 batches of up to an average batch size of 437 observations before both the randomnessand normality tests yielded a nonsignificant result.


Once the randomness and normality tests were passed, WASSP required only one additional iteration tosatisfy the precision requirement in most cases. Therefore, the number of batching operations, and hencethe computation time, were not excessive even in the most difficult cases. Furthermore, other batchingschemes require similar time to execute. For instance, after applying ASAP3 to the same test process thatrequired the largest sample sizes for WASSP, we found that the average sample size for ASAP3 was 113,336in the no precision case; and this implies that when ASAP3 was applied to the given test process without aprecision requirement, ASAP3 required about 10 iterations involving 256 batches of up to an average batchsize of 442 observations. The other operations in the WASSP algorithm require negligible computer time.

5. Performance evaluation of WASSP

Lada (2003) and Lada et al. (2004b) detail an extensive performance evaluation of WASSP. In this sec-tion we summarize the results of applying WASSP to the M/M/1 queue waiting time process. Here Xi is thewaiting time for the ith customer in a single-server queueing system with i.i.d. exponential interarrival timeshaving mean 10/9, i.i.d. exponential service times having mean 1, a steady-state server utilization of 90%,and an empty-and-idle initial condition. The steady-state mean waiting time is lX = 9.0.

The M/M/1 queue waiting time process with 90% server utilization and empty-and-idle initial conditionis a particularly difficult test process for the following reasons: (a) the initial transient is pronounced andpersists over an extended period of time; (b) the correlation function decays slowly with increasing lags oncethe system has reached steady-state operation; (c) the marginal distribution of waiting times is markedlynonnormal; and (d) the spectrum of the batch means process, p X ðmÞðxÞ, is sharply peaked in the neighbor-hood of zero frequency. This test process enabled us to perform a thorough evaluation of the robustness ofWASSP�s procedure for eliminating initialization bias as well as the robustness of WASSP�s wavelet-basedtechnique for estimating the SSVP of the original waiting time process {Xi}.

We used the following figures of merit to evaluate the performance of WASSP and its competitors: thecoverage probability of the delivered confidence intervals (CIs); the mean and variance of the half-lengthsof those CIs; and the mean and maximum of the required sample sizes. We performed 1000 independentreplications of WASSP to construct nominal 90% and 95% CIs that satisfied a given precision requirement.The following three precision requirements were used:

• no precision—that is, we set h* =1 in (10) and (11) so WASSP delivered the CI (9) using the batchcount and batch size required to pass the randomness and normality tests;

• ±15% precision—that is, WASSP delivered the CI (9) satisfying the relative precision requirement givenby (10) and (11) with r* = 0.15; and

• ±7.5% precision—that is, WASSP delivered the CI (9) satisfying the relative precision requirement givenby (10) and (11) with r* = 0.075.

For the sake of comparison, we also applied the following steady-state simulation analysis procedures tothe same M/M/1 queue waiting time process:

• the ASAP3 algorithm of Steiger et al. (2005); and• the extended spectral method of Heidelberger and Welch (1983), which incorporates a method for detect-

ing and eliminating initialization bias.

To make a fair comparison of the performance of WASSP with that of the extended spectral method ofHeidelberger and Welch (H&W), we first applied WASSP to a given realization of this process so as to obtainnot only the corresponding WASSP-generated CI but also a complete data set to which we could apply the


H&W procedure. In particular on each replication of WASSP and the H&W procedure, we set the maximumrun length tmax for the H&W method equal to the final sample size required by WASSP for that replication.Tables 7 and 8 summarize the results we obtained from our experiments with the M/M/1 queue.

Based on all our computational experience with WASSP, we believe that the results given in Tables 7 and8 are typical of the performance of WASSP in many practical applications. Since each CI with a nominalcoverage probability of 90% was replicated 1000 times for the H&W method as well as for WASSP, thestandard error of each coverage estimator is approximately 0.95%. The coverage estimators for ASAP3�snominal 90% CIs have a standard error of approximately 1.5% since only 400 replications of ASAP3 wereperformed. Because of the extensive disk space requirements for the version of the ASAP3 software thatwas used to carry out the experimental performance evaluation, it was not practical to generate 1000 inde-pendent replications of ASAP3 (N.M. Steiger, personal communication). For nominal 95% coverage prob-ability, the standard errors of the coverage estimators for WASSP, the H&W method, and ASAP3 are allapproximately equal to 1%. As explained below, these levels of accuracy in the estimated coverage proba-bilities turn out to be sufficient to reveal significant differences in the performance of WASSP comparedwith that of the H&W procedure and ASAP3 in the given test problem.

As can be seen from Tables 7 and 8, WASSP outperformed the H&W method with respect to CI cov-erage for all three precision requirements. Furthermore, since the H&W method terminates once tmax isreached, it is possible that the H&W algorithm could run out of data before the precision requirement issatisfied. Of the 1000 nominal 90% CIs delivered by the H&W method, only 939 CIs actually satisfiedthe precision requirement for the ±15% precision level; and only 918 of the CIs delivered by the H&Wmethod satisfied the precision requirement for the ±7.5% precision level.

From Tables 7 and 8 it is also evident that in the no precision case, WASSP and ASAP3 delivered similarresults in terms of CI coverage. However WASSP required nearly half as many observations as ASAP3 inthe no precision case. For the case of ±7.5% precision, WASSP and ASAP3 performed essentially the same,

Table 7Performance of WASSP (using A = 7), Heidelberger and Welch�s spectral method (H&W), and ASAP3 for the M/M/1 queue waitingtime process with 90% server utilization and empty-and-idle initial condition; results are based on independent replications of nominal90% CIs

Precision requirement Performance measure Procedure

WASSP H&W ASAP3

None # Replications 1000 1000 400CI coverage (%) 87.7 67.8 87.5Avg. sample size 18,090 2714 31,181Max. sample size 241,152 36,173 185,344Avg. CI half-length 3.0715 4.0535 2.0719Var. CI half-length 2.0026 4.4582 0.3478

±15% # Replications 1000 1000 400CI coverage (%) 87.2 81.3 91Avg. sample size 92,049 62,112 103,742Max. sample size 688,256 348,434 424,536Avg. CI half-length 1.1103 1.1486 1.1820Var. CI half-length 0.0387 0.0406 0.0259

±7.5% # Replications 1000 1000 400CI coverage (%) 90.4 85 89.5Avg. sample size 388,000 275,610 287,568Max. sample size 2,614,458 1,323,572 700,700Avg. CI half-length 0.5866 0.5899 0.6273Var. CI half-length 0.0072 0.0072 0.0023

Table 8Performance of WASSP (using A = 7), Heidelberger and Welch�s spectral method (H&W), and ASAP3 for the M/M/1 queue waitingtime process with 90% server utilization and empty-and-idle initial condition; results are based on independent replications of nominal95% CIs

Precision requirement Performance measure Procedure

WASSP H&W ASAP3

None # Replications 1000 1000 400CI coverage (%) 93.4 76.2 91.5Avg. sample size 17,971 2696 31,181Max. sample size 171,456 25,719 185,344Avg. CI half-length 3.9987 5.1817 2.5209Var. CI half-length 3.6999 7.9996 0.5350

±15% # Replications 1000 1000 400CI coverage (%) 93 88.6 95.5Avg. sample size 143,920 98,838 140,052Max. sample size 953,424 482,673 418,263Avg. CI half-length 1.1342 1.1550 1.2059Var. CI half-length 0.0314 0.0347 0.0205

±7.5% # Replications 1000 1000 400CI coverage (%) 97 91.8 94Avg. sample size 598,020 431,590 382,958Max. sample size 3,408,016 1,517,616 956,610Avg. CI half-length 0.5950 0.5903 0.6324Var. CI half-length 0.0056 0.0078 0.0020


suggesting that as the relative precision requirement goes to zero, WASSP and ASAP3 will produce com-parable results for this test process in terms of coverage probability, average CI half-length, and variance ofthe CI half-length.

6. Conclusions and recommendations for future work

In this article we proposed a wavelet-based spectral procedure for constructing an approximate confi-dence interval for the steady-state mean of a simulation output process. This procedure, called WASSP,addresses two fundamental problems associated with analyzing stochastic output from a nonterminatingsimulation, namely, the start-up and correlation problems.

One of the main advantages of a spectral approach to steady-state output analysis is that it enables us towork with approximately independent periodogram values rather than with a highly correlated outputsequence. WASSP uses wavelets to estimate the log-smoothed-periodogram of the associated batch meansprocess, from which we obtain an estimator of the steady-state variance parameter of the original (un-batched) process. Together with a sample mean that has been suitably truncated to eliminate initializationbias, the estimator of the steady-state variance parameter is used to construct a confidence interval for thesteady-state mean response that satisfies a user-specified absolute or relative precision requirement and thathas approximately the user-specified probability of covering the steady-state mean.

There are several key differences between WASSP and previous spectral methods for steady-state simu-lation output analysis—in particular, the spectral method of Heidelberger and Welch (H&W). First, tosmooth the periodogram, the H&W method requires averaging nonoverlapping pairs of adjacent perio-dogram values. In contrast to this approach, WASSP allows the user to select a moving average of width5, 7, 9, or 11 points on the periodogram of the batch means, with the default taken to be a 7-point moving


average. The main advantage of using a larger number of points in the moving average is that a more stable(less noisy) estimator of the batch means spectrum can be obtained. On the other hand, in a highly depen-dent process whose spectrum has a sharp peak at zero frequency, it may be advantageous to select a smallernumber of points in the moving average so as to obtain a less biased estimator of the batch means spectrumat zero frequency, and hence a less biased estimator of the steady-state variance parameter of the original(unbatched) process. In both WASSP and the H&W procedure, the half-length of the final confidence-inter-val estimator of the mean is proportional to the square root of the final estimator of the steady-state var-iance parameter of the original (unbatched) process.

A second key difference between WASSP and the H&W method is that the latter procedure uses stan-dard least-squares regression techniques to approximate the batch means log-spectrum with a quadraticpolynomial that has been fitted to the log-smoothed-periodogram in a small neighborhood to the rightof zero frequency that does not include the latter point; and then the fitted quadratic polynomial is extrap-olated to zero frequency. In contrast to this approach, WASSP uses wavelets to approximate the batchmeans log-spectrum on the entire interval ½� 1

2; 1

2�, which includes proper neighborhoods of zero frequency.

The main premise behind using wavelets is that this approach can yield a more flexible and accurate esti-mator of the batch means spectrum than standard regression techniques can provide, not only in a smallneighborhood of zero frequency but also over the full frequency range from � 1

2to 1

2cycles per unit of time.

A third key difference between WASSP and the H&W method concerns their techniques for detectingand eliminating initialization bias. Although the original H&W method described in Heidelberger andWelch (1981a) does not include a technique for handling initialization effects, the extended H&W methoddescribed in Heidelberger and Welch (1983) incorporates a technique for detecting and eliminating initial-ization effects based on the simulation analysis method of standardized time series. On the other hand,WASSP uses the von Neumann test for randomness to determine an initial spacer size (and hence adata-truncation point) sufficiently large to ensure that the spaced batch means are approximately i.i.d.(and hence are approximately free of initialization effects). The method of standardized time series is knownto require large sample sizes to work properly in some applications; see page 535 of Law and Kelton (2000).In contrast, the von Neumann test for randomness is known to have adequate power for detecting the pres-ence of practically significant trends even in relatively small samples; see Fishman (1978), Fishman andYarberry (1997), and Young (1941).

The developments presented in this article provide some evidence that WASSP advances the methodol-ogy of spectral methods for simulation output analysis. From the experimental results detailed in Lada(2003) and Lada et al. (2004a,b), we also concluded that WASSP outperformed the H&W spectral methodin the given test problems. To provide more comprehensive evidence of WASSP�s performance in practice,we are currently applying WASSP to a suite of communication-, inventory-, queueing-, and production-sys-tem simulations that are typical of a broad class of steady-state simulation applications.

In the future, it will be necessary to establish key asymptotic properties of the confidence intervals deliv-ered by WASSP. First we need to determine if there is a nontrivial class of discrete-event stochastic systemsfor which WASSP�s confidence intervals are asymptotically valid—that is, the confidence intervals deliveredby WASSP have coverage probabilities equal to (or no less than) the user-specified nominal levels—as theuser�s absolute or relative precision specification tends to zero. We also need to resolve the question ofWASSP�s asymptotic efficiency in the sense of Chow and Robbins (1965) and Nadas (1969).

Another direction of future research is to modify WASSP so that the value of the smoothing parameterA is automatically determined within the procedure based on the observed characteristics of the target out-put process as well as the user�s specification of a confidence coefficient and a precision requirement for thefinal confidence interval. Furthermore, we would also like to determine an appropriate method for elimi-nating the bias that is introduced when in step [10] of WASSP we exponentiate bfX ðmÞð0Þ, the wavelet-basedestimator of the batch means log-spectrum at zero frequency, to obtain our estimator of the steady-state variance parameter. If we could formulate a method for estimating Var½bfX ðmÞð0Þ�, then we believe a


more accurate estimator of the steady-state variance parameter might be based on the ‘‘MLE-delta’’ meth-od of Irizarry et al. (2003).

Additional theoretical and experimental results, follow-up papers, and revised software for WASSP willbe available on the Web site www.ie.ncsu.edu/jwilson.

Acknowledgments

The authors thank Stephen D. Roberts and Charles E. Smith (North Carolina State University); NatalieM. Steiger (University of Maine); and David Goldsman and Brani Vidakovic (Georgia Institute of Tech-nology) for many enlightening discussions on this article. Partially on the basis of work documented in thisarticle, the first author won the 2004 Pritsker Doctoral Dissertation Award (First Place) from the Instituteof Industrial Engineers. This research was partially supported by NSF grant DMI-9900164 and by theAmerican Association of University Women (AAUW) through an AAUW Educational Foundation Engi-neering Dissertation Fellowship. Additional support was provided by the Statistical and Applied Mathe-matical Sciences Institute and the Center for Research in Scientific Computation at North CarolinaState University.

Appendices

In Appendices A and B, we compute the mean and variance of eLX ðmÞð‘kÞ, the log-smoothed-periodogramof the batch means at frequency ‘

k, for ‘ 2 Gk so as to obtain a bias adjustment to eLX ðmÞð‘kÞ that will yield anunbiased estimator of fX ðmÞð‘kÞ, the batch means log-spectrum. Throughout the appendices,fv2

uð2Þ : u ¼ 1; 2; . . .g denotes a set of i.i.d. chi-squared variates, each with 2 degrees of freedom.

Appendix A. Bias adjustment in estimating the batch means log-spectrum

A.1. Bias adjustment at frequency ‘k for ‘ = 0, ‘ ¼ k

2, and a < j‘j < k

2� a

For ‘ = 0, smoothing parameter A = 2a + 1, and IX ðmÞð0Þ as defined in Eq. (22), we see from (25) that

eI X ðmÞð0Þ ¼1

A

Xa

u¼�a

IX ðmÞuk

� �¼ 1

a

Xa

u¼1

IX ðmÞuk

� �� 1

a

Xa

u¼1

p X ðmÞð0Þv2

uð2Þ2� p X ðmÞð0Þ

v2ð2aÞ2a

; ð42Þ

since in general the sum of n i.i.d. chi-squared random variables each with m degrees of freedom is a chi-squared random variable with nm degrees of freedom. Taking the batch means log-smoothed-periodogramat zero frequency, we have

eLX ðmÞð0Þ ¼ ln½eI X ðmÞð0Þ� ��

ln½p X ðmÞð0Þ� þ lnv2ð2aÞ

2a

� �ð43Þ

so that

E½eLX ðmÞð0Þ� � fX ðmÞð0Þ þ E lnv2ð2aÞ

2a

� �� . ð44Þ

Thus Efln½v2ð2aÞ=ð2aÞ�g is the bias introduced at frequency zero by taking the logarithm of eI X ðmÞð0Þ. If arandom variable has the form B ¼ ln½v2ðmÞ=m�, then its moment generating function is given by



MBðtÞ � E etB� �

¼Z 1

0

exp t lnxm

� �h i xm=2�1e�x=2

2m=2Cðm=2Þdx ¼ Cðt þ m=2Þ

Cðm=2Þðm=2Þtfor all t <

1

2. ð45Þ

It follows directly from (45) that the cumulant generating function of B is given by

KBðtÞ � ln MBðtÞ½ � ¼ ln Cðt þ m=2Þ½ � � ln Cðm=2Þ½ � � t lnðm=2Þ for all t <1

2ð46Þ

(Stuart and Ord, 1994); and thus we have

E ln½v2ðmÞ=m�

¼ E½B� ¼ K 0BðtÞjt¼0 ¼ Wðm=2Þ � lnðm=2Þ; ð47Þ

where W(Æ) is the digamma function (30).Using the result (47) with m = 2a, we see from (44) that E eLX ðmÞð0Þ

h i� fX ðmÞð0Þ þWðaÞ � lnðaÞ. Perform-

ing a similar analysis for the frequency ‘k ¼ 1

2, we find that E eLX ðmÞð12Þ

h i� fX ðmÞð12Þ þWðaÞ � lnðaÞ.

A similar argument yields the result E eLX ðmÞð‘kÞh i

� fX ðmÞð‘kÞ þWðAÞ � lnðAÞ for a <j ‘ j< k2� a.

A.2. Bias adjustment at frequency ‘k for 1 6 j‘j 6 a and k

2� a 6 j‘j 6 k

2� 1

At the frequency ‘k for 1 6 j‘j 6 a, we observe that eI X ðmÞð‘kÞ is a weighted average of independent chi-

squared variates in which the weights are all positive constants that are not necessarily equal in value;and thus the results of Satterthwaite (1941) and Welch (1956) ensure that an excellent approximation tothe distribution of the smoothed periodogram at the frequency ‘

k for 1 6 j‘j 6 a is given byeI X ðmÞð‘kÞ�� p X ðmÞð‘kÞv2ðmj‘jÞ=mj‘j, where mj‘j denotes the ‘‘effective’’ degrees of freedom inherent in eI X ðmÞð‘kÞ. We

have eLX ðmÞð‘kÞ ¼ ln½eI X ðmÞð‘kÞ� ��

ln½p X ðmÞð‘kÞ� þ ln½v2ðmj‘jÞ=mj‘j�, from which it follows immediately that

E½eLX ðmÞð‘kÞ� � fX ðmÞð‘kÞ þ Efln½v2ðmj‘jÞ=mj‘j�g. In general an expression specifying the effective degrees of free-

dom for eI X ðmÞð‘kÞ is given by

m‘ ¼$

2=CV2 eI X ðmÞ‘

k

� �� %; ð48Þ

where CV½eI X ðmÞð‘kÞ� is the coefficient of variation of eI X ðmÞð‘kÞ so that

CV2 eI X ðmÞ‘

k

� �� Var eI X ðmÞ

‘

k

� �� E2 eI X ðmÞ

‘

k

� �� ð49Þ

(Satterthwaite, 1941). For the frequency ‘k with 1 6 j‘j 6 a, we can easily show that

E eI X ðmÞ‘

k

� �� ¼ p X ðmÞ

‘

k

� �. ð50Þ

To compute the variance of the smoothed periodogram of the batch means, we observe that for 1 6 ‘ 6 a,

eI X ðmÞ‘

k

� �¼ 1

A2aþ 1

a

� �Xa�‘u¼1

IX ðmÞuk

� �þ aþ 1

a

� � Xa

u¼a�‘þ1

IX ðmÞuk

� �þX‘þa

u¼aþ1

IX ðmÞuk

� �" #; ð51Þ

and it follows immediately from (51) that:

Var eI X ðmÞ‘

k

� �� p 2

X ðmÞ‘

k

� �4a2 � 2a‘þ 4a� 2‘þ 1

að2aþ 1Þ2for 1 6 ‘ 6 a. ð52Þ


Substituting (50) and (52) into (49), we have

CV2 eI X ðmÞ‘

k

� �� 4a2 � 2a‘þ 4a� 2‘þ 1

aA2for 1 6 ‘ 6 a. ð53Þ

In view of (48) and (53), we see that (29) yields the effective degrees of freedom in eI X ðmÞ‘k

� �for 1 6 ‘ 6 a;

and thus E eLX ðmÞð‘kÞh i

� fX ðmÞ‘k

� �þWðmj‘j=2Þ � lnðmj‘j=2Þ for 1 6 j‘j 6 a. When k

2� a 6 j‘j 6 k

2� 1, a similar

argument yields the result eI X ðmÞð‘kÞ��

p X ðmÞð‘kÞv2ðmk=2�j‘jÞ=mk=2�j‘j; and it follows that E½eLX ðmÞð‘kÞ� �fX ðmÞð‘kÞ þWðmk=2�j‘j=2Þ � lnðmk=2�j‘j=2Þ for k

2� a 6 j‘j 6 k

2� 1.

Appendix B. Variance reduction analysis for estimating the batch means log-spectrum

If a random variable has the form B ¼ ln½v2ðmÞ=m�, then by (46) its variance is given by

Var½B� ¼ K 00BðtÞjt¼0 ¼C00 m

2

� �C m

2

� �� C0 m

2

� �C m

2

� �� 2¼ W0

m2

� �; ð54Þ

where W0ðzÞ ¼ ddz WðzÞ for all z with Re(z) > 0 is the trigamma function (Gradshteyn and Ryzhik, 2000).

Using (54) together with analyses that parallel the derivations of the bias terms in displays (42)–(53) above,we obtain the entries in Table 4. For example, from (43) we have Var½eLX ðmÞð0Þ� � Varfln½v2ð2aÞ=ð2aÞ�g ¼W0ðaÞ by taking m = 2a in (54). All the other variances in Table 4 are derived in a similar fashion.

References

Brillinger, D.R., 1981. Time series: Data analysis and theory, expanded ed. Holden-Day, San Francisco.Bruce, A., Gao, H.-Y., 1996. Applied Wavelet Analysis with S-PLUS. Springer, New York.Chatfield, C., 1984. The Analysis of Time Series: An Introduction, third ed. Chapman and Hall, London.Chen, E.J., Kelton, W.D., 2003. Determining simulation run length with the runs test. Simulation Modelling Practice and Theory

11 (3–4), 237–250.Chien, C., Goldsman, D., Melamed, B., 1997. Large-sample results for batch means. Management Science 43 (9), 1288–1295.Chow, Y.S., Robbins, H., 1965. On the asymptotic theory of fixed-width sequential confidence intervals for the mean. The Annals of

Mathematical Statistics 36, 457–462.Donoho, D.L., 1995. De-noising by soft-thresholding. IEEE Transactions on Information Theory 41 (3), 613–627.Donoho, D.L., Johnstone, I.M., 1994. Ideal spatial adaptation by wavelet shrinkage. Biometrika 81 (3), 425–455.Donoho, D.L., Johnstone, I.M., 1995. Adapting to unknown smoothness via wavelet shrinkage. Journal of the American Statistical

Association 90 (432), 1200–1224.Fishman, G.S., 1978. Grouping observations in digital simulation. Management Science 24 (5), 510–521.Fishman, G.S., 1998. LABATCH.2 for analyzing sample path data. Technical report, Department of Operations Research, University

of North Carolina, Chapel Hill, NC. Available on-line via www.unc.edu/depts/or/downloads/tech_reports/fishman/uncor97-04.ps[accessed April 12, 2005].

Fishman, G.S., Kiviat, P.J., 1967. The analysis of simulation-generated time series. Management Science 13 (7), 525–557.Fishman, G.S., Yarberry, L.S., 1997. An implementation of the batch means method. INFORMS Journal on Computing 9 (3), 296–

310.Gao, H.-Y., 1997. Choice of thresholds for wavelet shrinkage estimate of the spectrum. Journal of Time Series Analysis 18 (3), 231–

251.Gradshteyn, I.S., Ryzhik, I.M., 2000. Table of Integrals, Series, and Products, sixth ed. Edited by Alan Jeffrey and translated from the

Russian by Scripta Technica, Inc. Academic Press, San Diego.Heidelberger, P., Welch, P.D., 1981a. A spectral method for confidence interval generation and run length control in simulations.

Communications of the ACM 24 (4), 233–245.Heidelberger, P., Welch, P.D., 1981b. Adaptive spectral methods for simulation output analysis. IBM Journal of Research and

Development 25 (6), 860–876.

http://www.unc.edu/depts/or/downloads/tech_reports/fishman/uncor97-04.ps


Heidelberger, P., Welch, P.D., 1983. Simulation run length control in the presence of an initial transient. Operations Research 31 (6),1109–1144.

Irizarry, M.A., Kuhl, M.E., Lada, E.K., Subramanian, S., Wilson, J.R., 2003. Analyzing transformation-based simulationmetamodels. IIE Transactions 35 (11), 271–283.

Lada, E.K., 2003. A wavelet-based procedure for steady-state simulation output analysis. Ph.D. dissertation, Graduate Program inOperations Research, North Carolina State University, Raleigh, NC. Available on-line via <www.lib.ncsu.edu/theses/available/etd-04032003-141616/unrestricted/etd.pdf> [accessed April 12, 2005].

Lada, E.K., Wilson, J.R., 2003. Online companion to ‘‘A wavelet-based spectral procedure for steady-state simulation analysis.’’Technical report, Department of Industrial Engineering, North Carolina State University, Raleigh, NC. Available on-line via<ftp.eos.ncsu.edu/pub/jwilson/lw1oc.pdf> [accessed April 12, 2005].

Lada, E.K., Wilson, J.R., Steiger, N.M., 2003. A wavelet-based spectral method for steady-state simulation analysis. In: Chick, S.,Sanchez, P.J., Ferrin, D., Morrice, D.J. (Eds.), Proceedings of the 2003 Winter Simulation Conference, Institute of Electrical andElectronics Engineers, Piscataway, NJ, pp. 422–430. Available on-line via <www.informs-sim.org/wsc03papers/052.pdf> [accessedApril 12, 2005].

Lada, E.K., Wilson, J.R., 2004. WASSP software and user�s manual. Department of Industrial Engineering, North Carolina StateUniversity, Raleigh, North Carolina. Available on-line as a self-extracting archive via <ftp.eos.ncsu.edu/pub/jwilson/installw-assp.exe> [accessed April 12, 2005].

Lada, E.K., Steiger, N.M., Wilson, J.R., Joines, J.A., 2004a. Performance evaluation of a wavelet-based spectral method for steady-state simulation analysis. In: Ingalls, R.G., Rossetti, M.D., Smith, J.S., Peters, B.A. (Eds.), Proceedings of the 2004 WinterSimulation Conference, Institute of Electrical and Electronics Engineers, Piscataway, NJ, pp. 694–702. Available on-line via<www.informs-sim.org/wsc04papers/084.pdf> [accessed April 12, 2005].

Lada, E.K., Wilson, J.R., Steiger, N.M., Joines, J.A., 2004b. Performance of a wavelet-based spectral procedure for steady-statesimulation analysis. INFORMS Journal on Computing, in review. Available on-line via <ftp.ncsu.edu/pub/eos/pub/jwilson/lada04joc.pdf> [accessed April 12, 2005].

Law, A.M., Carson, J.S., 1979. A sequential procedure for determining the length of a steady-state simulation. Operations Research27 (5), 1011–1025.

Law, A.M., Kelton, W.D., 2000. Simulation Modeling and Analysis, third ed. McGraw-Hill, Boston.Mallat, S.G., 1989. Multiresolution approximations and wavelet orthonormal bases of L2(R). Transactions of the American

Mathematical Society 315 (1), 69–87.Moulin, P., 1994. Wavelet thresholding techniques for power spectrum estimation. IEEE Transactions on Signal Processing 42 (11),

3126–3136.Nadas, A., 1969. An extension of a theorem of Chow and Robbins on sequential confidence intervals for the mean. The Annals of

Mathematical Statistics 40 (2), 667–671.Percival, D.B., Walden, A.T., 2000. Wavelet Methods for Time Series Analysis. Cambridge University Press, Cambridge.Royston, J.P., 1982. An extension of Shapiro and Wilk�s W test for normality to large samples. Applied Statistics 31 (2), 115–124.Royston, P., 1993. A toolkit for testing for non-normality in complete and censored samples. The Statistician 42 (1), 37–43.Royston, P., 1995. Remark AS R94: A remark on algorithm AS 181: The W-test for normality. Applied Statistics 44 (4), 547–551.Satterthwaite, F.E., 1941. Synthesis of variance. Psychometrika 6 (5), 309–316.Shapiro, S.S., Wilk, M.B., 1965. An analysis of variance test for normality (complete samples). Biometrika 52 (3–4), 591–611.Steiger, N.M., Wilson, J.R., 2002. An improved batch means procedure for simulation output analysis. Management Science 48 (12),

1569–1586.Steiger, N.M., Lada, E.K., Wilson, J.R., Alexopoulos, C., Goldsman, D., Zouaoui, F., 2002. ASAP2: An improved batch means

procedure for simulation output analysis. In: Yucesan, E., Chen, C.-H., Snowdon, J.L., Charnes, J.M. (Eds.), Proceedings of the2002 Winter Simulation Conference, Institute of Electrical and Electronics Engineers, Piscataway, NJ, pp. 336–344. Available on-line via <www.informs-sim.org/wsc02papers/043.pdf> [accessed October 2, 2004].

Steiger, N.M., Lada, E.K., Wilson, J.R., Joines, J.A., Alexopoulos, C., Goldsman, D., 2005. ASAP3: A batch means procedure forsteady-state simulation analysis. ACM Transactions on Modeling and Computer Simulation 15 (1), 39–73.

Stuart, A., Ord, J.K., 1994. Kendall�s advanced theory of statistics, sixth ed. Distribution Theory, 1 Edward Arnold, London.Vidakovic, B., 1999. Statistical modeling by wavelets. Wiley, New York.von Neumann, J., 1941. Distribution of the ratio of the mean square successive difference to the variance. The Annals of Mathematical

Statistics 12, 367–395.Wahba, G., 1980. Automatic smoothing of the log periodogram. Journal of the American Statistical Association 75 (369), 122–132.Wahba, G., Wold, S., 1975. Periodic splines for spectral density estimation: The use of cross validation for determining the degree of

smoothing. Communications in Statistics 4 (2), 125–141.Walden, A.T., Percival, D.B., McCoy, E.J., 1998. Spectrum estimation by wavelet thresholding of multitaper estimators. IEEE

Transactions on Signal Processing 46 (12), 3153–3165.Welch, B.L., 1956. On linear combinations of several variances. Journal of the American Statistical Association 51 (273), 132–148.

http://www.lib.ncsu.edu/theses/available/etd-04032003-141616/unrestricted/etd.pdf

http://www.lib.ncsu.edu/theses/available/etd-04032003-141616/unrestricted/etd.pdf

http://www.informs-sim.org/wsc03papers/052.pdf




Wilcox, H.J., Myers, D.L., 1978. An introduction to Lebesgue Integration and Fourier Series. R.E. Krieger Publishing Co., Huntington,NY.

Wilson, J.R., Pritsker, A.A.B., 1978a. A survey of research on the simulation startup problem. Simulation 31 (2), 55–58.Wilson, J.R., Pritsker, A.A.B., 1978b. Evaluation of startup policies in simulation experiments. Simulation 31 (3), 79–89.Young, L.C., 1941. On randomness in ordered sequences. The Annals of Mathematical Statistics 12, 293–300.

a wavelet-based spectral procedure for steady-state …stochastics and statistics a wavelet-based...

Documents