a sequential procedure for determining the length of a steady-state simulation

A Sequential Procedure for Determining the Length of a Steady-State SimulationAuthor(s): Averill M. Law and John S. CarsonSource: Operations Research, Vol. 27, No. 5 (Sep. - Oct., 1979), pp. 1011-1025Published by: INFORMSStable URL: http://www.jstor.org/stable/170064 .

Accessed: 09/05/2014 19:21

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact [email protected].

.

INFORMS is collaborating with JSTOR to digitize, preserve and extend access to Operations Research.

http://www.jstor.org

This content downloaded from 195.78.109.71 on Fri, 9 May 2014 19:21:33 PMAll use subject to JSTOR Terms and Conditions

http://www.jstor.org/action/showPublisher?publisherCode=informs

http://www.jstor.org/stable/170064?origin=JSTOR-pdf

http://www.jstor.org/page/info/about/policies/terms.jsp


A Sequential Procedure for Determining the Length of a Steady-State Simulation

AVERILL M. LAW and JOHN S. CARSON University of Wisconsin, Madison, Wisconsin

(Received April 1977; accepted October 1978)

A common problem faced by simulators is that of constructing a confidence interval for the steady-state mean of a stochastic process. We have reviewed the existing procedures for this problem and found that all but one either produce confidence intervals with coverages which may be considerably lower than desired or have not been adequately tested. Thus, in many cases simulators will have more confidence in their results than is justified. In this paper we present a new sequential procedure based on the method of batch means for constructing a confidence interval with coverage close to the desired level. The procedure has the advantage that it does not explicitly require a stochastic process to have regeneration points. Empirical results for a large number of stochastic systems indicate that the new procedure per- forms quite well.

TET {Xi, i > 1) be a stochastic process with steady-state mean p = 14 mn yco ~1E7 Xi/n (with probability 1). A common problem faced by simulators is that of constructing a confidence interval (c.i.) for [L. There are two basic approaches: (1) construct a c.i. from an arbitrary fixed sample size or (2) sequentially increase the sample size until an "acceptable" c.i. can be constructed.

For the fixed sample size approach, five methods have been suggested in the simulation literature: replication, batch means, spectrum analysis, autoregressive representation, and regeneration cycles. (See Crane and Iglehart [6, 7], Fishman [10, 11], Iglehart [14], and Law [17].) Unfortu- nately, because of various approximations, all of these methods have the drawback that if the total sample size chosen is too small, then the actual coverage of a constructed c.i. may be considerably lower than desired. This was empirically shown for replication and batch means in [17] and will be demonstrated for the other three methods in Law [18]. (These results were reported at the 1976 Winter Simulation Conference.)

If the total sample size is made sufficiently large, then the actual coverage realized by each of these five methods will be close to the desired level; however, what is sufficiently large for one stochastic system may not be adequate for another. Thus, in practice, a simulator will not know the actual coverage of his c.i.

1011

Operations Research 0030-364X/79/2705-1011 $01.25 Vol. 27, No. 5, September-October 1979 (C 1979 Operations Research Society of America



1012 Law and Carson

The above summary suggests that a procedure is needed to determine the total sample size necessary to achieve "acceptable" coverage. Five such procedures have been suggested. Fishman [9] applied a procedure based on autoregressive representation to 100 independent simulation runs of an M/M/1 queue with p = 0.9. On each run he attempted to construct a 90% c.i. for the mean number of customers in system, L, and found that between 66 and 79% of the c.i.'s covered L, depending on the choice of the initial sample size. Robinson [22] applied a similar sequential procedure based on regeneration cycles to 100 runs of the M/M/1 queue with p = 0.5. On each run he attempted to construct a 90% c.i. for the mean delay in queue, d, finding that between 60 and 63% of the c.i.'s covered d, depending on the initial sample size. Mechanic and McKay [20] developed a procedure based on batch means that was difficult to understand and was never substantially tested. However, it was their work that provided the initial motivation for the sequential procedure presented in the next section. Fishman [12] presented a procedure based on the regenerative method and the sequential procedure of Chow and Robbins. Unfortunately, no empirical evidence is presented as to the coverage that can be expected when using the procedure.

To the best of our knowledge, the only procedure in the literature which has been shown to perform well is one developed by Lavenberg and Sauer [15]. Their procedure, which is based on the regenerative method, requires that a simulation run until the ratio of the c.i. half length to the point estimate is less than a specified constant y. They prove the asymptotic validity of the c.i.'s produced by their procedure as y approaches zero and also present empirical results for a large number of queueing systems which indicate that the procedure gives good coverage in most cases for y = 0.025. One limitation of the procedure is that it requires the stochastic process of interest to have (frequently occurring) regeneration points, a property not shared by many real-world systems (e.g., a queueing system with two or more non-Poisson arrival streams).

In Section 1 we give a general description of the new procedure. Section 2 describes the stochastic models which are used to test specific versions of the procedure in Section 3. Finally, in Section 4 we summarize our findings and comment on the computational efficiency of the procedure.

1. THE PROCEDURE

Suppose we make a simulation run of length n and then divide the resulting observations Xi, X2, -, Xn, which are assumed to be from a covariance stationary process, into k batches of length m (n = k im). Let Xj(m) (j = 1, 2, * , *k) be the sample mean of the m observations in the jth batch and let X(k, m) be the grand sample mean. We will estimate



Steady-State Simulation 1013

the variance of X(k, m) by a2[X(k, m)] = Yj=i [Xj(m) - X(k, m)]2/k(k -

1). For i = 09 1, n-* n-1, let Ci = Cov[Xj, Xj~j] and for i = 0, 1, ,

k - 1, let Ci(m) = Cov[Xj(m), Aj+,(m)] and pi(m) = Ci(m)/Co(m). The following lemma, which is proved in the appendix, shows that pi(m) 0 0 (i > 1) for sufficiently large m.

LEMMA 1. If 0 < El'=-. Cl < oo, then pi(m) - 0as m o for i = 1, 2, ,k - 1.

Let b(k, m) be such that E{&2[X(k, m)]} = b(k, m)a2[X(k, m)]. We now prove that &2[X(k, m)] is asymptotically unbiased as m -- oo.

THEOREM 1. If 0 < El=-o C < <o, then b(k, m) -- 1 as m -- oo for all k > 2.

Proof. It is easy to show that (see [17]) b(k, m) = ({k/[l + 2 >7/-12 (1 - i/k)p(m)]} -1)/(k - 1). The desired result follows since pi(m) -O 0 as m--oofori= 1,2,9 2 ,k-1bythelemma.

The above expression for b(k, m) shows that &2[X(k, m)] is unbiased when pi(m) = 0 for i = 1, 2, ... , k - 1, has a negative bias (b(k, m) < 1) when pi(m) > 0 V i, and has a positive bias (b(k, m) > 1) when pi(m) < 0 V i. The case pi(m) > 0 V i is of the greatest concern since u2[X(k, m)] will then be underestimated and the coverage of the resulting c.i. (see (1) below) is likely to be less than desired.

If we choose m large enough so that the Xj(m)'s are approximately normally distributed in addition to being uncorrelated, then it becomes plausible to proceed as if the Xj(m)'s were independent, identically distributed normal random variables (r.v.'s) and to use the following to construct an approximate 100(1 - a)% c.i. for A:

X(k, m) + tkl,lj_./2J2[X(k, m)], (1)

where tk-1,1-a/2 is the 1 - a/2 point for a t distribution with k - 1 degrees of freedom.

There are three potential sources of error when using (1) to construct a c.i.: (1) bias in &[X(k, m)] when m is too small for the Xj(m)'s to be uncorrelated; (2) nonnormality of the Xj(m)'s; and (3) the fact that {Xi, i - 1} is not, in practice, covariance stationary. However, for simple queueing models (e.g., M/M/1) Law [17] found that the bias in &2[X(k, m)] was the most serious source of error and that nonnormality was not a problem for k approximately 20 or more.

We conclude that a sequential procedure based on batch means must be able to determine that batch size, m, for which the X(m)'s are approximately uncorrelated. One approach which has been suggested in the literature (see, for example, Gross and Harris [13, p. 426]) is to fix k



1014 Law and Carson

and then increase m until the estimated lag 1 correlation between the Xj(m)'s is small, say, less than 0.05. The difficulty with this approach is that correlation estimators are generally biased and for small k have a large variance. For example, consider the process {Di, i - 1) for the (stationary) M/M/1 queue with p = 0.9, where Dt is the delay in queue of the ith customer. We made 100 independent simulation runs of this process, each of length 14400 observations. Each run was divided into 30 batches of length 480 and pl(480) was estimated by the jackknifed estimator ji1 (30,480) which is defined by (2) below. For 23 of the 100 runs, p1(30,480) was less than 0.05 (22 were less than 0); however, it can be shown that (see Daley [8]) pl(480) = 0.284 and, furthermore, that b(30,480) = 0.597. Further empirical investigations showed that a sequential procedure based on directly estimating pi(m) from 30 (or any small

pl(m) 1.0

0.4 --

- FIRST - ACCEPTABLE m

0 I J- I I I I I I I II I I

2 Figure 1. Type 1 behavior.

number of) batch means would produce c.i.'s with coverages which may be considerably lower than desired. We also found that for some systems, k might have to be as large as 400 to get a precise estimate of pi (m). Unfortunately, obtaining 400 or more uncorrelated batch means is likely to require a prohibitive number of observations.

These considerations suggest using lk (1 - 2) batches of length m to infer whether k batches of size 1m are approximately uncorrelated. To investigate the relationships between pi(m) and pi(lm) which can occur in practice, we studied the following processes for which pi(m) and b(k, m) can be analytically computed: (1) (Di, i - 1) for the M/M/1 queue (see [8]) with p = 0.5, 0.8, and 0.9; (2) (Ei, i > 1) for an (s, S) inventory system (see [17] for details), where Ej is the expenditure in the ith period; and (3) thirty different AR(1), AR(2), and ARMA(1, 1) time series models (see Box and Jenkins [3, p.. 46]) with parameters chosen over the entire




range of feasible values. From studying these 34 stochastic processes, we found essentially three types of behavior for pi(m) as a function of m, examples of which are shown in Figures 1-3. For type 1 behavior the lag 1 correlation pi(m) strictly decreases to zero. For example, suppose we choose 1 = 10 and k = 40. If for some m, pi(m) < 0.4, then pi(lOm) < 0.05 and 0.9 < b(40, 1Gm) < 1. (This holds for all type 1 models we considered.) The M/M/l queue exhibits type 1 behavior.

In type 2 behavior, pi(m) changes direction one or more times and then strictly decreases to zero. Consider again the example I = 10 and k = 40. If for some m, pi(m) < 0.4 and pi(m') is decreasing for m' > m, then pi(lOm) < 0.05 and 0.9 < b(40, lOm) < 1. (This holds for all type 2 models we considered with one exception, where pi(m) < 0.4 and pi(m') decreasing for m' ?-m implies pI(lOm) = 0.059 and b(40, lOm) = 0.894.) Certain

pl(m) 1.0

0.4 -- +

- FIRST o ACCEPTABLE m

Figure 2. Type 2 behavior.

of the time series models are of this type. For type 3 behavior, pi(m) < 0 and b(40, lOm) > 1, for all m. In this

case the Xj(1Gm)'s may be correlated, but the coverage will be at least as great as that desired. The (s, S) inventory system exhibits type 3 behavior.

We certainly do not claim that the above three types of behavior are the only ones that can occur. In fact, for some of the time series models we studied, pi(m) can be positive and negative. However, if 0 < pi(m) < 0.4 for some m, then pi(lOm) < 0.05 (similar to type 1); and, if pi(m) < 0 for some m, then b(40, lOm) > 1 (similar to type 3).

The above discussion was predicated on knowing pi(m) which, in fact, must be estimated. The usual estimator of p1(m) is i (k, m) = jki1 [Xj(m) - X(k, m)][X1+j(m) - X(k, m)]/?j*- [Xj(m) X(k, m)]2. However, if 111(k/2, m) and p,2(k/2, m) are, respectively, the usual lag 1 estimators



1016 Law and Carson

based on the first k/2 and last k/2 batches (k is assumed to be even), then we can also estimate pi(m) by the jackknifed estimator

11(k, m) = 211(k, m) - [311(k/2, m) + p12(k/2, m)]/2. (2)

We will use 11(k, m) rather than b?(k, m) to estimate pi(m) since, in general, it will be less biased (see Miller [21]).

We now give a general statement of our sequential procedure. Step 0. Fix positive integers 1, k, no, ni (1k even, lk/2 even, no < ni <

2no, ni and 2no each divisible by 1k), the stopping value c > 0, the relative precision y > 0; let i = 1; collect n1 observations.

Step 1. a. Divide the n- observations into 1k batches of size m = n,/lk. Compute ki(lk, m) from Xj(m) (j = 1, 2, ,lk). If p1(lk, m) > c (see Note 1), go to Step 2. If 61(lk, m) c 0 (see Note 2),

-0.5-

-1.0 _ PI (m)

Figure 3. Type 3 behavior.

go to Step lc. Otherwise, go to Step lb. b. Divide ni into lk/2 batches of size 2m. Compute p1 (lk/2, 2m)

from Xj(2m) (j = 1, 2, * * A, lk/2). (See Note 3.) If 5i (lk/2, 2m) < p1(lk, m) (see Note 4), go to Step lc. Otherwise, go to Step 2.

c. Divide ni into k batches of size im. Compute X(k, im) and 6 = tk-ll-(k/2 a2[X(k, im)] from Xj(lm) (j = 1, 2, ** , k). (See Note 3.) If 8/X(k, im) < y (see Note 5), then use X(k, im) and 6 in (1) to construct a c.i. for It and stop. Otherwise, go to Step 2.

Step 2. Replace i by i + 1, set ni = 2ni2 (see Note 6), collect the additional ni - ni1- observations required, and go to Step la.




Notes: 1. This check was suggested by type 1 behavior. 2. This check was suggested by type 3 behavior. 3. An appropriate number of the Xj(m)'s may be averaged to com-

pute the X1(2m)'s or the Xj(lm)'s. 4. This check was suggested by type 2 behavior. We test to see if

p1(m') is decreasing for m' > m by checking to see whether pi(2m) < pi(m).

5. This additional check allows a simulator to require that the relative precision of the c.i. (i.e., the c.i. half length divided by the point estimate) be less than a specified number -y.

6. The successive sample sizes are ni, 2no, 2nM, 4no, 4nj, ... . Thus, the total sample size is doubled every other iteration.

TABLE I

VALUES OF d FOR THE QUEUEING SYSTEMS

System p d

M/M/1 .5 0.50 M/M/1 .8 3.20 M/M/1 LIFO .8 3.20 M/M/1 SIRO .8 3.20 E4/M/1 .8 1.81 M/H2/1 .8 8.00 M/M/2 .8 2.84 M/M/l/M/l .8 6.40

2. THE STOCHASTIC MODELS

In this section we describe the stochastic models which are used to test specific versions of the procedure in Section 3.

Queueing Systems

We first considered a variety of simple queueing systems. Let E4, M, and H2 denote, respectively, the Erlang-4, the exponential, and the hyperexponential distributions with coefficients of variation 0.5, 1, and 2. (See Law [16] for further discussion of H2.) For each of the queueing systems M/M/1 (p = 0.5), M/M/1 (p = 0.8), M/M/1 LIFO (p = 0.8), M/M/1 with service in random order (SIRO) (p = 0.8), E4/M/1 (p = 0.8), M/H2/1 (p = 0.8), M/M/2 (p = 0.8), M/M/l/M/l (p = 0.8), we simulated the stochastic process (D1, i 1) and attempted to construct a 90% c.i. for the steady-state mean delay, d. For each system, the mean interarrival time was 1 and no customers were present at time zero. The values of d for these systems are given in Table I.



1018 Law and Carson

Computer Models

To see how well the sequential procedure works on more complex systems, we simulated two models of computer systems. These models resemble more closely the type of system that might actually be simulated in the real world than do the simple models of the previous subsection. We first considered a model of a time-shared computer system which was studied by Adiri and Avi-Itzhak [1] and more recently in a simulation context by Sargent [23]. There are N terminals and a single central processing unit (CPU) as shown in Figure 4. The operator of each terminal "thinks" for an amount of time which is an exponential r.v. with rate A, and then sends a message with service time which is an exponential

TERMINALS COMPUTER --

- -- -I

IEA I UNFINISHED JOBS I I I

2H] 1 11 I~0 00 CPU I I ~ ~ II

- i

FINISHED JOBS Figure 4. Time-shared computer model.

r.v. with rate tL2. The arriving jobs join a FIFO queue at the CPU. The CPU allocates to each job a maximum service quantum of length q. If the (remaining) service time of a job, s, is less than or equal to q, then the CPU spends time s plus T (a fixed overhead) processing the job and the job returns to its terminal. If s > q, then the CPU spends time q plus r processing the job and the job joins the end of the queue. This process is repeated until the job's service time is eventually completed.

We let N = 35, , = 1/25, A2 = 5/4, q = 0.1, T = 0.015, and then simulated the stochastic process {Ri, i : 1}, where Ri is the response time of the ith job requesting service and all terminals are in the think state at time zero. Our objective was to construct a 90% c.i. for the steady-




state mean response time, r = 8.246. We next considered the central-server model of a multiprogrammed

computer system (see Buzen [4]). There is a CPU (unit 1) and M - 1 peripheral units (units 2 through M) as shown in Figure 5. Each unit has its own FIFO queue and the service time at unit i is an exponential r.v. with rate /i (i = 1, 2, * * *, M). It is assumed that there are N jobs in the system at all times. When a job completes service at the CPU it leaves the system with probability Pi or it goes to peripheral unit i with probability pi (i = 2, 3, . .. , M); a job leaving the system is instantaneously replaced by a new job which joins the end of the CPU queue. A job leaving a peripheral unit also joins the CPU queue.

Pi P200 2 , , P2/

3 000 3 00 I

I I PM 9 CPU \

O M

PERIPHERAL UNITS

Figure 5. Central-server computer model.

Let a response time be the time between successive arrivals of a job at the CPU queue and let Ri be the response time for the ith job to arrive at the CPU queue. We simulated the process (Ri, i - 1) for four different sets of parameters and attempted to construct a 90% c.i. for the steady- state mean response time, r. The parameters for these models, which will henceforth be denoted by central-server model i (i = 1, * , 4), are given in Table II. Also included are the steady-state probability pi that unit i is busy and the state of the system at time zero, (1i, 12, * * m, iM), where li is the number of jobs at unit i.

An Inventory System

The last type of model we considered was an (s, S) inventory system with zero delivery lag and backlogging. This system is described in detail



1020 Law and Carson

TABLE II

PARAMETERS FOR THE CENTRAL-SERVER MODELS WITH M = 3, ii = 1, AND pi = 0

Initial Model N ji2

ti:3 P2 P3 Pi P2 P3 r State

1 4 0.50 .50 .5 .5 .67 .67 .67 6.000 (1,1,2) 2 8 0.50 .50 .5 .5 .80 .80 .80 10.000 (1,1,6) 3 8 0.45 .05 .9 .1 .44 .88 .88 18.279 (5,1,2) 4 8 1.79 .20 .9 .1 .99 .50 .50 8.072 (7,1,0)

in [17]. We simulated the process (E1, i - 1) and attempted to construct a 90% c.i. for the steady-state mean expenditure per period, e = 112.108.

3. EMPIRICAL RESULTS

In order to see how specific versions of the procedure work, we simulated the stochastic models described above. The random numbers, { Uj, i - 1), used in these simulations were generated from the following generator which is available on the Univac 1110: Yj = (515 Yj-j + 1) mod 235 (i = 1, 2, * * *) and Uj = Yi/235 (i =1, 2, *.. ), where Yo is a specified seed. For a discussion of this generator, see Coveyou and Macpherson [5].

Because of the large number of parameters needed to specify com- pletely the procedure, it was not possible to vary all parameters simul- taneously. As a result, we decided to choose 1 = 10 and k = 40. Although these values of 1 and k are certainly not the only possibilities, they were

TABLE III

PROPORTION OF COVERAGE, AVERAGE SAMPLE SIZE, AND AVERAGE RELATIVE PRECISION

FOR Y = 00

Model C =.4 c =.3 c =.2

M/M/-1, p = .5 .91 + .05 3768 .136 .90 ? .05 5448 .115 .86 ? .06 8704 .093 M/M/1, p = .8 .85 ? .06 32960 .096 .86 ? .06 51648 .079 .88 ? .05 88960 .064 M/M/1 LIFO, p = .8 .64 ? .08 3120 .236 .72 ? .07 10740 .172 .78 ? .07 25512 .117 M/M/1 SIRO, p = .8 .92 ? .05 33600 .095 .92 ? .05 51520 .080 .91 ? .05 87552 .063 E4/M/1, p = .8 .84 ? .06 20160 .107 .83 ? .06 33248 .087 .89 ? .05 51968 .070 M/H2/1, p = .8 .88 ? .05 86144 .106 .90 + .05 133632 .088 .91 ? .05 203264 .072 M/M/2, p = .8 .85 ? .06 35008 .104 .85 ? .06 48448 .089 .87 ? .06 81664 .074 M/M/1/M/1, p = .8 .88 ? .05 44608 .069 .90 ? .05 66944 .057 .91 ? .05 110208 .046 Time-shared model .92 ? .05 17696 .049 .93 ? .04 23360 .043 .89 + .05 39360 .034 Central-server model 1 .87 + .06 828 .050 .91 + .05 1016 .046 .90 ? .05 1748 .036 Central-server model 2 .87 ? .06 1220 .045 .90 + .05 2240 .034 .90 + .05 4248 .025 Central-server model 3 .87 ? .06 1152 .097 .86 ? .06 1168 .097 .86 ? .06 1944 .091 Central-server model 4 .92 ? .05 1496 .045 .90 ? .05 2120 .039 .94 + .04 3632 .030 (s,S) Inventory system .98 ? .02 800 .013 .98 ? .02 800 .013 .98 ? .02 800 .013 Average coverage* .863 .875 .885 Worst coverage .640 .720 .780

* Excluding inventory model.




chosen because our empirical investigations indicated that 400 batches are sufficient to obtain a precise correlation estimate and also because k must be large enough to obviate the nonnormality of the Xj(m)'s. We then chose nf = 800 since it is the smallest sample size which allows 400 batches of size at least 2 and we chose no = 600 to make n2i = (n2i-I +

n2i+1)/2 for i = 1, 2, ...

For each of the 14 stochastic models of Section 2, we made 100 independent runs using the procedure for each of the values y = co, 0.15, 0.075 and c = 0.4, 0.3, 0.2. (The range for c was suggested by the theoretical results for the 34 stochastic processes discussed in Section 1.) For each run a 90% c.i. was constructed and it was determined whether

TABLE IV


FOR y = .15

Model c =.4 c =.3 c =.2

M/M/1, p = .5 .90 ? .05 4008 .131 .90 ? .05 5448 .115 .86 ? .06 8704 .093 M/M/1, p = .8 .85 ? .06 32960 .096 .86 ? .06 51648 .079 .88 ? .05 88960 .064 M/M/1 LIFO, p = .8 .76 ? .07 13944 .131 .77 ? .07 16888 .126 .81 ? .06 27696 .105 M/M/1 SIRO, p = .8 .92 ? .05 33600 .095 .92 ? .05 51520 .080 .91 ? .05 87552 .063 E4/M/1, p = .8 .84 ? .06 20160 .107 .83 ? .06 33248 .087 .89 ? .05 51968 .070 M/H2/1, p = .8 .88 ? .05 86144 .106 .90 ? .05 133632 .088 .91 ? .05 203264 .072 M/M/2, p = .8 .85 ? .06 35008 .104 .85 ? .06 48448 .089 .87 ? .06 81664 .074 M/M/1/M/1, p = .8 .88 ? .05 44608 .069 .90 ? .05 66944 .057 .91 ? .05 110208 .046 Time-shared model .92 ? .05 17696 .049 .93 ? .04 23360 .043 .89 ? .05 39360 .034 Central-server model 1 .87 ? .06 828 .050 .91 ? .05 1016 .046 .90 ? .05 1748 .036 Central-server model 2 .87 ? .06 1220 .045 .90 ? .05 2240 .034 .90 ? .05 4248 .025 Central-server model 3 .87 ? .06 1180 .096 .86 ? .06 1196 .095 .86 ? .06 2004 .089 Central-server model 4 .92 ? .05 1496 .045 .90 ? .05 2120 .039 .94 ? .04 3632 .030 (s,S) Inventory system .98 ? .02 800 .013 .98 ? .02 800 .013 .98 ? .02 800 .013 Average coverage* .872 .879 .887 Worst coverage .760 .770 .810


the interval contained the known steady-state mean A. In Tables III, IV, and V we give the proportion, i, of the 100 c.i.'s which contained A, a 90% c.i. for the true coverage, the average sample size at termination, and the average relative precision at termination for y = oo, 0.15, and 0.075, respectively. The 90% c.i. for the true coverage was computed from i ? 1.645 up(1 - ))/100.

Following Lavenberg and Sauer [15], we will say that a c.i. for p is valid when the c.i. for the true coverage contains the desired coverage 0.9. Observe that all c.i.'s for It are valid with the exception of those for the (s, S) inventory system, the M/M/1 LIFO queue (y = co and 0.15), and the E4/M/1 queue (y = oc and 0.15, c = 0.3). The results for the inventory system (type 3 behavior) can be explained by the fact that pi(m) < 0 (i



1022 Law and Carson

-1) and b(40, 1Gm) > 1 for all m (see [17] for some actual values). The poor coverage for the LIFO model is due to p1(m)'s being a more slowly decreasing function of m than for the other models considered in this paper.

We also tested the procedure for c = 0.5, 0.6, 0.7, 0.8 and found that the resulting coverages were not as good. By way of example, for the time- shared model with y = 0.15 and c = 0.6, we obtained i = 0.81 (compare with Table IV).

TABLE V


FOR y .075

Model c=.4 c=.3 C=.2

M/M/1, p = .5 .87 ? .06 19472 .064 .87 ? .06 19472 .064 .87 ? .06. 19472 .064 M/M/1, p = .8 .87 ? .06 75648 .065 .87 ? .06 75904 .065 .89 ? .05 94336 .061 M/M/1 LIFO, p = .8 .84 ? .06 74624 .064 .84 ? .06 74624 .064 .84 ? .06 75648 .064 M/M/1 SIRO, p = .8 .92 ? .05 78464 .065 .92 ? .05 78796 .065 .93 ? .04 93440 .060 E1/M/L, p = .8 .86 ? .06 58752 .064 .86 ? .06 58752 .064 .88 ? .05 63424 .062 M/H2/1, p = .8 .90 ? .05 229632 .067 .90 ? .05 231680 .067 .90 ? .05 247040 .065 M/M/2, p = .8 .89 ? .05 102528 .065 .89 ? .05 103552 .064 .89 ? .05 112640 .063 M/M/1/M/1, p = .8 .87 ? .06 49920 .065 .89 ? .05 67968 .057 .91 ? .05 110208 .046 Time-shared model .92 ? .05 17696 .049 .93 ? .04 23360 .043 .89 ? .05 39360 .034 Central-server .87 ? .06 828 .050 .91 ? .05 1016 .046 .90 ? .05 1748 .036

model 1 Central-server .87 ? .06 1220 .045 .90 ? .05 2240 .034 .90 ? .05 4248 .025



model 4 (s,S) Inventory sys- .98 ? .02 800 .013 .98 ? .02 800 .013 .98 ? .02 800 .013

tern Average coverage* .883 .888 .893 Worst coverage .840 .840 .840


4. SUMMARY

For the stochastic models, steady-state means, and 90% level of confidence we considered, the sequential procedure described here produces valid c.i.'s in almost all cases. If one desires a c.i. with a high probability of being valid and is not particularly concerned about the resulting sample size, then a choice of y = 0.075 (or some other small value) and c = 0.4 seems appropriate. A smaller value of c will not appreciably improve coverage, but will increase the required sample size. If one is making an exploratory run and is willing to accept a lower probability of obtaining




a valid c.i., then a choice of y = oo and c = 0.4 seems reasonable. A smaller value of c greatly increases the required sample size.

The sequential procedure described here is fairly easy to program, is computationally efficient, and requires only 800 storage locations to store the batch means, with the parameters chosen as in Section 3. A FOR- TRAN program for the procedure and an explanation of how to use it may be found in Law and Carson [19].

APPENDIX

Our objective is to prove Lemma 1 (see Section 1).

LEMMA A.1. If ET=, Cl converges, then SI` l /m -> 0 as m -* oo.

Proof E'l-IC'/nm = ET=lm1 Cl - E.-I' (m - i)C/m. The result follows since the second sum on the right-hand side converges to D2=I Cl by Lemma 8.3.1 of Anderson [2, p. 460].

LEMMA A.2. If Ej=i Cl converges, then Sil(m)r) = j (1 - /m) Cim+j > 0 as r -00 oofori?1.

Proof Si'(m) = E=+1m)m-l C1 C- El=+1 C /m + Eim lCi/r +

i L$+im+1 Cl. The first and fourth sums converge to 0 since D2=I Cl converges. The second and third sums converge to 0 by Lemma A.1.

LEMMA A.3. If X,1i Cl converges, then Si2(m) -mj.-1 (1 - j/m) Cim-j -* 0 as r -00 oofor i -1.

Proof S-2(m) = amj-1 (im -)Cim-//m (i - 1) a m= C1 = 1 i-)J7-; i\ C-The Et~ml 1ICI/m _ E IC/m-(i-1) l=U(-1)m+1 Cl. The first and second sums converge to 0 by Lemma A.1. The third sum converges to 0 since ET=, Cl converges.

LEMMA 1. If 0 < = Cl < o, then pi(m) -0 as m o for i = 1, 2,

Proof Since pi(m) = mCi(m)/rCo(m) and mCo(m) - =_ Cl as m -

oo (see [2, p. 459]), it suffices to show that mCi(m) 0 as m -> oo. However, mCi(m) = Cim + Si1(m) + Si2(m) (see [17]) and Cim -* 0 as m -* 0o since ET. Cl converges. The desired result now follows from Lemmas A.2 and A.3.

ACKNOWLEDGMENTS

We would like to thank Stephen Lavenberg, Thomas Moeller, Charles Sauer, and Peter Welch, all of IBM Watson Research Center, for their comments on the stopping value c and also their suggestion to use the



1024 Law and Carson

central-server computer model. The authors are also grateful to David Kelton for his reading of the manuscript.

This research was supported by the Office of Naval Research under contract N00014-76-C-0403 (NR 047-145).

REFERENCES

1. I. ADIRI AND B. AvI-ITzHAK, "A Time-Sharing Queue with a Finite Number of Customers," J. Assoc. Comput. Mach. 16, 315-323 (1969).

2. T. W. ANDERSON, The Statistical Analysis of Time Series, John Wiley & Sons, New York, 1970.

3. G. E. P. Box AND G. M. JENKINS, Time Series Analysis: Forecasting and Control, Holden-Day, San Francisco, 1970.

4. J. BUZEN, "Queueing Network Models of Multiprogramming," Ph.D. disser- tation, Division of Engineering and Applied Physics, Harvard University, 1971.

5. R. R. COVEYOU AND R. D. MACPHERSON, Fourier Analysis of Uniform Random Number Generators," J. Assoc. Comput. Mach. 14, 100-119 (1967).

6. M. A. CRANE AND D. L. IGLEHART, "Simulating Stable Stochastic Systems I: General Multi-Server Queues," J. Assoc. Comput. Mach. 21, 103-113 (1974).

7. M. A. CRANE AND D. L. IGLEHART, Simulating Stable Stochastic Systems II: Markov Chains," J. Assoc. Comput. Mach. 21, 114-123 (1974).

8. D. J. DALEY, "The Serial Correlation Coefficients of Waiting Times in a Stationary Single Server Queue," J. Aust. Math. Soc. 8, 683-699 (1968).

9. G. S. FISHMAN, "Estimating Sample Size in Computer Simulation Experi- ments," Mgmt. Sci. 18, 21-38 (1971).

10. G. S. FISHMAN, Concepts and Methods in Discrete Event Digital Simulation, John Wiley & Sons, New York, 1973.

11. G. S. FISHMAN, "Statistical Analysis for Queueing Simulations," Mgmt. Sci. 20, 363-369 (1973).

12. G. S. FISHMAN, "Achieving Specific Accuracy in Simulation Output Analysis," Comm. Assoc. Comput. Mach. 20, 310-315 (1977).

13. D. GROSS AND C. M. HARRIS, Fundamentals of Queueing Theory, John Wiley & Sons, New York, 1974.

14. D. L. IGLEHART, "Simulating Stable Stochastic Systems, V: Comparison of Ratio Estimators," Naval Res. Logist. Quart. 22, 553-565 (1975).

15. S. S. LAVENBERG AND C. H. SAUER, "Sequential Stopping Rules for the Regenerative Method of Simulation," IBM J. Res. Dev. 21, 545-558 (1977).

16. A. M. LAW, "Efficient Estimators for Simulated Queueing Systems," ORC 74-7, Operations Research Center, University of California, Berkeley, 1974.

17. A. M. LAW, "Confidence Intervals in Discrete Event Simulation: A Compar- ison of Replication and Batch Means," Naval Res. Logist. Quart. 24, 667- 678 (1977).

18. A. M. LAW, "Confidence Intervals for Steady-State Simulations, I: A Survey of Fixed Sample Size Procedures," Technical Report No. 78-5, Department of Industrial Engineering, University of Wisconsin, Madison, 1978.




19. A. M. LAW AND J. S. CARSON, "A Sequential Procedure for Determining the Length of a Steady-State Simulation: User's Manual," Technical Report No. 77-13, Department of Industrial Engineering, University of Wisconsin, Madison, 1977.

20. H. MECHANIC AND W. McKAY, Confidence Intervals for Averages of De- pendent Data in Simulations II," Report No. ASDD 17-202, IBM Corp., Yorktown Heights, New York, 1966.

21. R. G. MILLER, "The Jackknife-A Review," Biometrika 61, 1-15 (1974). 22. D. W. ROBINSON, "Determination of Run Length in Simulations of Stable

Stochastic Systems," Technical Report No. 86-21, Control Analysis Cor- poration, Palo Alto, Calif., 1976.

23. R. G. SARGENT, "Statistical Analysis of Simulation Output Data," Proceed- ings of the Symposium on the Simulation of Computer Systems IV, Boulder, Col., August 10-12, 1976.



a sequential procedure for determining the length of a steady-state simulation

Documents