Proceedings of the 2002 Winter Simulation Conference

E. Yücesan, C.-H. Chen, J. L. Snowdon, and J. M. Charnes, eds.

ANSWERS TO THE TOP TEN INPUT MODELING QUESTIONS

Bahar Biller Barry L. Nelson

Dept. of Manufacturing & Operations Management Department of Industrial Engineering

Graduate School of Industrial Administration & Management Sciences
Carnegie Mellon University Northwestern University
Pittsburgh, PA 15213, U.S.A Evanston, IL 60208-3119, U.S.A.

ABSTRACT plug in mean values for all the uncertainties and expect to
discover the mean performance of the system itself.
In this tutorial we provide answers to the top ten input-
modeling questions that new simulation users ask, point out 2 DOES THE PARTICULAR
common mistakes that occur and give relevant references. INPUT MODEL MATTER?
We assume that commercial input-modeling software will
be used when possible, and only suggest non-commercial Absolutely! Simply injecting some uncertainty into the
options when there is little else available. Detailed examples simulation is not enough. The simulation outputs can be
will be provided in the tutorial presentation. quite sensitive to the particular input model chosen, and
matching the mean alone is rarely sufficient.
1 WHY USE INPUT MODELS AT ALL? For instance, in the reliability example described in the
answer to Question 1, suppose you modeled the component
This question could be rephrased as, “Why do stochastic life time as having a uniform distribution between 0 and 4
simulation?” The premise behind stochastic simulation— years, because this distribution has the right mean (2 years)
simulation that includes randomness—is that the uncertainty and is easier to work with than the exponential. Under the
in the system cannot be wished away without painting an uniform model, the expected loss on each component is
unrealistic picture of system performance. Input models $3500, rather than $967. So if you were trying to negotiate
represent the uncertainty. For the purpose of this tutorial, a different contract that was profitable, the uniform model
“input modeling” will mean selecting and fitting a probability would cause you to overprice the component (and lose
distribution (perhaps multivariate) to represent some process business to a competitor who has better input models).
whose behavior cannot be predicted with certainty. Input modeling error is particularly nasty because it is
For example, suppose you are a supplier of a component very difficult to quantify. This is in contrast to the estima-
that is supposed to last for one year, a component that you tion error in the simulation output performance measures.
know has a mean time to failure of 2 years. A client is Estimation error can be measured via a confidence interval
willing to pay $1000 for your component, but wants you or standard error, and reduced by making more replications
to pay a penalty of $5000 if failure occurs in less than one or longer runs. Unfortunately, you can not simulate your
year. Should you take this contract? way out of an inaccurate input model.
If you ignore the uncertainty in the component’s life
time and base your decision on the average two-year life, 3 WHY NOT JUST REUSE THE DATA YOU HAVE?
then this is a no-brainer: You will pocket $1000 for each
component you sell. On the other hand, if you know that When process data are available, then using that data to
the distribution of time to failure is well modeled as being drive the simulation model can be a very good idea (we
exponentially distributed (an input model) with mean 2 discuss about how to use it appropriately in the answer to
years, then it turns out that you can expect to lose about Question 9). However, there are a number of reasons why
$967 on each component you sell. In this simple example it is often better to fit an input model. These include the
the expected or long-run average profit can be computed following:
so you do not need simulation to estimate it, but the same
point applies to simulation experiments: You cannot just • To fill in gaps and smooth the data: A finite sample
of data is nearly always an imperfect representation

35
 Biller and Nelson

of the process that produced it. There may be gaps although it will not speed it up. With a parametric
in which values are possible, but none occurred in input model (a probability distribution) you can
this particular sample. Or there may be collections change its parameters, or even select a new distri-
of values that are overrepresented, just by chance. bution, to reflect the changes. If you are reusing
One way to think about input modeling is that you data then somehow you must change the data.
are trying to infer characteristics of the true un-
derlying process that are not perfectly represented 4 WHY ARE THERE SO MANY CHOICES?
in the data. In fact, the physics of the process
may provide a basis for choosing a particular input Even a low-budget input modeling tool will have ten to
model, independent of the data (see Question 4). twelve different distributions from which to choose. Some
• Insure that tail behavior is represented: This is tools have twenty or more. One reason that there are so many
similar to the previous point. By definition, highly choices is that distributions arise naturally when considering
unusual events do not occur very often; therefore, certain physical processes. The normal distribution is a well-
they may not be appropriately represented in a known example. If the time to do some task—assemble the
sample of data, particularly if the sample size is components of a computer, for example—is the result of
small. But these rare events often correspond to adding together the times to do a large number of individual
the extreme conditions (power spikes, long service tasks (each having some variability), then the total time to
times, or early equipment failure) that make sys- complete the task may, according to the Central Limit
tems perform badly. A simulation model that does Theorem, be approximately normally distributed. Thus,
not include the chance of extreme events will not the physical nature of the process (sums of random times)
correctly represent the risks to the system. By fit- leads naturally to a particular type of distribution. To take a
ting an input model you can infer the tail behavior less well-known example, consider the Weibull distribution.
that may not be present in the data. The Weibull can be derived by considering the minimum
• Reflect dependencies in the inputs: For certain (think first event to occur) of a number of random variables.
types of data sets, specifically those that exhibit Because time to failure is often the time when the first of a
dependence or nonstationary behavior, the data number of possible breakdowns occurs, the Weibull arises
set cannot be naively resampled. Consider, for in- as a natural choice in reliability modeling. The number
stance, an input model that represents a customer’s of input model choices is large because the number of
behavior on a commercial web site. The customer physical processes of interest is large. For descriptions of
may undertake a sequence of transactions, such the physical basis of a number of standard distributions see
as connecting, logging in, browsing, adding to a Banks et al. (2001, Chapter 9).
shopping list, more browsing, comparative pric- Although the number of choices is often large, there may
ing, reading product information, more browsing, be fewer distinct choices than it first appears. For instance,
checking out, and disconnecting. Although differ- input-modeling packages often include the gamma, Erlang
ent customers will exhibit different behaviors, cer- and exponential distributions. However, the Erlang and
tain patterns are more likely than others, and some exponential are special cases of the gamma (arising from
may even be forced to occur in sequence (one has restrictions on the gamma’s parameters), so there is really
to connect before logging in, for instance). Thus, only one choice.
it would be wrong to independently resample indi- A practical consequence of this nesting of distributions
vidual transactions because customers do not chose is that algorithms for automatically selecting input models
their transactions independently. In this example, typically select the most flexible member of a family, and
you would need to resample the entire customer not the others (e.g, gamma instead of Erlang or exponential).
session instead. Unfortunately, your simulation This makes sense because a more flexible distribution can
will not see any behavior patterns that were not in more easily accommodate the hills and valleys present in
the sample, a particular problem if the number of a sample of data. To see this for yourself, try doing the
observed sessions is small. following exercise: Use your simulation software to generate
• Incorporate changes in the input process: Suppose data sets of various sizes (100, 500, 1000, 5000) from an
you are not only interested in getting a good model exponential distribution, then ask your software to find the
for an input process, but also in seeing how the “best fit.” Frequently, the exponential will not be selected
system will react to changes in that input. For until the sample size is very large, if it is selected at all.
instance, suppose that an input to your simulation
will be the time a worker requires to assemble
a component. You believe that a new piece of
equipment will reduce the variability in this time,
36
 Biller and Nelson

5 WHAT IS A “GOOD FIT?” can be confusing, especially in this context, so keep this
simple rule in mind: A large p-value supports your choice
The direct answer is that a good fit occurs when an input of input model, and p-values greater than 0.10 are typically
model represents the key features of the real process that considered to be “large.”
have a significant impact on the simulation output measures
of interest. As a practical matter this definition of “good 5.2 Graphical Comparisons
fit” is very difficult to quantify, so others have been derived.
A feature of all modern input-modeling software is the
5.1 Goodness-of-Fit Tests facility to compare a fitted distribution to data. The most
intuitive graphs are based on comparing a fitted density
Undoubtedly the most popular approach to evaluating input function to a histogram of the data. Unfortunately, your
model fit is statistical goodness-of-fit (gof) testing. Under- perception of the fit is highly dependent on the width of the
standing gof tests is important because they can be both histogram cells. The fit may look good when the histogram
useful and misleading. is formed with a few, wide cells, but poor with a large
The gof test starts with the premise that there is a number of small cells. In fact, if the number of cells is
true input model to discover; it then proceeds to determine too large (imagine one cell for each data point), then no
whether there is substantial evidence that the model you distribution will appear to fit. Thus, if you use histogram-
have chosen is not the truth. In gof tests, the null or status based graphical comparisons, try different cell divisions to
quo hypothesis is that you are correct (you have found the see how they change your perception of fit.
true distribution and its parameters), and the alternative is Although less intuitive, graphs based on the cumulative
that you are wrong. The test will reject your choice only distribution function (cdf) do not require data grouping and
if there is overwhelming evidence that you are wrong. The are sensitive to lack of fit and to where the lack of fit occurs.
more data that are available, the easier it is for the test to The q − q plot is a typical example of this type of graphical
deduce that you are wrong. This only makes sense: if you assessment tool and is highly recommended. See Vincent
had a single data point, for instance, then who could say (1998) for a thorough discussion of graphical comparisons.
that any choice was incorrect?
One problem with gof tests is that you know, before 5.3 A Note on Parameter Estimates
you run the test, that your model choice is wrong! You
know this because real data come from real processes, Input models nearly always come with parameters that
not probability distributions. Probability distributions are can be tuned to the data set at hand. For instance, the
mathematical entities that approximate real processes, they Poisson distribution has one parameter, its mean, while the
are not real processes. So if there are enough data, the test lognormal distribution has two parameters, its mean and
will definitely reject your distribution choice, whatever it standard deviation (or variance). For some distributions
is. Thus, having lots of data—usually considered to be a estimating the values of their parameters is a messy numerical
good thing—is bad if your goal is to get your input model analysis problem. One of the nice things that input-modeling
endorsed by a gof test. The statistical term for this is power: software does is parameter estimation.
the more data there are, the more powerful the test is for When statisticians attack the parameter estimation prob-
detecting differences between your distribution choice and lem they look for criteria that lead to estimators with good
the process data. On the other hand, if you do not have statistical properties. The methods of maximum likelihood,
much data then almost any choice will be accepted by the least squares and moment matching are three standard ap-
test. proaches. Should you be worried about what parameter-
So how should gof tests be used, if at all? We suggest estimation methods your software implements? The answer,
they should be advisory only. If you are happy with the fit typically, is no. All of the standard methods have plusses
based on other factors (physical basis, graphical analysis), and minuses. What is more important is that the software
and the gof test fails to reject your choice, then take that as implements them correctly, using numerically stable algo-
additional evidence in favor of your selection. If the gof test rithms, and provides diagnostics like gof tests and graphical
rejects, then you may want to more carefully examine your comparisons. If you are interested in parameter estimation,
choice, but not necessarily give up on it. This is especially see Banks et al. (2001, Chapter 9) and Law and Kelton
true if you have a large data set so that rejection is likely. (2000, Chapter 6).
See Law and Kelton (2000, Chapter 6) for an excellent
treatment of gof testing. 6 WHY NOT JUST USE THE “BEST FIT?”
We have been describing gof tests as if they provide a
go/no-go decision. More typical is that the input-modeling Commercial input-modeling software invariably includes a
software will present a p-value for the test. The p-value feature that will automatically select or recommend a distri-
37
 Biller and Nelson

bution that best fits the sample of data. To our knowledge distribution choice, then use it even if it is not the “best fit.”
these automated features only apply to models of indepen- And avoid histogram-based summary measures, if possible,
dent and identically distributed (i.i.d.) data (see Question 7 when asking the software for its recommendation.
below for what to do with dependence, and Question 8 for
what to do with distributions that change over time). The 7 WHAT IF THERE IS DEPENDENCE
following is a generic description of how these features IN THE PROCESS?
work (details will differ from package to package):
First and foremost, don’t ignore it!
1. Obtain information from the user that could elim- Here are some examples of input processes that might
inate certain candidate distributions. Examples exhibit dependence:
include whether the data are discrete or continu-
ous valued; whether there are known, unknown, 1. A distributor places monthly orders for your prod-
or no bounds on the range of possible values; and uct. Because the distributor may hold inventory
specific candidate distributions to try. (which is outside the scope of your model), a large
2. Fit all feasible candidate distributions to the sample order from the distributor one month is likely to be
of data by estimating values for any parameters. followed by a smaller order the following month,
3. Rank all the fitted distributions by some summary followed by a larger order the next month, etc.
measure of fit, such as the p-value of a goodness- Modeling the monthly orders as independent ran-
of-fit test. dom variables misses this month to month depen-
4. Recommend the distribution with the best summary dence.
measure of fit. 2. Customers who log on to your web site have char-
acteristics that influence their behavior, including
There is nothing inherently wrong with this approach, age, sex, income level and where they live. To
and it never hurts to see what the software recommends. treat these customer characteristics as independent
But it is a mistake to slavishly take the recommendation random variables misses the obvious relationship
for the following reasons: between age and income, for instance.
3. In the first example, suppose that the distributor has
• The selection is based on a summary measure of fit, several warehouses and each places monthly orders
and different summary measures lead to different for your product. The month-to-month dependence
recommendations. Which summary measure is the still exists, but there may also be dependence be-
right one? The answer depends on characteristics tween the orders from different warehouses in the
of the data and on what sort of lack of fit bothers same month if they are able to share inventory or
you most. Do you want to get the tails or the supply the same customers.
center of the distribution right? Are you interested
in minimizing the largest discrepancy between the The first example calls for a time series input model,
data and the fitted distribution or the average of a sequence of random variables that all have the same
all the discrepancies? Do you believe that there is probability distribution, but exhibit dependence. The de-
indeed a “true distribution” or are you only trying pendence is often measured by the autocorrelation, which
to find a close approximation to the given data? is the correlation between observations within the series.
• Some measures of fit are sensitive to how your data The second example calls for a random vector input
are grouped. In particular, the popular chi-squared model, where each component of the vector—age, sex, in-
statistic depends on the number and size of the cells come level and location—may be described by a different
in your histogram, as described in Question 5. If probability distribution, but the components depend on the
you change the grouping of your data you may other. This dependence is often characterized by a corre-
end up with a different recommendation. lation matrix whose elements are the pairwise correlations
• The software usually does not account for the phys- between the components.
ical basis of the data (see Question 4), and the The third example calls for a vector time series input
physical basis may provide the best indication of model that has dependence in sequence (month to month) and
the right family of distributions to choose. across components (the orders from different distributors).
• You are smarter than the software. All simulation software includes input models for i.i.d.
processes, and all input modeling packages fit distributions
Our recommendation is to use every graphical tool to i.i.d. data. Few of the products include facilities for
available in the software to examine the fit, and if it is a modeling dependent input processes. Thus, there is an
histogram-based tool to be sure to play with different widths almost overwhelming temptation to use i.i.d. models. Un-
of the cells. If there is a strong physical basis for a particular
38
 Biller and Nelson

fortunately, many studies have shown that ignoring depen- However, if you believe that values between the observed
dence can greatly distort the simulation output performance data points are possible, then there are various interpolation
measures. For instance, if there is actually positive auto- schemes that can be used to smooth the empirical cdf, and
correlation between the interarrival times of customers to a even add tails. We highly recommend these. Banks et al.
queue but you ignore it, then the simulation of the queue (2001, Chapter 8) shows one way to do it.
can grossly underestimate the congestion that will actually As mentioned in the answer to Question 3, simple
occur. Vincent (1998) describes techniques for assessing resampling is not appropriate when the process exhibits
whether or not there is dependence in a data set. dependence or nonstationary. Dependence can occur in one
Multivariate input models based on the normal of two ways, or both: (1) There is dependence in sequence
distribution—including time series, random vectors, and (a time series), such as the values of a stock index recorded
vector time series—are well known to statisticians and easy every 10 minutes; or (2) there is dependence across different
to fit and simulate. Recently, researchers have developed input processes, such as the dependence between sales of
tools that transform input models with normal distributions new cars and the sales of car tires. In case (1) you should
into input models with (any) other distributions. See Nelson resample an entire series of values, while in case (2) you
and Yamnitsky (1998) for an overview, and <www.iems. should resample pairs (or in general vectors) of values that
northwestern.edu/˜nelsonb> for software. were observed together.
Nonstationarity means that the input process changes
8 WHAT IF THE PROCESS over time. For instance, consider the number of users
CHANGES OVER TIME? connected to an Internet Service Provider (ISP) by time
of day. There are clearly peaks and valleys in the user
Again, don’t ignore it! load. Similar to the case of dependence, when there is
Input processes that change over time are said to be nonstationary behavior then entire cycles must be resampled
nonstationary. A typical example is an arrival process in (entire days of user load profiles in the example).
which the arrival rate varies by the time of day, day of
the week, etc. For instance, nonstationarity occurs in the 10 WHAT IF I HAVE NO DATA?
arrival of customers to a restaurant (rate is greater around
meal times), arrival of e-mail messages to a mail server The short answer is, be resourceful and be creative. When
(lower rate at night), and the times of discovery of bugs in no data are available you have to use anything you can find
a software product (rate tends to decrease over time). as a basis for your input models: engineering standards and
The Poisson arrival process—where times between ar- ratings; expert opinion; physical or conventional limits or
rivals of customers are independent, exponential random bounds; and the physics of the process itself. Here are a
variables—is a standard input model used when arrivals few examples:
occur “at random” (as opposed to, say, on a schedule). The
Poisson arrival process has a constant or stationary arrival • To model the time it takes to do computer data
rate. A generalization of the Poisson arrival process allows entry you could research the world record for typing
the arrival rate to vary with time. Such a process is called a speed to provide an upper bound, and spend a few
nonstationary or nonhomogeneous Poisson arrival process. minutes doing some one-finger typing to find a
Good references are Law and Kelton (2000, Chapter 6) and lower bound. You probably would not use either
Nelson and Yamnitsky (1998). of these numbers, but any input model you selected
should clearly take values between these extremes.
9 HOW CAN I REUSE THE DATA I HAVE? • In designing a new work cell containing a number
of machining processes you might use the manu-
As mentioned in the answer to Question 3, there are reasons facturers’ ratings for cycle time as a basis for input
not to reuse input data that you have collected. However, models on actual cycle times.
when an adequate sample is available, the data are thought • If your model requires the number of defective
to be representative and there is no compelling reason to use items found in a shipment of parts, and each item
a probability model (including the case that nothing appears is independently good or bad, then the physics of
to fit well), then using the data themselves is clearly an the situation implies that a binomial distribution
option. The idea is to resample the data to produce inputs is appropriate. You then have to supply a size
for the simulation. for the shipment and a probability that an item is
When the data are believed to be approximately i.i.d., defective.
then they should be sampled, with replacement, in such a
way all the data points are equally likely. This is known as By far the most common approach when data are not
using the empirical cdf and it has good statistical properties. available is to use “expert opinion,” meaning that you draw

39
 Biller and Nelson

on the knowledge and experience of people who are familiar value will be generated, which makes no sense in this ex-
with the process you want to model. Experts are often able ample. When an input model with infinite range is used,
to estimate the center and the extremes. However, even then be sure to check for values that are generated outside
though people may feel comfortable provide an average the feasible range for that process.
value, what they may mean by “average” is “most likely,” Assessing the sensitivity of simulation output results
which is not necessarily the same thing. Thus, it is better to the input models chosen is always important, and this is
to ask for the most likely value directly and interpret what especially true when the input models are determined without
you get that way. data. Sensitivity to both the center of the distribution and
The triangular distribution is a an easy-to-use input its variability should be checked. For instance, if you were
model that is specified by minimum, most likely and maxi- using a triangular distribution, then you could shift the most
mum possible values, things experts often can supply. Avoid likely value and move the minimum and maximum closer
the temptation to use the uniform distribution, which only together and farther apart. Those distributions that have
requires minimum and maximum values. There are very a substantial impact on the simulation output should be
few real processes in which the extremes are as likely as reexamined with more care.
the center, but that is what the uniform distribution implies.
If there are a small number of discrete outcomes, then REFERENCES
you want to ask the expert for the percentage chance of
each. For instance, if the event is whether or not you Banks, J., J. S. Carson, B. L. Nelson and D. Nicol. 2001.
win the contract, then elicit the expert’s subjective chance Discrete-Event System Simulation. 3d ed. Upper Saddle
of each outcome. Even when there are a large number River, New Jersey: Prentice Hall.
of outcomes—far too many to specify the chance of each Law, A. M. and W. D. Kelton. 2000. Simulation Modeling
one individually—an expert might be able to provide a and Analysis. 3d ed. New York: McGraw-Hill.
probability of meeting or exceeding several targets. As an Nelson, B. L. and M. Yamnitsky. 1998. Input modeling
example, sales people are sometimes comfortable making tools for complex problems. In Proceedings of the
statements such as the following: “We will definitely do 1998 Winter Simulation Conference, ed. D. J. Medeiros,
$300,000 in sales because we have those orders locked up. E. F. Watson, J. S. Carson and M. S. Manivannan, 105–
I think we have a 50% chance of exceeding $600,000, and a 112. Piscataway, New Jersey: Institute of Electrical
10% chance of beating $700,000. The absolute limit in sales and Electronics Engineers.
for next year is $850,000 if we get the entire market.” These Vincent, S. 1998. Input data analysis. In The Handbook
breakpoints—numerical values and the chance of exceeding of Simulation, ed. J. Banks, 55–91. New York: John
(or, equivalently, not exceeding) them—can be used to Wiley & Sons.
specify the piecewise continuous distributions incorporated
into nearly all simulation languages. See Banks et al. (2001, AUTHOR BIOGRAPHIES
Chapter 9) for a detailed example.
In some contexts an expert may be willing to supply BAHAR BILLER is an assistant professor of Operations
a central value and a percentage variation around it. For Management and Manufacturing at Carnegie Mellon Uni-
instance, “the average time to pick an order is 20 minutes, versity. She received her Ph.D. from Northwestern Univer-
plus or minus 10%.” This might suggest a normal distribu- sity. Her research interests are in computer simulation of
tion for picking time with mean 20 minutes and standard stochastic systems and stochastic input modeling.
deviation 20 × 0.10 = 2 minutes. This could be fine, but
there are some cautions to keep in mind. As mentioned BARRY L. NELSON is the Krebs Professor of Industrial
above, the mean and the most likely value do not always Engineering and Management Sciences at Northwestern
have to be the same. More critically, people do not naturally University, and is Director of the Master of Engineering
think in terms of “standard deviations.” In this context you Management Program there. His research centers on the
would need to insure that “plus or minus 10%” means the design and analysis of computer simulation experiments on
average deviation from 20 minutes, not the most extreme models of stochastic systems. He has published numerous
deviation. For a normal distribution with mean µ and stan- papers and two books. Nelson has served the profession
dard deviation σ , roughly 33% of the values will be outside as the Simulation Area Editor of Operations Research and
the range [µ − σ, µ + σ ]. If the expert meant that virtually President of the INFORMS (then TIMS) College on Simula-
all orders take between 18 and 22 minutes, then 2 minutes tion. He has held many positions for the Winter Simulation
might better correspond to 3 standard deviations, not one. Conference, including Program Chair in 1997 and current
Finally, be careful with models like the normal that have membership on the Board of Directors. His e-mail and web
an infinite range. If 0 is within 3 standard deviations of addresses are <[email protected]> and
the mean then there is a nontrivial chance that a negative <www.iems.northwestern.edu/˜nelsonb/>.
40