School of Industrial Engineering & Management - International University – VNU-HCM SIMULATION

Chapter 10-2

Verification and Validation

of Simulation Models

Discrete-Event System Simulation

Steps in a Simulation
Study
 Four phases:
– Problem formulation, and setting
objective and overall design
(step 1 to 2).
– Modeling building and data
collection (step 3 to 7)
– Running of the model
(step 8 to 10).
– Implementation (step 11 to 12).
 An iterative process.

2
Calibration and Validation (Cal. & Val.)
• No model is ever a perfect representation of the system
– The modeler must weigh the possible, but not guaranteed,
increase in model accuracy versus the cost of increased validation
effort.

• Three-step validation approach by Naylor & Finger [1967]:

1. Build a model that has high face validity.
2. Validate model assumptions.
3. Compare the model input-output transformations
with the real system’s data.
(to corresponding input-output transformations for the real system)
[validating input-output transformations]

3
1. Build a model that has high Face Validity
[Cal. & Val.]
• The first goal of the simulation modeler is to construct a
model that appears reasonable on its face to model users
and other who are knowledgeable about the real system
being simulated.
• Ensure a high degree of realism: Potential users should be
involved in model construction (from its conceptualization to its
implementation).
• The model builder must attempt to choose the most critical
input variables for testing (Sensitivity analysis can also be
used to check a model’s face validity).
– Example: In most queuing systems, if the arrival rate of customers
were to increase, it would be expected that server utilization, queue
length and delays would tend to increase.
4
Advantages of user involvement
• High degree of realism is built into the model through
reasonable assumptions regarding system structure and
reliable data
• Increase in the models perceived validity or credibility without
which manager will not be willing to trust simulation results as
the basis for decision making
• Sensitivity analysis can also be used to check model’s face
validity
• The model user is asked if the model behaves in the expected
way when one or more input variables is changed
• If it is too expensive or more time consuming to vary all input
variables, then the model builder the most critical input
variables for testing

5
2. Validation of Model Assumptions [Cal. & Val.]
• General classes of model assumptions:
1. Structural assumptions:
• How the system operates.
• Simplification and abstraction of reality
For example:
- Customer queuing and service facility in a bank.
- Customer waiting in one line versus many
lines, served FCFS versus priority.

6
2. Validation of Model Assumptions [Cal. & Val.]
• General classes of model assumptions:
1. Data assumptions:
• Based on collection of reliable data and
• Its (correct) statistical analysis (of the data).
For example data was collected on:
Interarrival time of customers (peak, rush, slack)
Service times for commercial/personal accounts.
Verify data reliability with bank managers.
Test correlation and goodness of fit for data (see Chapter 9 for
more details).

7
Compare the model input-output transformations
[Cal. & Val.]

Procedure for data input data:

1.Identifying appropriate probability
distribution
2.Estimating the parameters of hypothesized
distribution
3.Validating the assumed statistical model by
goodness-of-fit test such as Chi-square test,
KS test and by graphical method
8
3. Validating I-O Transformation [Cal. & Val.]
• Goal: Validate the model’s ability to predict future behavior
– The only objective test of the model.
– The structure of the model should be accurate enough to make
good predictions for the range of input data sets of interest.

• One possible approach: use historical data that have been

reserved for validation purposes only.
• Criteria: use the main responses of interest.

9
Step 3. Validating I-O Transformation [Cal. & Val.]

• In this phase of validation process, the model is viewed as

input-output transformation i.e. is, the model accepts the
values of input parameters and transforms these inputs
into output measure of performance. It is this
correspondence that is being validated
• Instead of validating the model input-output transformation
by predicting the future, the modeler may use past
historical data which has been served for validation
purposes that is, if one set have been used to calibrate the
model, it’s recommended that a separate data test be
used as final validation test

10
3. Validating I-O Transformation [Cal. & Val.]

• Thus accurate “prediction of the past” may replace

prediction of the future for purpose of validating the future
• A necessary condition for input-output transformation is
that some version of the system under study exists so that
the system data under at least one set of input condition
can be collected to compare to model prediction
• If the system is in planning stage and no system operating
data can be collected, complete input-output validation is
not possible
• Validation increase modeler’s confidence that the model of
existing system is accurate

11
3. Validating I-O Transformation [Cal. & Val.]

• Changes the computerized representation of the system,

ranging from relatively minor to relatively major include:
1. Minor changes of single numerical parameters: Speed of
machine, arrival rate of customer
2. Minor changes of the form of a statistical distribution:
Distribution of service time or time to failure of a machine
3. Major changes in the logical structure of a subsystem: Queue
discipline, the scheduling rule for a job shop model
4. Major changes involving a different design for a new system:
computerized inventory control system replacing a non
computerized one

12
Validating I-O Transformation [Cal. & Val.]
Bank Example

• Example 10.2 (p. 399): One drive-in window serviced by

one teller, only one or two transactions are allowed.
– Data collection: 90 customers during 11 am to 1 pm.
• Observed service times {Si, i = 1,2, …, 90}.
• Observed interarrival times {Ai, i = 1,2, …, 90}.

– Data analysis let to the conclusion that:

• Interarrival times: exponentially distributed with rate  = 45
• Service times: N(1.1, 0.22)

13
Validating I-O Transformation [Cal. & Val.]
Bank Example

14
Fig. 10.4 Drive-in window at the Fifth National Bank
Validating I-O Transformation [Cal. & Val.]
Bank Example

15
Fig. 10.5 Model input-output transformation
Validating I-O Transformation [Cal. & Val.]
Bank Example
The Black Box
• A model was developed in close consultation with bank management and
employees
• Model assumptions were validated
• Resulting model is now viewed as a “black box”:
Model Output Variables, Y
Input Variables
Primary interest:
Poisson arrivals Y1 = teller’s utilization
 = 45/hr: X11, X12, … Y2 = average delay
Uncontrolled
Services times, Model Y3 = maximum line length
variables, X
N(D2, 0.22): X21, X22, … “black box”
f(X,D) = Y Secondary interest:
D1 = 1 (one teller) Y4 = observed arrival rate
Controlled
Decision D2 = 1.1 min Y5 = average service time
variables, D (mean service time) Y6 = sample std. dev. of
D3 = 1 (one line) service times
Y7 = average length of time
16
Validating I-O Transformation [Cal. & Val.]
Bank Example

Comparison with Real System Data

• Real system data are necessary for validation.

– System responses should have been collected during the same
time period (from 11am to 1pm on the same Friday.)

• Compare the average delay from the model Y2 with the

actual delay Z2:
– Average delay observed, Z2 = 4.3 minutes, consider this to be the
true mean value 0 = 4.3.
– When the model is run with generated random variates X1n and X2n,
the observed values of average delay Y2 should be close to Z2 = 4.3
minutes.
– Six statistically independent replications of the model, each of 2-
hour duration, are run (given in Table 10.2-slide 19).
17
Validating I-O Transformation [Cal. & Val.]
Bank Example

Hypothesis Testing

• Compare the average delay from the model Y2 with the

actual delay Z2 (continued):
– Null hypothesis testing: evaluate whether the simulation and the real
system are the same (w.r.t. output measures):

H 0: E(Y2 )  4.3 minutes

H1: E(Y2 )  4.3 minutes
• If H0 is not rejected, then, there is no reason to
consider the model invalid
• If H0 is rejected, the current version of the model is
rejected, and the modeler needs to improve the model 18
Validating I-O Transformation [Cal. & Val.]
Bank Example
Hypothesis Testing

– Conduct the t test:

• Chose level of significance ( = 0.5) and sample size (n = 6),
see result in next Slide.
• Compute the same mean and sample standard deviation over
the n replications: n

1 n  (Y 2i  Y2 ) 2
Y2  
n i 1
Y2i  2.51 minutes S i 1
n 1
 0.82 minutes

• Compute test statistics:

Y2  0 2.51  4.3
t0    5.24  t0.025,.5  2.571 (for a 2 - sided test)
S/ n 0.82 / 6

• Hence, reject H0. Conclude that the model is inadequate.

• Check: the assumptions justifying a t test, that the observations
19
(Y2i) are normally and independently distributed.
Validating I-O Transformation [Cal. & Val.]
Bank Example

Table10.2 Results of six representations of the Bank Model

Y4 Y5 Y2=Average Delay
Replication )Arrival/Hour( )Minutes( )Minutes(
1 51 1.07 2.79
2 40 1.12 1.12
3 45.5 1.06 2.24
4 50.5 1.10 3.45
5 53 1.09 3.13
6 49 1.07 2.38
Sample mean 2.51
Standard deviation 0.82

20
Validating I-O Transformation [Cal. & Val.]
Bank Example

Table10.3 Results of six representations of the REVISED Bank Model

Y4 Y5 Y2=Average Delay
Replication )Arrival/Hour( )Minutes( )Minutes(
1 51 1.07 5.37
2 40 1.11 1.98
3 45.5 1.06 5.29
4 50.5 1.09 3.82
5 53 1.08 6.74
6 49 1.08 5.49
Sample mean 4.78
Standard deviation 1.66

• Similarly, compare the model output with the observed

output for other measures:
Y4  Z4, Y5  Z5, and Y6  Z6 21
Validating I-O Transformation [Cal. & Val.]

22
 HW 9
1. Average response time (for inspection) for four designs of car
packing are given in table below. The overall confidence level
should be at least 95%. Design one is as current design where as
others three are considered as possible alternatives. Which of
these designs differs from the current, what is your conclusion
when comparing

Replications 1 2 3 4 5 6 7 8 9 10

Design 1 63.72 32.24 40.28 36.94 36.29 56.94 34.1 63.36 49.29 87.2

Design 2 63.06 31.78 40.32 37.71 36.79 57.93 33.39 62.92 47.67 80.79

Design 3 57.74 29.65 36.52 35.71 33.81 51.54 31.39 57.24 42.63 67.27

Design 4 62.63 31.56 39.87 37.35 36.65 57.15 33.3 62.21 47.46 79.6

23
 HW 9
Literature Review: Your group shall review
and write at least 20 key points from each
of the two attached paper (This will serve
as a summary for interdisciplinary studies
and to develop your skills in writing a
literature review)
a.Concepts for Simulation Based Value
Stream Mapping
b.VERIFICATION AND VALIDATION OF
SIMULATION MODELS

24