On Event Ordering in Parallel Discrete Event Simulation

Robert Ranngren and Michael Liljenstam

Dept. of Teleinformatics, Royal Institute of Technology (KTH)
Electrum 204, S-164 40 Kista (Stockholm), Sweden
[ robertr, mili) @it.kth.se

Abstract choosing of the next owner, without the releasing entity having
Ordering of simulraneous wenfs in DES is an imporranr yet become a contender for the resource. (2) Selection of the next
issue as it has on impact on modelling expressiveness, model COT- resource owner is deferred until the releasing entity has become a
recrness LISwell as causal dependencies. In sequenriol DES rhis is contender. (3) “Neither of the above”; i.e. the releasing entity
(I problem which has attracted much orrenrion over rhe years and immediately recaptures the resource. The alternatives are not
most systernr pmvide the user with fools fo deal wilh such issues. intrinsically “right” or “wrong” but produce different results. We
This has also arrracred some ntrenrion within Ihe PDES commu- note here that to be able to ensure, in a convenient way, that the
nity and we present on overview of these eflorrs. We have, how- desired effect is reached, may require precise (user) control of the
evel: no, yer found CI scheme which pmvides us wirh rhe desired ordering of events.
function&y. Thus, we present and evolunre some simple As noted by Wieland in [IS], simultaneous events may also
schemes ro achieve LI well defined ordering of ewnfsand means occur due to limited time resolution in measurements and model-
lo idenrify borh cousol/y dependent and independenr ewnrs with ling. In this case it may be preferable to simulate the model over
identical rimesromps in the confezl of oprimisfic simulations. different permutations of orderings of the simultaneous events to
These schemes should be applicable also to conservarive PDES. evaluate the model properly.
Below, Section 2 provides some more detail on the issues rel-
evant to determinism and causality and specifies requirements on
1. Introduction an ordering scheme for PDES. Related work on the ordering of
Ordering of simultaneously occurring events. i.e. events with events in PDES systems is discussed in Section 3. In Section 4
identical simulation time occurring at the same entity (LP), in we describe the implementation of a scheme to provide determin-
Discrete Event Simulations (DES) is a multifaceted problem. It ism in our Time Warp kernel. Experimental results concerning
has impact on model correctness and interpretation of simulation the costs for this scheme presented in Section 5. Section 6 dis-
results for sequential models. When employing Parallel Discrete cusses the implications of introducing user priorities on events
Event Simulation (PDES) techniques, the problems encountered and a scheme for event ordering that meets the requirements of
in a sequential context are further compounded by issues con- Section 2. Finally, our conclusions are presented in Section 7.
cerning deterministic simulation output and the maintaining of
causal relationships. 2. PDES Implications: Determinism and
Simultaneous events may occur in a model for different rea- Causality
sons. When decomposing a model into parts it may be convenient
to use multiple events that do not advance simulated time to What we want to accomplish is to remove some of the peculi-
express the interaction between some parts. However, ensuring arities often found in many PDES systems related to event order-
correct model behaviour (i.e. behaviour consistent with the mod- ing [IO] and provide the same support in this respect as one
ellers intentions) in these cases can lead to complex event sched- expect to find in a sequential simulation system. ?lus the first
uling schemes as demonstrated by some commercial simulation question is what could we expect in the sequential case?
tools [14][15]. To exemplify this we cite one of the examples Sequential simulation is deterministic in the sense that two
given by Schriber and Brunner in 114). They describe how three executions of one simulation model with identical input produce
different commercial simulation packages make different identical output, i.e. the results are reproducible. This is an
assumptions about the treatment of simultaneous events and how important property when developing simulatioti models as it
this can affect the semantics of the model. facilitates debugging of the models as well as comparisons
Consider a manufacturing system simulation where a pan between different models of the same system. The reproducibility
(entity) releases a machine [resource) and then immediately re- stems from the fact that the order in which events with identical
competes for the machine, all at the same simulated time. The timestamps are retrieved lrom the event set (and executed) is
objective is to let a more highly qualified waiting part be the next determined by the real-time order in which the events were
to capture the machine; in the absence of such a part, the releas- inserted into the event set. This results in an implicit order by
ing part itself is tore-capture the machine. There are at least three which the events in the simulation can be complerely ordered.
possibilities for scheduling events that result in different policies: This is sometimes used in sequential simulation models to an
(1)Coupled with giving up of the resource is the immediate extent that they can rely on this implicit ordering of events for
their correctness. At this point we should note that the above

38
low4097/99 510.00 0 1999 IElm
described ordering of events not only preserves causal dependen- and process interactions, i.e. the sending and receiving of mes-
cies but also imposes an order on causally independent events sages. To obtain a panial ordering of the events, each Process P;
with identical timestamps. As indicated by the example in Sec- maintains a logical clock C,. which is incremented according to
tion I this is not always what the user wants. Consequently, most the following two rules:
sequential simulation systems provide mechanisms for the user to
I. Each process Pi increments Ci between any two succes-
control the ordering of causally independent events.
This can be contrasted to parallel discrete event simulation sive events.
(PDES) where the result of a simulation run is not necessarily 2. Each message m is timestamped with the logical clock
reproducible. That is, one cannot in general anticipate that two of the sending process (Pi). such that T,,,=C,. The receiv-
IU~S of the same simulation model with identical input generate ing process Pj then, upon receiving message m. sets its
identical output. This is due to that in PDES there exist no clock CC,)greater than or equal to its present value and
implicit real-time order in which events are generated. Further-
greater than T,,,.
more. the local causality principle 141,which sets the criteria for
This defines a partial order on the causal or “happened before”
correctness of a PDES system, does not state anything on the exe-
relation between events. The clock definition has the important
cution order of events with identical timestamps at an LP. Conse-
quently two events occurring at an LP with identical timestamps consistency property:
may be generated in and executed in different order in two other- IF a causally precedes b THEN C;<o> < Cj<b>
wise identical simulation experiments. This may in tom generate where C,<o> is the logical time of event o that occurred in proc-
different simulation results.
ess Pi. The scalar Lamport clock is however not strongly consist-
Determinism is not only useful for modelling purposes but
ent, that is:
also for the debugging of models. Consider state saving in opti-
mistic PDES. In most implementations that we are aware of, it is C,<a> < C,cb> does not imply that (I caused b
the responsibility of the application programmer to specify which Note that the Lamport clock is a different concept from the simu-
variables belong to the simulation state and need to be state lation time clock of DES, and is used here for a different purpose.
saved. The simulation kernel may then perform the state saving From now on, when we refer to a “logical clock” we mean a
in some more or less transparent fashion. If, however, the pro- mechanism used for ordering events with equal simulated time.
grammer mistakenly omits identifying one or more variables the To be able to completely order events, i.e. to break ties, when
results will be erroneous and indeterministic. If there is not a using Lampon clocks we need an arbitrary total ordering of the
reproducible event ordering. it is very hard to tell whether inde-
processes (s). For this purpose Lampon suggested to use,
terministic results are caused by the model itself or by omissions
unique, process identifiers. Thus, event o in P; is before event b in
in state saving. That is, the first case may be acceptable depend-
ing on the system under study, but the second case is not. Pj if and only if: either (i) C,<a> c Cpb> or (ii) C,a> = Cj<b>
To summarize. what we want from an event ordering scheme and Pi u Pj Because the Lampon clocks are not strongly consist-
for PDES is to be able to: i) provide determinism, ii) maintain ent we may order simultaneous events that are not causally
causality, iii) provide mechanisms for the user to order causally dependent. As we will show there may occur situations where we
independent events with identical timestamps at an LP, and iv) if want to be able to identify such events. To accomplish this we
possible assist the user in finding such events. In the following need a logical clock concept that captures strong consistency. In
we will mainly consider optimistic simulation systems as meth- Section 6 we proceed with a more detailed discussion on clocks
ods applicable in this setting also should be applicable to con- and ordering for PDES. For a good survey of diiferent representa-
servative simulation systems. tions of time in distributed systetns, see Raynal and Singhal [IZ].

3. Related Work 3.2. Event bundling

Most previous work on event ordering in PDES can roughly In the YADDES PDES system [I I1 all events occurring at a”
be categorized as: i) collecting or bundling simultaneous events LP with the same timestamp are guaranteed to be delivered in a
at an LP and then let the user handle them or ii) specifying a well- bundle (or %ent combination”) to the LP. It is then up to the
defined implicit ordering of the events. In many cases the work user to take care of the events. This approach offloads the whole
appears to having been influenced by an underlying assumption problem of handling simultaneous events onto the user. Moreo-
that “the user knows what to do since s/he knows the applica- YCT,it may caose problems in case of zero-delay cycles in a simu-
tion”. However, before moving on to what has been done in the lation model. In such a case, all events with identical timestamp
context of event ordering in a PDES context we should briefly cannot be generated beforehand nor delivered in a single bundle
review some of the theory developed for the ordering of events in to the LP.
distributed systems. Reiher et al. [ 131 implemented a similar scheme in the Time
Warp Operating System for ensuring determinism. They
3.1. Event ordering in distributed systems extended the event bundling scheme to allow the user to impose
an arbitrary ordering, or ranking, of simultaneous events. The
Lamport [61 has described schemes based on scalar logical ranking corresponds to a user defined event priority as may be
clocks for partial and total ordering of events in distributed sys-
found in sequential simulation tools to promote predictable tie-
tems. Events in this context are state transitions within a process

39
breaking. However, in this scheme there is no provisioning for 3.3. Event ordering approaches
guaranteeing causal correctness in rank. In the advent of zero-
Agre and Tinker [I] proposed a scheme for deterministic exe-
delay cycles in a model, the user may define ranking schemes for
cution of optimistic PDES based on extending the timestamps of
events resulting in effectively scheduling events “backwards in
events (messages). The single field of time, e.g. float time, is
time”. To explain the problem with the TWOS scheme’, let us replaced by the tuple:
first consider just imposing a ranking on events without bundling.
dime, rank, send-time, msg-ids
Let us assume that each LP is given a unique identity 1,2,... and
that events are ranked by the identity of the sending LP Funher- where
more, a lower ranking implies higher priority (i.e. should be han- msg-id = corder, sender-id. receiver-id>
dled first). Figure I illustrates a situation where LP 3 sends im In their scheme, two of the components, order and sender-id, act
event (LI) to LP 1 at time 1. LP 1 sends event b to LP 2 and LP 2 similarly to a Lampon clock. Thus if we do not consider the user
responds to LP I with event c. all at time 1. ls, denotes the times- defined ranking of events this scheme deterministically orders all
tamp (tuple) of event a Ordering the events by rank implies that events. However, the user defined rank of events takes prece-
LP 1 will interpret the situation as having received events (1 and c dence over the Lampon (logical) clock. Thus, we arrive back at
out of order. But, the culprit is actually LP I itself, having effec- the problem discussed previously where the user defined ranking
tively scheduled event b “backwards in time”. may cause violation of causal constraints.
Mehl [9] proposed a scheme that, at first glance, looks like a
LpI1\ stripped down version of Agre and linker’s scheme. A timestamp
is defined by the tuple <time. age. sender-id>. Where the age
ts, = <I, 3> UC = <t.2>
field resembles a Lampon clock with the important difference
i / b that it is only updated locally without taking received timestamps
LP2 I
into accooot. Consequently, there is no guarantee that the times-
tamps will be increasing. This is because causal relations with
events from other LPs are once again missed.
Lp3i We conclude that Reiher‘s and Mehl’s schemes fail to capture
causal dependencies between events that cause problems for opti-
Figure 1. Example of ordering with user ranking.
mistic synchronization. Agre and linker’s scheme, on the other
Adding event bundling to the scheme does not solve the prob- hand. can only be guaranteed to be correct if the user does not use
lem in an optimistically synchronized context since it is not pos- the ranking of events in these cases (the presence of zero-delay
sible to determine whether events arriving later (in real time), cycles in the model). Wielaod’s observations brings us some ose-
should be interpreted as stragglers, or if they were caused by a fol insights while his scheme raises similar concerns on how to
zero-delay cycle. If the late event, e.g. event c in the example, is a deal with user defined rankings of events and it is also not clear
straggler it should have been included in the last bundle (with how to handle causally dependent events with identical times-
event a), otherwise it should form a separate bundle. The problem ,ZXTlPS.
here is to uphold causal relationships between events when order-
ing them. In a sequential simulation it is obvious that event c can
4. Implementation of Lamport Clock Based
nwer precede event a since it is the execution of (I that (indi-
rectly) causes even, c.
Ordering in Time Warp Simulations
Wieland [15], argues that the user not always knows what to As a first step in ordering events we look at providing deter-
do with simultaneous events. He observed that some models may minism and have implemented a Lamport clock based scheme in
be highly sensitive to the ordering policy adopted in a way that is our Parallel Simulation Kernel (PSK). The PSK is a C++ imple-
not necessarily obvious to the user. Moreover, whether two mentation of the Time Warp mechanism [4] for shared or distrib-
events should be considered simultaneous ultimately depends on uted memory platforms and is described in some more detail in
the time resolution in measurements and modelling. Hence, he [S]. This first scheme is sufficient for models that can tolerate an
argues, the correct way of dealing with the problem is to take all arbitrary tie-breaking of events.
possible orderings into account when evaluating the output of the For an implementation on shared memory, we can define the
simulation. What is not clear in Wielaod’s proposal, however, is timestamp as the following tuple:
the treatment of simultaneous causally connected events. This is
<float ReceiveTime. unsigned int Order,
a problem that lies at the heart of some of the other work in this
unsigned int SenderLPida
area. Also related to this is the issue of progress. In [5] Jha and
where Order is generated by a Lamport Logical clock at the send-
Bagrodia investigate how user defined event ordering schemes
ing LP. We assume that all LP identities are unique. ‘limestamps
affect the lookahead needed to ensure progress in particular with
with identical Receivelime are ordered according to increasing
respect to conservative PDES.
Order and lastly increasing SenderLPidx This is equivalent to a
deterministic Lampon ordering with tie breaking. cf. Section 3.1.
1. One of the anonymous referees kindly informed us that the TWOS Thus, as long as the Order number counters are state saved in the
scheme was later mc&ied to overcome some of these problems. LPs this will create a reproducible outcome also in the presence
Unfortunately, documentation of this is not readily available and we
have not yet been able to access it. of rollbacks. However, it should be noted that we are not guano-

40
teeing that all existing even& at any particular time have unique 5. The Cost of Determinism
timestamps. There may be, due to rollbacks, multiple events en
route with the same receive time-stamp. However, the duplicates Using the PSK and the implementadon of Lamport clock
are results of reexecution of the same even, and consequently will event ordering we conducted some experiments to study the per-
eventually be cancelled. Hence, we canno, use the timestamps, as formance cosfs associated with this scheme. These experiments
presented above, to uniquely identify events during the course of serve to give us an idea about a lower hound on costs. Experi-
execution. On a shared-memory machine, where direct cancella- ments were carried out on an 8 processor SUN SparcCenter 2CQO
tion [2] can be used this is not a problem since pointers are used with 512 Mbytes of memory. Version 4.2 of SUN’s C++ Sparc-
to refer to the even, to be cancelled. compiler was used with the “-0” switch on. The current PSK
On distributed memory we use an additional componen, in the implementation uses a scheduling queue for LPs and even,
time-stamps to help us distinguish between multiple generorions queues in each LP. Both types of queues are based on linked lists
of events: at present.
In the firs, se, of experiments. “SelfPing”. a single LP repeat-
<float ReceiveTime, unsigned int Order, unsigned int
SenderLPid. unsigned in, Generation> edly sends an event 10 itself. Consequently. the main part of the
work is merely in creating the event, insening it in the event
The Generation counter is updated between each message send
queue and rescheduling the LP. The time increment between each
and is not state saved. Consequently, different generations of the
event is one and the message Population is one. Consequently,
same even, will have different Generation markings.
there are no simultaneous events. I, provides the very minimum
In the PSK implementation we introduce a base class TimeS-
event grain, and this gives an indication of the minimum over-
tamp that implements this default ordering scheme. It holds the
head associated with sending an even,. The even, rate without
rimestamp tuple and defines the comparison operators for TimeS-
event ordering is around 23Mw) events& giving an overhead in
tamps <. ==, >. Upon creation of a TimeStamp object the Order
the order of 44 ps. Results shown in Table I indicate that the cost
and Generation fields must be appropriately se,.
for ordered timestamps is relatively large, as much as 17%. Suc-
To avoid implementation problems with the order counter
cessively adding timestamping code indicated that a large part of
rolling-over to zero, it is possible to reset it when simulated time
the added cost, more than half, stems from simply encapsulating
has increased. That is, when there is a time difference, locally and
the (float) timestamp into a class and adding the extra data fields.
in received timestamps (lookahead). causality is maintained with-
The rest comes from logical clock updates and comparison oper-
out any logical clock. However, to preserve uniqueness of times-
ator code. Furthermore, the experiments indicated a sensitivity to
tamps we substitute Order by <SendTime, O&D. Thus, we can
inlining of operators such that inlining of operators could, in
use the tuple:
some cases, significantly degrade perfomxmce.
<float ReceiveTime. float SendTime, unsigned in, Order,
unsigned in, SenderLPid, unsigned int Generation>
The order counters are updated when an even, starts executing:
TABLE 1. SelfPing execution times on 1 PE.
IF ReceiveTimewn,L,f >
for 2000000 events. 5 experiments

ReceiveTimew,,t,nt z SendTime,,,c,,
THEN Order:= 0
ELSE Order:= m&Order. Ord~,,,~,,) + 1
Since Order essentially is a Lamport clock it is incremented after
each message sent. This also assumes that the Order counfer is
state saved and that the time of the last received message can be
retrieved after a rollback, e.g. through state saving. The correct- Next a se, of experiments using a Parallel Hold (PHold) [3]
ness of this is based on the fact that ordering by the ReceiveTime model were run. This consisted of 64 LPs in a torus configuration
and SendTime fields, when strictly increasing, is causally correct passing received messages on (with an exponentially distributed
and supersedes the logical clock ordering. delay with mean I) 10 one of the four neighbours chosen at ran-
Once we have imposed a total event ordering we can also dom. The message population was constant a, 640 messages in
make the following observation concerning lime Warp execu- total. ‘Ihe probability for simultaneous even& is very small, so
tions: We know fha, each timestamp in a committed event is again this provides only a lower bound on costs. Results for
unique. Thus. whenever we encounter multiple events with the PHold on 8 processors are shown in Table 2. Costs are similar to
same timestamp, apan from the Generation field, we know that the previous case. The slight increase in event granularity is
the older even, is “stale”. That is. it is the result of a rolled back likely 10 be offset by more calls to the timestamp comparison
execution where the antimessage has not caught up yet. By avoid- operators, resulting in similar or somewhat larger relative cost of
ing to execute the stale even, during reexecution (either by block- timestamp ordering. Efficiency, i.e. the fraction of processed
ing the LP or skipping the event), we can avoid some transient events that are eventually committed, is also shown. Since no
error states. Transient error states is one of the peculiarities of additional ordering is expected 10 he needed there is practically
optimistic synchronization that can cause problems when paral- no significant change to the rollback behaviour. The efficiency
lelizing a model [IO].

41
 results should be seen as approximations as they are based on the 6. Event Ordering with Priorities
mean of the efficiency (fractions) reported by each replication.
The question we set out to answer is: Can we produce a user-
defined priority ordering (ranking) of events where the system
checks that causal constraints are maintained? This is the func-
TABLE 2. PHold execution limes on 8 processors,
tionality we get in sequential simulation, so unless there is some
64 LPs and 10 messages/LP. 5 replications
fundamental problem we would expect nothing less from a dis-
of 5000 units of simulated time each. tributed simulation environment.
However, allowing user defined ranking of events is not
““OKlC,Cd Ordered uncomplicated even in the sequential case. Two examples illus-
Time Time Change
Stamps Stamps trating this are found in Figure 2. In both examples we consider
user defined priority ordering of events with equal simulation
Execution lime [s] 150.4 183.3 +22% time. In the first case (a), event a schedules an event c with higher
90% conf. int. (+/-) 3.2 9.6 - priority than that of event b (a high priority corresponds to a low
Efficiency numerical value cf. Figure 1). Thus event c should be executed
71% 73% +2%
before event b. In a sequential simulation the execution order of
90% conf. int. (+/-) 0.6% 1.4% the events will not necessarily capture the intentions of the simu-
Finally, experiments were carried out with a wireless cellular lation modeller as event b may executed before event a. In con-
network simulator “MobSim”. The simulator models radio trast a PDES execution based on lime Warp of the same example
resource allocation in an FfTDMA based system. This encom- would always result in the intended schedule. Example(b) in Fig-
passes systems such as the GSM cellular phone system. More ure 2 extends the first example by event b scheduling an event d
details on the model can be found in (71.7he important aspects of for LPI with a higher priority than that of event a. lo this case we
the model, for the purposes of this study, are the following: The cannot find a schedule which will both satisfy the priority order-
simulated system consists of 81 base stations and a total of 90 ing and the causal ordering of the events. These problems stem
radio channels. Each radio channel constitutes an LP. Mobile sta- from that the user effectively attempts to schedule events back-
tions roam the region covered by the system and connect to the wards in such a way that they become circularly dependent on
base stations. When selecting B new radio channel for a mobile each other. A sequential simulation. in which we have global
station, a query is broadcasted to all potential channel LPs. Using control, could resolve this by arbitrarily breaking the priority
90 channels and a cluster size of 9, the query would be broadcast ordering by executing either o or b first. A case like this could,
to IO other LPs. The channels reply back at the same simulated however, be detrimental to a PDES simulation which either
time. Hence. when using a total event ordering scheme, all replies would deadlock (conservative) or livelock (optimistic). A simple
will arrive in the same order in different executions. Although, solution to these problems is not to allow the user to increase the
not forming a stress case this demonstrates a more realistic sce- priority of successive events with equal simulated time, which
nario where the mechanism is invoked to arbitrate some of the will prohibit the user from scheduling events backwards in time.
events. In the simulations performed here about one in eight com- ts, = <t. 21 ts, = <t, 2> \
mitted events required tie breaking. Results are shown in Table 3.
The event granularity and state saving overheads are substantially
larger in this case, granularity and state saving Lime are both in
the order of 200 ps. For this reason we might expect the cost of
timestamps to be negligible. However, since the reply messages
are now ordered, there are more rollbacks (as indicated by a drop
in efficiency) which induces an additional cost.
64
Figure 2. Examples
(b)
of possibly incorrect event ordering
TABLE 3. MobSim execution times on 8 processors.
5 replications of 2000 seconds (simulated 6.1. Causality in DES
time) each. Before proceeding, we need to revisit the concept of causality in
DES to create a firm foundation from where we can continue
““ordered Ordered exploring the problems of event ordering. Foremost, we need to
Time Time Change understand and define the causality concepts underlying DES.
Stamps Stamps Using this definition we can define classes of timestamp mecha-
Execution lime Is] 95.6 109.5 +14% nisms that capture our causality concept to different degrees and
90% conf. int. (+I-) analyse their properties. Based on this we can conrinue to study
2.0 9.0 -
problems such as those discussed in the previous section and pro-
Efficiency 78% 71% -7% ceed to investigate mechanisms that allow user defined ranking of
90% conf. inf. (+/-) 0.7% 0.4% events as well as the issue of identification of simultaneous
C”C”1S.

42
 In DES events are primarily ordered by simulation time. Sim- eny which may be useful in the context of conservafive PDES
ulation time does not define the causal constraints, ra,ber the way methods. A timestamping scheme is srmngly consisrenr if:
we can assign simulation time to events has 10 reflect the underly-
ing causal constraints. These causal requirements are tha,: i)
(generarim dependencyJ if one even, e, genera,es even, eZ Lhen Given a strongly consisten, timeslamping scheme one can dew-
e, muff be executed before Ed, and; ii) @rare depmdenq) an mine causal independence simply by inspecting ,he timestamps
of the events. Determinism results from a timestamping scheme
event e m”s, be executed after all e”en,s that could have affected
which is both consistent, according lo some of the definitions
,he state from which e was generated. A more formal definition
above. and which genera,es unique timestamps.
of ,his causality concept is given in the Appendix. Consequently,
Let us revisit the examples of Figure 2 to identify the causes
we propose 10 use the following causalily constrain, when order-
of the problems encoun,ered. Applying our causal definition 10
ing simultaneous events:
lhese examples we find the following causal dependencies in (a):
o + b iff me execution 01 even, a (direcfly or indirectly)
a-+c andin( o-+c+d+c and b+d+c+d.Tlmt
generated even, b or could have atfected the
is. in example (b) we have imroduced circular dependencies
state from which bwas generated
which implies violation of the causal cons,rain,s. The problem is
We define causal independence as: that if we allow case (a) we also must be prepared ,o deal with
ollbilfnoto+b andnotb+o case (b) as we allow timestamps 10 be selected such ,hat ,s, > ,sb
though a + b Tba, is, the timestamp scheme is no, even mini-
To exemplify this notion of causality, consider the evenrs
mal consistent according 10 our definitions. Such a scheme could
depicted in Figure 3a. From the causal constrain, follows that
work in a sequen,ial simulation with global comrol as one may
went c depends on both even, (1 and even, b.
break the circular dependency in such way that the end result
appears 10 be acceptable 10 the user. Though this is nothing we
endorse. In PDES we do no, have global control mechanisms but
L”l’+b b+c 11x1 dlle use the “local causality principle” (41, which relies on that rhe
timestamps are causally consistent for its correctness. Conse-
quen,ly, we require that the timestamping mechanism used in
PDES is a, leas, minimally consistent.
k=) (b)
Figure 3. Causal dependencies 6.2. A user controlled ranking scheme
In analogy ,o logical clocks for distribuled systems we can One of our initial in,en,ions was to find mechanisms that
identify different classes of logical clocks, or timestamp mecha- allow user controlled rankings of cvco,s. A straigh,-forward soIu-
nisms, based on the degree ,o which ,hey reflect our causal defini- lion 10 accomplish this is fo ex,end the timestamp tuple defined in
tions. The firs, of these is based on the minimal timestamping Section 4 by a user defined priority, Rank, where a lower numeri-
constrain, normally cncoumered in DES which swes ,ha, an cal value indicates a higher priority. As ,he user ranking is ,o take
even, cannot have a lower limeslamp than any of its predecessors. precedence over the Lampon clock ordering we place Rank
i.e. that “time” cannot go backwards. Thus, we call such times- before Order in the tuple. For the shared memory case we get ,he
tamping mechanisms minimally consirrenr: following timeslamp tuple:
-&at ReceiveTime. unsigned in1 Rank. unsigned int Order,
unsigned int SenderLPidz
We note tha, this does no, allow us to correctly order causally
This resembles the scheme proposed by Agre and Tinker [I].
dependent evenfs solely by the timestamps. The reason that mini-
However, based on our discussion in the previous section we
mol consis,ency in general is sufficient for causal correcmess is
require ,he timeslamps 10 be minimolly consisten,, i.e. non-
that parts of the causality concept is implicitly captured in the
decreasing. This is accomplished by imposing restrictions on the
even, scheduling mechanism. In general this guarantees that each
user defined rankings. These restrictions could easily be enforced
LP never schedules a no, ye, execuled even, before any already
by ,he simulations system, which preferably could notify ,he user
executed even, with equal or lower ,imes,amp. In other words, an
whenever i, detects any violation. The following should hold in
event will never be scheduled before any even, on which it may
Ihc case an even, o directly generates another even, b:
be causally dependent. This is an inherent property found in ordi-
if ReceiveTime, -z ReceiveTimeb then Rank+ can be se,
nary sequential simulation and most PDES systems.
freely
Analogous to logical clock schemes for diswibuted systems
else if ReceiveTime, = ReceiveTimeb then Rank, > Rank,
we say that a timestamp scheme is weakly consistenr (cf.
Section 3.1) if: It is evident that this scheme is minimoify consistent for the cases
where events are causally dependent and have differen, simula-
a + b a ts, < lsb tion time (ReceiveTime) and for events that are generation
Thus, a weakly consistent timestamping scheme guarantees that depcnden,. Finally, we consider the case where an even, a has
there will bc no zero-delay cycles in a simulation model. A prop- stale causal dependencies to some even,~ wilh equal simulation
time, call this set of events E. These events must, by definition,
have been executed before the even, lhat genera,ed 0. Thus, any

43
 event that (I has a srare causal dependency to mutt have a lower or (i.e. (1 or b) and search backwards in the dependency graph for
equal timestamp than the evenl that generaed a Consequently, the other event, i.e. the potential causal predecessor. 7he search
we know that o must have at Ieat as high timestamp as any of the in a branch of the dependency graph can be terminated if we find
evenls in E and that the ranking scheme is minimally consishu the event we are searching for or when an event with a lower
also in this case. As the Lampon clock (Ihe Order field) will guar- Order field than that of the potential predecessor is encountered.
antee that timestamps will be increasing along any causal chain As we are only interested in finding simultaneous events we
(i.e. any chain of events that are causally dependent) this scheme need only construct the dependency graph for events with equal
is also wok/y consistent. simulation time (ReceiveTime). However, in many practical cases
This scheme has the benefil of adding little overhead com- the usefulness of building and searching dependency graphs will
pared to the scheme in Section 4. However, it does not capture be limited due lo time and memory overheads as these grow with
strong consistency. the number of events that have equal simulation time.
Al~ematively we can define a logical clock concept that cap-
6.3. Identification of simultaneous events tures srmng consistency. Such a device could be defined in anal-
The last problem we set out to study is how to identify simul- ogy2 to vector clocks for distributed systems [I?.]. The cost for
taneous events. By simultaneous events we mean causally inde- such a scheme is largely determined by the fact that veaor clock
pendent events occurring at the same LP with equal simulation timestamp has one entry for each process in the system. Thus the
time (ReceiveTime) for which the user has not defined a rank or size of the timestamp and the complexity of comparison opera-
for which the rank is equal. This could be used to: i) inform the tions on it grows with the number of LPs.
user of possible “race-conditions” in the model: ii) that arbitrary
tie breaking has been enforced or; iii) to explore differem possi- 6.4. Permutation of simultaneous events
ble event permutations. As previously mentioned, knowledge on simultaneous events
By definition a causal dependency cannot exist unless the could be used to exec~le (any possible) pemmtations of such
causing event has been executed. Thus, all future events with events. The reason to do this is that the ordering of such events
identical simulation time (ReceiveTime) and Rank at an LP are could have a large impact on the simulation output in some cases
simultaneous. In the general case this is not sufficient to base a [IS]. First we note that the simultaneous events that can be identi-
scheme to identify all simultaneous events on. Consider the fied depend on the schedule of events already processed. Thus, if
example in Figure 4. Assume that event e and f have equal simu- simultaneous events are permutated at one LP this could affect
lation time and rank If event e has already been executed when the set of simultaneous events at another LP. Consider the exam-
event f arrives we need further information on their causal ple in Figure 5 where all events occur at the same simulation
dependencies to identify that e did not causef: Thus we need to time. Permutating the evems at LPI (a) will result in that we
find methods to trace causal dependencies. This could be accom- affect the dependencies at LP2 resulting in the situation in (b).
plished either by tracing the causal dependencies in the form of a Attempting to pennutate events at LPI and LP2 (Figure 5 (a)) in
graph or by inventing a strongly consistent logical clock. parallel could lead to causally inconsistent schedules, i.e. this
The causal dependencies of an event form a directed acyclic could lead to the situation in Figure Z(b). Consequently. permuta-
graph (unless some causal constraint is violated). This graph is tion of simultaneous events has to be serialized. and performed at
formed in the following way. Each event has a chain of generat- one LP at a time. Moreover, we nae that if we want to be able to
ing predecessors (i.e. the generarion dependencies). The genera- permutate already executed events we need some kind of rollback
tion dependency chains may in turn be inter-connected by state mechanism to roll back the state of the simulation lo a situation
dependencies, see example in Figure 4. where these events are unexecuted.

e f
generation dependency ---)
Figure 5. Simultaneity depends on event execution order
state dependency
Figure 4. Causal dependency graph 7. Conclusions
Two events o and b are causally independent if one do not find
In this paper we have studied event ordering in PDES. Our
a when searching the dependency graph backwards from b and
goal has been to find event ordering schemes which allow us to
vice versa. The search can be limited if the Lampon clock (the
give the same support and functionality as a user would expect
Order field) is utilized. The Lamport clock is weakly consistent
with our causal definition, that is if o--f b then we know that
Order, c Orderb As a consequence. we can use the following
2. N.B. WCuse a slightly different causality concept. Ii~s, vecmr clocks
search strategy Start from the event with the highest Order field 8s defined for general diso-ibuted systems are not direcdy applicable to
DES.

44
from a sequential simulation system. That is we wan, to: i) pro- IO. Nicol, D. M. and Liu, X. 1997. The Dark Side of Risk, In Pro-
vide determinism, ii) mainlain causali,y. iii) provide mechanisms ceedings of the I Ilh Workshop on Parallel and Distributed Sim-
for the user to order causally independent events with identical ulation (PADS97), pages 188-195.
simulation time a, an LP. and iv) if possible assist ,he user in fmd- I I. Preiss. B. 1989. The Yaddes Distributed Discrete Even, Simula-
,ion Specification Language and Execution Environment, In
ing such events ,o take appropriate actions. Seemingly simple,
Proceedings of the SCS Multiconference on Dislriboted Simu-
lhis is a problem which turns out to be more complex than what la,ion. pages 139.144
could be andcipated at firs, glance. This is exemplified by the fact 12. Raynal, M. and Singhal. M. 1996. Capruring Causality in Dis-
that in our review of existing schemes we have failed to find tribuwd Systems. IEEE Computer 29(2). 49-56.
schemes which provide the desired functionality. Inslead we have 13. Reiher, P., Wieland, F. and Hontalas. P. 1990. Providing Deter-
found that several of the proposed schemes are no, useful as it is minism in the Time Warp Operating System -Costs, Benefits.
easy 10 cons,ruct examples where they fail to maintain causality, and Implications. Proceedings of the Second IEEE Workshop
panicularly in ,he con,ex, of optimistic PDES. In this paper we ,
on Enoerimemal Distribuled Svstems. Huntsville.~,~~~~~~~~
.
Alabaman
pages i13-II8
have proposed a definition of causality in DES. Based on this we 14. Schriber. T. and Bmnner, D. 199s. Inside Simula,ion Sofware:
have devised two schemes for dewministic. causally correct, How it Works and Why it Matters. Proceedings of the 1995
ordering of even& in PDES based on exisling theory on event Winter Simolalion Conference, Washington DC. pages I IO-1 I7
ordering in distributed systems. The more elaborate scheme 15. Wieland, F. 1997. The Threshold of Even, Simultaneity. Pro-
allows the user lo specify the ordering of simultaneous events by ceedings of dx I Ith Workshop on Parallel and Distribuled Sim-
assigning priorities to events. We have also discussed mecha- ulation (PADS’97). Lockeohaus, Austria. pages 56-59
nisms Lo detect and take appropriate action on causally independ-
ent events which are not ordered by ,he user, i.e. what could be
considered as “race conditions” in the model. The associated Appendix
costs for these schemes are no, negligible as our experimental Definition of the DES causality notion. Events that are not
studies indicate. However, we believe ,hat it is crucial for PDES dependent according to this relation can be reordered by user
systems 10 provide even, ordering mechanisms similar to those ranking. By stare. in this description, we refer to the state a, a
encouolered in sequential simulation. unique time-point in the execution path of an LP. Thus, a state in
this sense can never be repeated and is consequently unique,
8. References which is not tree in the usual sense of the term.
I Agre, J. and Tinker, P. 1991. Useful Extensions to a Time Warp
Simulation Sys,em. Proceedings of the SCS Multiconference Causality relation:
on Advances in Parallel and Dislributed Simulation, Anaheim, I. Consider event generation function 8 and new state fonc-
California. pages 78-85 tion n Then g(e, 5) = E’ means ,hat even, e executed at
2 Fujimoto. R. 1989. Time Warp on a Shared Memory Mulliproc-
essor. Transactions of the Society for Computer Simulation 6, state s generated the se, tZ of events.
3. July 1989. pages 21 l-239. Similarly n(e, s) = 5’ means that event e executed a, state
3 Fujimoto. R. 1990. Perfomnnce of Time Warp under Synthetic s resulted in the new stafe s’
Workloads. In Proceedings of the SCS Multiconference on Dis-
2. Define the generation dependency relations ; as:
tributed Simulation. pages 23-28.
4 Jefferson, D. 1985. Vinual Time. ACM Transaclions on Pro- If g(e,s) = f? then V~‘E ,?‘=,a/.
gramming Languages and Sys,ems, Vol. 7, No. 3. July 1985. Define the state dependency relations ; as:
pages 404-425. If n(e, s) = s’ then s:s’
5 Iha, V. and Bagrodia. R. 1996. Simultaneous Events and
Look&ad in Simulation Protocols. UCLA CSD Technical Define event-state interdependencies b, , Te as:
Repon 960043. If n(e. s) = s’ and g(e’, s’) = E” thin
6 Lampon, L. 1978. Time Clocks and the Ordering of Even& in a e-s’ and ‘V’e” E I? =a s’+e”.
Distributed System. Communications of the ACM. 21(7), 558. e, * s, e
565. 3. Causal dependency -i:
7 Liljenslam, M. and Ayani, R. 1996. A Model for Parallel Simu- Events e’ --t e” iff there is a transitive dependency
lation of Mobile Telecommunication Systems. Proceedings of from e’ to e” using 2 , ;, >, , and +
the International Workshop oo Modeling, Analysis, and Simo- *, e
lation of Computer and Telecommunication Sys,ems (MAS- Consequently, an went e’ depends on the event e that (directly
CDTS’96). San JosC,CA. pages 168-173
8 Liljenstem, M. and Ayani, R. 1997. MobSim++, An Environ- or indirectly) generated e’ and all events that could have affected
men, for Parallel Simulation of PCNs. Proceedings of the the slate s a, which P’ was generated.
World Conaress on Svstems Simuladon (WCSs’97) (2nd Joint
Conferenc<of ln,e&,ional Simulation Societies),~Singapore.
pages 272-28 I.
9, Mehl. H. 1992. A detenoinislic Tie-Breaking Scheme for
Sequemial and Distributed Simulation, lo Proceedings of the
6,h Workshop on Parallel and Distributed Simulation
(PADS’92). pages 199.200.

45