DISTRIBUTED SYSTEMS

Principles and Paradigms

Second Edition
ANDREW S. TANENBAUM
MAARTEN VAN STEEN

Chapter 6
Synchronization
Dr. Saeed Mahmud Ullah
Professor
EEE, DU
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Why Synchronize?
 Often important to control access to a single,
shared resource.
 Also often important to agree on the ordering of
events.
 Synchronization in Distributed Systems is much
more difficult than in uniprocessor systems.
 Clock Synchronization

Figure 6-1. When each machine has its own clock, an

event that occurred after another event may
nevertheless be assigned an earlier time.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 When the programmer has finished changing all the source
files, he runs make (in Unix), which examines the times at
which all the source and object files were last modified.
 If the source file input. c has time 2151 and the
corresponding object file input.o has time 2150, make knows
that input.c has been changed since input.o was created, and
thus input.c must be recompiled.
 On the other hand, if output.c has time 2144 and output.o has
time 2145, no compilation is needed.
 Thus make goes through all the source files to find out
which ones need to be recompiled and calls the compiler to
recompile them.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Now imagine what could happen in a distributed system in
which there were no global agreement on time.
 Suppose that output.o has time 2144 as above, and shortly
thereafter output.c is modified but is assigned time 2143
because the clock on its machine is slightly behind, as
shown in Fig. 6-1. Make will not call the compiler. The
resulting executable binary program will then contain a
mixture of object files from the old sources and the new
sources.
 It will probably crash and the programmer will go crazy
trying to understand what is wrong with the code.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Clock Synchronization
 Synchronization based on “Actual Time”.
 Note: time is really easy on a uniprocessor system.
 Achieving agreement on time in a DS is not trivial.
 Question: is it even possible to synchronize all the
clocks in a Distributed System?
 With multiple computers, “clock skew” ensures that
no two machines have the same value for the “current
time”. But, how do we measure time?

In practice, when a system has n computers, all n crystals will

run at slightly different rates, causing the (software) clocks
gradually to get out of synch and give different values when read
out. This difference in time values is called clock skew.
 How Do We Measure Time?
 Turns out that we have only been measuring time
accurately with a “global” atomic clock since Jan. 1st,
1958 (the “beginning of time”).
 Bottom Line: measuring time is not as easy as one
might think it should be.
 Algorithms based on the current time (from some
Physical Clock) have been devised for use within a
DS.
 Atomic clock
 With the invention of the atomic clock in 1948, it became possible to
measure time much more accurately, by counting transitions of the cesium
133 atom.
 The physicists took over the job of timekeeping from the astronomers and
defined the second to be the time it takes the cesium 133 atom to make
exactly 9,192,631,770 transitions.
 The choice of 9,192,631,770 was made to make the atomic second equal to
the mean solar second in the year of its introduction.
 Currently, several laboratories around the world have cesium 133 clocks.
 Periodically, each laboratory tells the Bureau International de l'Heure (BIR)
in Paris how many times its clock has ticked. The BIR averages these to
produce International Atomic Time, which is abbreviated TAI.
 Thus TAI is just the mean number of ticks of the cesium 133 clocks since
midnight on Jan. 1,1958 (the beginning of time) divided by 9,192,631,770.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Although TAI is highly stable, there is a serious problem with it; 86,400 TAI
seconds is now about 3 msec less than a mean solar day (because the mean
solar day is getting longer all the time).
 BIR solves the problem by introducing leap seconds whenever the
discrepancy between TAI and solar time grows to 800 msec.
 The use of leap seconds is illustrated in Fig. 6-3. This correction gives rise to
a time system based on constant TAI seconds but which stays in phase with
the apparent motion of the sun. It is caned Universal Coordinated Time, but
is abbreviated as UTC. UTC is the basis of all modern civil timekeeping.
 To provide UTC to people who need precise time, the National Institute of
Standard Time (NIST) operates a shortwave radio station with call letters
WWV from Fort Collins, Colorado. WWV broadcasts a short pulse at the
start of each UTC second. The accuracy of WWV itself is about ±l msec

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
Figure 6-3. TAI seconds are of constant length, unlike solar
seconds. Leap seconds are introduced when necessary to
keep in phase with the sun.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
Clock Synchronization Algorithms
 If one machine has a WWV receiver, the goal becomes keeping
all the other machines synchronized to it.
 If no machines have WWV receivers, each machine keeps track
of its own time, and the goal is to keep all the machines together
as well as possible. Many algorithms have been proposed for
doing this synchronization.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
Clock Synchronization Algorithms
 Each machine is assumed to have a timer that causes an
interrupt H times a second. When this timer goes off, the
interrupt handler adds 1 to a software clock that keeps track of
the number of ticks (interrupts) since some agreed-upon time
in the past. Let us call the value of this clock C.
 When the UTC time is t, the value of the clock on machine p is
Cp. In a perfect world, we would have Cp = t for all p and all t.
 In other words, Cp= dCldt ideally should be 1.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
Clock Synchronization Algorithms

Figure 6-5. The relation between clock time and UTC

when clocks tick at different rates.
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Network Time Protocol
 A common approach in many protocols and originally
proposed by Cristian (1989) is to let clients contact a time
server.
 A time server can accurately provide the current time, for
example, because it is equipped with a WWV receiver or an
accurate clock.
 The problem, of course, is that when contacting the server,
message delays will have outdated the reported time. The
trick is to find a good estimation for these delays.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Network Time Protocol

Figure 6-6. Getting the current time from a time server.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 A will send a request to B, timestamped with value T1.
 B, in turn, will record the time of receipt T2 (taken from its own
local clock), and returns a response timestamped with value T 3,
and piggybacking the previously recorded value T2.
 Finally, A records the time of the response's arrival, T4.
 Let us assume that the propagation delays from A to B is roughly
the same as B to A, meaning that T2-T1 =T4-T3- In that case, A can
estimate its offset relative to B as

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 In the case of the network time protocol (NTP), this protocol is set up pair-
wise between servers. In other words, B will also probe A for its current time.
The offset δ is computed as given above, along with the estimation δ for the
delay:

 Applying NTP symmetrically should, in principle, also let B adjust its clock to that of
A.
 However, if B's clock is known to be more accurate, then such an adjustment would
be foolish. To solve this problem, NTP divides servers into strata.
 A server with a reference clock such as a WWV receiver or an atomic clock, is known
to be a stratum-I server (the clock itself is said to operate at stratum 0).
 When A contacts B, it will only adjust its time if its own stratum level is higher than
that of B.
 After the synchronization, A's stratum level will become one higher than that of B. In
other words, if B is a stratum-k server, then A will become a stratum-(k+l) server
 If A's stratum level was lower than that of B, B will adjust itself to A

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 The Berkeley Algorithm
 Here the time server (actually, a time daemon) is active, polling
every machine from time to time to ask what time it is there
 Based on the answers, it computes an average time and tells all
the other machines to advance their clocks to the new time or slow
their clocks down until some specified reduction has been
achieved.
 This method is suitable for a system in which no machine has a
WWV receiver. The time daemon's time must be set manually by
the operator periodically.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 The Berkeley Algorithm (1)

Figure 6-7. (a) The time

daemon asks all the other
machines for their clock
values.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 The Berkeley Algorithm (2)

Figure 6-7.
(b) The machines answer.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 The Berkeley Algorithm (3)

Figure 6-7. (c) The time

daemon tells everyone how
to adjust their clock.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
Clock Synchronization in Wireless Networks:
RBS protocol
 Reference broadcast synchronization (RBS) is a clock
synchronization protocol that is quite different from other
proposals
 The protocol does not assume that there is a single node with
an accurate account of the actual time available. Instead it
aims at internally synchronizing the clocks, just as the
Berkeley algorithm does.
 Second, the solutions we have discussed so far are designed to
bring the sender and receiver into synch, essentially following
a two-way protocol. RBS deviates from this pattern by letting
only the receivers synchronize, keeping the sender out of the
loop.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 In RBS, a sender broadcasts a reference message that will allow its
receivers to adjust their clocks.
 A key observation is that in a sensor network the time to propagate a
signal to other nodes is roughly constant. Propagation time in this
case is measured from the moment that a message leaves the
network interface of the sender.
 As a consequence, two important sources for variation in message
transfer no longer play a role in estimating delays: the time spent to
construct a message, and the time spent to access the network. This
principle is shown in Fig. 6-8.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Clock Synchronization in Wireless
Networks (1)

Figure 6-8. (a) The usual

critical path in determining
network delays.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Clock Synchronization in Wireless Networks:
RBS (Reference broadcast synchronization)

Figure 6-8. (b) The

critical path in the
case of RBS.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 The idea underlying RBS is simple:
 when a node broadcasts a reference message m, each node p simply
records the time Tp,m that it received m. Note that Tp.m is read from p' s local
clock.
 Ignoring clock skew, two nodes p and q can exchange each other's delivery
times in order to estimate their mutual, relative offset:

where M is the total number of reference messages sent. This information is important:
node p will know the value of q's clock relative to its own value. Moreover, if it
simply stores these offsets, there is no need to adjust its own clock, which saves
energy

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Unfortunately, clocks can drift apart. The effect is that simply computing the
average offset as done above will not work: the last values sent are simply
less accurate than the first ones. Moreover, as time goes by, the offset will
presumably increase.
 Elson et al. use a very simple algorithm to compensate for this: instead of
computing an average they apply standard linear regression to compute the
offset as a function:

The constants α and β are computed from the pairs (Tp,k,Tq,k)

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 LOGICAL CLOCKS
 Clock synchronization is naturally related to real time.
 However, it may be sufficient that every node agrees on a current time,
without that time necessarily being the same as the real time.
 For running make, for example, it is adequate that two nodes agree that
input.o is outdated by a new version of input.c. In this case, keeping
track of each other's events (such as a producing a new version of
input.c) is what matters. For these algorithms, it is conventional to
speak of the clocks as logical clocks.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Lamport’s Logical Clocks (1)
 To synchronize logical clocks, Lamport defined a relation called
happens-before.
 The "happens-before" relation → can be observed directly in two
situations:
 If a and b are events in the same process, and a occurs before b, then
a → b is true.
 If a is the event of a message being sent by one process, and b is the
event of the message being received by another process, then a → b
 If a  b and b  c, then a  c.
 If two events, x and y, happen in different processes that do not exchange
messages (not even indirectly via third parties), then x y is not true, but
neither is y  x. These events are said to be concurrent, which simply
means that nothing can be said (or need be said) about when the events
happened or which event happened first.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 What we need is a way of measuring a notion of time such that for every event,
a, we can assign it a time value C(a) on which all processes agree. These time
values must have the property that if a  b, then C(a) < C(b).
 If a and b are two events within the same process and a occurs before b, then
C(a) < C(b).
 Similarly, if a is the sending of a message by one process and b is the reception
of that message by another process, then C (a) and C(b) must be assigned in
such a way that everyone agrees on the values of C (a) and C(b) with C(a) <
C(b).

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Consider the three processes depicted in Fig. 6-9(a).
 The processes run on different machines, each with its own clock, running at
its own speed.
 When the clock has ticked 6 times in process P1, it has ticked 8 times in
process P2 and 10 times in process P3. Each clock runs at a constant rate, but
the rates are different due to differences in the crystals.
 At time 6, process P1 sends message m1 to process P2. How long this
message takes to arrive depends on whose clock you believe. In any event, the
clock in process P2 reads 16 when it arrives. If the message carries the starting
time, 6, in it, process P2 will conclude that it took 10 ticks to make the
journey.
 According to this reasoning, message m2 from P2 to P3 takes 16 ticks
 Now consider message m3. It leaves process P3 at 60 and arrives at P2 at 56.
Similarly, message m4 from P2 to P1 leaves at 64 and arrives at 54. These
values are clearly impossible. It is this situation that must be prevented.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Lamport’s Logical Clocks (2)

Figure 6-9. (a) Three processes, each with its own clock.
The clocks run at different rates.
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Lamport's solution follows directly from the happens-before
relation.
 Since m3 left at 60, it must arrive at 61 or later. Therefore, each
message carries the sending time according to the sender's clock.
When a message arrives and the receiver’s clock shows a value
prior to the time the message was sent, the receiver fast forwards
its clock to be one more than the sending time. In Fig. 6-9(b) we
see that m3 now arrives at 61. Similarly, m4 arrives at 70.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Lamport’s Logical Clocks (3)

Figure 6-9. (b) Lamport’s algorithm corrects the clocks.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 What follows is typically part of the middleware layer.
 To implement Lamport's logical clocks, each process Pi
maintains a local counter G. These counters are updated as
following steps
 Before executing an event (i.e., sending a message over the
network, delivering a message to an application, or some
other internal event), Pi executes G= G + 1.
 When process Pi sends a message m to Pj it sets m’s
timestamp ts (m) equal to G after having executed the
previous step.
 Upon the receipt of a message m, process Pj adjusts its own
local counter as O= max{O, ts (m) }, after which it then
executes the first step and delivers the message to the
application.
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Lamport’s Logical Clocks (4)

Figure 6-10. The positioning of Lamport’s logical

clocks in distributed systems.
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Vector Clocks
 Lamport's logical clocks lead to a situation where all events in a distributed
system are totally ordered with the property that if event a happened before
event b, then a will also be positioned in that ordering before b, that is, C (a) < C
(b).
 The problem is that Lamport clocks do not capture causality. Causality can be
captured by means of vector clocks.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Vector Clocks (2)

Vector clocks are constructed by letting each

process Pi maintain a vector VCi with the
following two properties:
1. VCi [ i ] is the number of events that have
occurred so far at Pi. In other words, VCi [ i ] is
the local logical clock at process Pi .
2. If VCi [ j ] = k then Pi knows that k events have
occurred at Pj. It is thus Pi’s knowledge of the
local time at Pj .

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Vector Clocks (3)
Steps carried out to accomplish property 2 of
previous slide:
1. Before executing an event Pi executes
VCi [ i ] ← VCi [i ] + 1.
2. When process Pi sends a message m to Pj, it sets
m’s (vector) timestamp ts (m) equal to VCi after
having executed the previous step.
3. Upon the receipt of a message m, process Pj
adjusts its own vector by setting
VCj [k ] ← max{VCj [k ], ts (m)[k ]} for each k, after
which it executes the first step and delivers the
message to the application.
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
Causal communication by vector
clock
Consider three processes Po, P1 and P2 as shown in Fig. 6-13. At local
time (1,0,0), P0 sends message m to the other two processes. After its
receipt by P1, the latter decides to send m*, which arrives at P2 sooner
than m. At that point, the delivery of m* is delayed by P2 until m has
been received and delivered to P2's application layer.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5