Parallel and Distributed

Computing
Fall 2022
Dr. Zeshan Iqbal
Topic: Time and Ordering
Lecture 2: Introduction and Basics

Why Synchronization?

• You want to catch a bus at 6.05 pm, but your watch is off by
15 minutes
– What if your watch is Late by 15 minutes?
• You’ll miss the bus!
– What if your watch is Fast by 15 minutes?
• You’ll end up unfairly waiting for a longer time than
you intended
• Time synchronization is required for both
– Correctness
– Fairness

2
 Synchronization In The Cloud

• Cloud airline reservation system

• Server A receives a client request to purchase last ticket on flight ABC
123.
• Server A timestamps purchase using local clock 9h:15m:32.45s, and logs
it. Replies ok to client.
• That was the last seat. Server A sends message to Server B saying “flight
full.”
• B enters “Flight ABC 123 full” + its own local clock value (which reads
9h:10m:10.11s) into its log.
• Server C queries A’s and B’s logs. Is confused that a client purchased a
ticket at A after the flight became full at B.
• This may lead to further incorrect actions by C

Why is it Challenging?

• End hosts in Internet-based systems (like clouds)

– Each have their own clocks
– Unlike processors (CPUs) within one server or workstation
which share a system clock
• Processes in Internet-based systems follow an asynchronous
system model
– No bounds on
• Message delays
• Processing delays
– Unlike multi-processor (or parallel) systems which follow a
synchronous system model

4
 Some Definitions

• An Asynchronous Distributed System consists of a number of

processes.
• Each process has a state (values of variables).
• Each process takes actions to change its state, which may be an
instruction or a communication action (send, receive).
• An event is the occurrence of an action.
• Each process has a local clock – events within a process can be
assigned timestamps, and thus ordered linearly.
• But – in a distributed system, we also need to know the time order
of events across different processes.

Clock Skew vs. Clock Drift

• Each process (running at some end host) has its own clock.
• When comparing two clocks at two processes:
• Clock Skew = Relative Difference in clock values of two processes
• Like distance between two vehicles on a road
• Clock Drift = Relative Difference in clock frequencies (rates) of two
processes
• Like difference in speeds of two vehicles on the road
• A non-zero clock skew implies clocks are not synchronized.
• A non-zero clock drift causes skew to increase (eventually).
– If faster vehicle is ahead, it will drift away
– If faster vehicle is behind, it will catch up and then drift away

6
 How often to Synchronize?

• Maximum Drift Rate (MDR) of a clock

• Absolute MDR is defined relative to Coordinated Universal
Time (UTC). UTC is the “correct” time at any point of time.
• MDR of a process depends on the environment.
• Max drift rate between two clocks with similar MDR is 2 *
MDR
• Given a maximum acceptable skew M between any pair of
clocks, need to synchronize at least once every: M / (2 * MDR)
time units
– Since time = distance/speed

External vs Internal Synchronization

• Consider a group of processes

• External Synchronization
– Each process C(i)’s clock is within a bound D of a well-known clock S
external to the group
– |C(i) – S| < D at all times
– External clock may be connected to UTC (Universal Coordinated
Time) or an atomic clock
– E.g., Cristian’s algorithm, NTP
• Internal Synchronization
– Every pair of processes in group have clocks within bound D
– |C(i) – C(j)| < D at all times and for all processes i, j
– E.g., Berkeley algorithm

8
 Next

• Algorithms for Clock Synchronization

Basics
• External time synchronization
• All processes P synchronize with a time server S

RTT
Set clock to t
Time
P What’s the time?

Here’s the time t!

S
Check local clock to find time t

10
 What’s Wrong

• By the time response message is received at P, time

has moved on
• P’s time set to t is inaccurate!
• Inaccuracy a function of message latencies
• Since latencies unbounded in an asynchronous
system, the inaccuracy cannot be bounded

11

Cristian’s Algorithm
• P measures the round-trip-time RTT
of message exchange

RTT
Set clock to t
Time
P What’s the time?

Here’s the time t!

S
Check local clock to find time t

12
 Cristian’s Algorithm (2)
• P measures the round-trip-time RTT of message exchange
• Suppose we know the minimum P à S latency min1
• And the minimum S à P latency min2
– min1 and min2 depend on Operating system overhead to buffer
messages, TCP time to queue messages, etc.
RTT
Set clock to t
Time
P What’s the time?

Here’s the time t!

S
Check local clock to find time t

13

Cristian’s Algorithm (3)

• P measures the round-trip-time RTT of message exchange
• Suppose we know the minimum P à S latency min1
• And the minimum S à P latency min2
– min1 and min2 depend on Operating system overhead to buffer messages, TCP
time to queue messages, etc.
• The actual time at P when it receives response is between [t+min2, t+RTT-min1]
RTT
Set clock to t
Time
P What’s the time?

Here’s the time t!

S
Check local clock to find time t

14
 Cristian’s Algorithm (4)
• The actual time at P when it receives response is between [t+min2, t+RTT-min1]
• P sets its time to halfway through this interval
– To: t + (RTT+min2-min1)/2
• Error is at most (RTT-min2-min1)/2 ------- Bounded!
• The width of this range is (RTT - 2*min). This gives an accuracy of (RTT/2 - min).
– Considering min1=min2

RTT
Set clock to t
Time
P What’s the time?

Here’s the time t!

S
Check local clock to find time t

15

Gotchas
• Allowed to increase clock value but should never decrease clock value
– May violate ordering of events within the same process

• Allowed to increase or decrease speed of clock

• If error is too high, take multiple readings and average them

16
 NTP = Network Time Protocol

17

NTP = Network Time Protocol

• NTP Servers organized in a tree
• Each Client = a leaf of tree
• Each node synchronizes with its tree parent
Primary servers

Secondary servers

Tertiary servers

Client

18
 NTP Protocol
Message 1 recv time tr1
Message 2 send time ts2
Time
Child
Let’s start protocol
Message 2 ts1, tr2
Message 1
Parent
Message 2 recv time tr2
Message 1 send time ts1

19

What the Child Does

• Child calculates offset between its clock and parent’s clock

• Uses ts1, tr1, ts2, tr2
• Offset is calculated as
o = (tr1 – ts1 + ts2 – tr2 )/2

20
 Why o = (tr1 - ts1 + ts2 - tr2)/2?
• Offset o = (tr1 – ts1 + ts2 – tr2)/2
• Let’s calculate the error
• Suppose real offset is oreal
– Child is ahead of parent by oreal
– Parent is ahead of child by -oreal
• Suppose one-way latency of Message 1 is L1 (L2 for Message 2)
• No one knows L1 or L2!
• Then
tr1 = ts1 + L1 + oreal
tr2 = ts2 + L2 – oreal

21

Why o = (tr1 - ts1 + ts2 - tr2)/2?

• Then
tr1 = ts1 + L1 + oreal
tr2 = ts2 + L2 – oreal
• Subtracting second equation from the first
oreal = (tr1 – ts1 + ts2 – tr2)/2 + (L2 – L1)/2 => oreal = o + (L2 – L1)/2
=> |oreal – o| < |(L2 – L1)/2| < |(L2 + L1)/2|
– Thus, the error is bounded by the round-trip-time

22
 NTP in real systems

23

NTP in real systems

• The 64-bit timestamps used by NTP consist of a 32-bit part for seconds and a
32-bit part for fractional second
• A typical NTP client will regularly poll three or more servers on diverse
networks. To synchronize its clock, the client must compute their time offset
and round-trip delay. t is the client's timestamp of the request packet
0

• Time offset θ is defined by 𝑡1 − 𝑡0 + (𝑡2 − 𝑡3)

transmission,
t is the server's timestamp of the request packet
𝜃=
1

2 reception,
t2 is the server's timestamp of the response packet
• and the round-trip delay δ by transmission and
t3 is the client's timestamp of the response packet
𝛿 = 𝑡3 − 𝑡0 − (𝑡2 − 𝑡1) reception.
• The values for θ and δ are passed through filters and subjected to statistical
analysis. Outliers are discarded and an estimate of time offset is derived
from the best three remaining candidates.
• The clock frequency is then adjusted to reduce the offset gradually

24
 And yet…
• We still have a non-zero error!
• We just can’t seem to get rid of error
– Can’t, as long as message latencies are non-zero
• Can we avoid synchronizing clocks altogether, and still be able to
order events?

25

Lamport Timestamps

26
 Ordering Events in a Distributed System
• To order events across processes, trying to sync clocks is one approach
• What if we instead assigned timestamps to events that were not absolute time?
• As long as these timestamps obey causality, that would work
If an event A causally happens before another event B,
then timestamp(A) < timestamp(B)
Humans use causality all the time
E.g., I enter a house only after I unlock it
E.g., You receive a letter only after I send it

27

Logical (or Lamport) Ordering

• Proposed by Leslie Lamport in the 1970s

• Used in almost all distributed systems since then
• Almost all cloud computing systems use some form of logical
ordering of events

28
 Logical (or Lamport) Ordering(2)
• Define a logical relation Happens-Before among pairs of events
• Happens-Before denoted as ®
• Three rules
1. On the same process: a ® b, if time(a) < time(b) (using the local
clock)
2. If p1 sends m to p2: send(m) ® receive(m)
3. (Transitivity) If a ® b and b ® c then a ® c
• Creates a partial order among events
– Not all events related to each other via ®

29

Example
A B C D E
P1
Time
E F G
P2

P3 H I J
While P1 and P3 each have an event
Instruction or step
labeled E, these are different events as
they occur at different processes. Message

30
 Happens-Before
A B C D E
P1
Time
E F G
P2

H I J
P3
• AàB Instruction or step
• BàF Message
• AàF

31

Happens-Before (2)
A B C D E
P1
Time
E F G
P2

H I J
P3• HàG
• FàJ Instruction or step
• HàJ Message
• CàJ

32
 In practice: Lamport timestamps
• Goal: Assign logical (Lamport) timestamp to each event
• Timestamps obey causality
• Rules
– Each process uses a local counter (clock) which is an integer
• initial value of counter is zero
– A process increments its counter when a send or an instruction happens at
it. The counter is assigned to the event as its timestamp.
– A send (message) event carries its timestamp
– For a receive (message) event the counter is updated by
max(local clock, message timestamp) + 1

33

Example
P1
Time

P2

P3
Instruction or step
Message

34
 Lamport Timestamps
P1 0
Time

P2 0

P3 0
Instruction or step
Initial counters (clocks)
Message

35

Lamport Timestamps
P1 0
ts = 1
Time

P2 0
Message carries
P3 0 ts = 1
ts = 1
Message send Instruction or step
Message

36
 Lamport Timestamps
P1 0
1 ts = max(local, msg) + 1
= max(0, 1)+1 Time
=2
P2 0
Message carries
P3 0 ts = 1
1
Instruction or step
Message

37

Lamport Timestamps
P1 0 2
1
Message carries Time
ts = 2
P2 0
2
max(2, 2)+1
=3
P3 0
1
Instruction or step
Message

38
 Lamport Timestamps
max(3, 4)+1
=5
P1 0 2
1 3
Time

P2 0
2 3 4

P3 0
1
Instruction or step
Message

39

Lamport Timestamps
P1 0 2 5 6
1 3
Time

P2 0
2 3 4

P3 0
1 2 7
Instruction or step
Message

40
 Obeying Causality
A B C D E
P1 0 2 5 6
1 3
Time
E F G
P2 0
2 3 4

H I J
P3 0
1 2 7
• A à B :: 1 < 2 Instruction or step
• B à F :: 2 < 3
• A à F :: 1 < 3 Message

41

Obeying Causality (2)

A B C D E
P1 0 2 5 6
1 3
Time
E F G
P2 0
2 3 4

H I J
P3 0
1 2 7
• H à G :: 1 < 4 Instruction or step
• F à J :: 3 < 7
• H à J :: 1 < 7 Message
• C à J :: 3 < 7

42
 Not always implying Causality
A B C D E
P1 0 2 5 6
1 3
Time
E F G
P2 0
2 3 4

H I J
P3 0
1 2 7
• ? C à F ? :: 3 = 3 Instruction or step
• ? H à C ? :: 1 < 3
• (C, F) and (H, C) are pairs of Message
concurrent events

43

Concurrent Events
• A pair of concurrent events doesn’t have a causal path from one event to
another (either way, in the pair)
• Lamport timestamps not guaranteed to be ordered or unequal for concurrent
events
• Ok, since concurrent events are not causality related!
• Remember
E1 à E2 Þ timestamp(E1) < timestamp (E2), BUT
timestamp(E1) < timestamp (E2) Þ
{E1 à E2} OR {E1 and E2 concurrent}

44
 Next

• Can we have causal or logical timestamps

from which we can tell if two events are
concurrent or causally related?

45

Vector Clocks

46
 Vector Timestamps
• Used in key-value stores like Riak
• Each process uses a vector of integer clocks
• Suppose there are N processes in the group 1…N
• Each vector has N elements
• Process i maintains vector Vi[1…N]
• jth element of vector clock at process i, Vi[j], is i’s knowledge of
latest events at process j

47

Assigning Vector Timestamps

• Incrementing vector clocks

1. On an instruction or send event at process i, it increments only its
ith element of its vector clock
2. Each message carries the send-event’s vector timestamp
Vmessage[1…N]
3. On receiving a message at process i:
Vi[i] = Vi[i] + 1
Vi[j] = max(Vmessage[j], Vi[j]) for j ≠ i

48
 Example
A B C D E
P1
Time
E F G
P2

H I J
P3
Instruction or step
Message

49

Vector Timestamps
P1(0,0,0)
Time

P2
(0,0,0)

P3
(0,0,0)
Initial counters (clocks)

50
 Vector Timestamps
P1(0,0,0) (1,0,0)
Time

P2
(0,0,0)

Message(0,0,1)
P3
(0,0,0) (0,0,1)

51

Vector Timestamps
P1(0,0,0) (1,0,0)
Time

P2
(0,0,0) (0,1,1)

Message(0,0,1)
P3
(0,0,0) (0,0,1)

52
 Vector Timestamps
P1(0,0,0) (1,0,0) (2,0,0)
Message(2,0,0) Time

P2
(0,0,0) (0,1,1) (2,2,1)

P3
(0,0,0) (0,0,1)

53

Vector Timestamps
P1(0,0,0) (1,0,0) (2,0,0) (3,0,0) (4,3,1) (5,3,1)
Time

P2
(0,0,0) (0,1,1) (2,2,1) (2,3,1)

P3
(0,0,0) (0,0,1) (0,0,2) (5,3,3)

54
 Causally-Related …
• VT1 = VT2,
iff (if and only if)
VT1[j] = VT2[j], for all j = 1, … , N
• VT1 ≤ VT2,
iff VT1[j] ≤ VT2[j], for all j = 1, … , N
• Two events are causally related iff
VT1 < VT2, i.e.,
iff VT1 ≤ VT2 &
there exists j such that
1 ≤ j ≤ N & VT1[j] (Not Equal) VT2 [j]

55

… or Not Causally-Related

• Two events VT1 and VT2 are concurrent

iff
(VT1 > VT2) AND (VT2 > VT1)

We’ll denote this as VT2 ||| VT1

56
 Obeying Causality
A B C D E
P1(0,0,0) (1,0,0) (2,0,0) (3,0,0) (4,3,1) (5,3,1)
Time
E F G
P2
(0,0,0) (0,1,1) (2,2,1) (2,3,1)

H I J
P3
(0,0,0) (0,0,1) (0,0,2) (5,3,3)
• A à B :: (1,0,0) < (2,0,0)
• B à F :: (2,0,0) < (2,2,1)
• A à F :: (1,0,0) < (2,2,1)

57

Obeying Causality (2)

A B C D E
P1(0,0,0) (1,0,0) (2,0,0) (3,0,0) (4,3,1) (5,3,1)
Time
E F G
P2
(0,0,0) (0,1,1) (2,2,1) (2,3,1)

H I J
P3
(0,0,0) (0,0,1) (0,0,2) (5,3,3)
• H à G :: (0,0,1) < (2,3,1)
• F à J :: (2,2,1) < (5,3,3)
• H à J :: (0,0,1) < (5,3,3)
• C à J :: (3,0,0) < (5,3,3)

58
 Identifying Concurrent Events
A B C D E
P1(0,0,0) (1,0,0) (2,0,0) (3,0,0) (4,3,1) (5,3,1)
Time
E F G
P2
(0,0,0) (0,1,1) (2,2,1) (2,3,1)

H I J
P3
(0,0,0) (0,0,1) (0,0,2) (5,3,3)
• C & F :: (3,0,0) ||| (2,2,1)
• H & C :: (0,0,1) ||| (3,0,0)
• (C, F) and (H, C) are pairs of concurrent events

59

Logical Timestamps: Summary

• Lamport timestamps
– Integer clocks assigned to events
– Obeys causality
– Cannot distinguish concurrent events
• Vector timestamps
– Obey causality
– By using more space, can also identify concurrent events

60
 Time and Ordering: Summary
• Clocks are unsynchronized in an asynchronous distributed
system
• But need to order events, across processes!
• Time synchronization
– Cristian’s algorithm
– NTP
– Berkeley algorithm
– But error a function of round-trip-time
• Can avoid time sync altogether by instead assigning logical
timestamps to events

61