Chapter 2 - Communication

Introduction
 Interprocess communication is at the heart of all distributed
systems
 Communication in distributed systems is based on message
passing as offered by the underlying network as opposed to
using shared memory
 Modern distributed systems consist of thousands of processes
scattered across an unreliable network such as the Internet
 Unless the primitive communication facilities of the network are
replaced by more advanced ones, development of large scale
Distributed Systems becomes extremely difficult

2
 Objectives of the Chapter
 Review of how processes communicate in a network (the rules or
the protocols) and their structures
 Introduce the four widely used communication models for
distributed systems:
 Remote Procedure Call (RPC)
 Remote Method Invocation (RMI)
 Message-Oriented Middleware (MOM)
 Stream-Oriented Communication
 Multicast Communication

3
 2.1 Network Protocols and Standards

 A protocol is a set of rules that governs data communications

 A protocol defines what is communicated, how it is communicated,
and when it is communicated
 For instance, for one computer to send a message to another
computer, the first computer must perform the following general
steps
 Break the data into small sections called packets
 Add addressing information to the packets identifying the source
and destination computers
 Deliver the data to the network interface card for transmission
over the network

4
 The receiving computer must perform the same steps, but in
reverse order
 Accept the data from the NIC
 Remove transmitting information that was added by the
transmitting computer
 Reassemble the packets of data into the original message
 The key elements of a protocol are syntax, semantics, and
timing
 Syntax: refers to the structure or format of the data
 Semantics: refers to the meaning of each section of bits
 Timing: refers to when data should be sent and how fast they
can be sent

 Functions of protocols
 Each device must perform the same steps the same way so
that the data will arrive and reassemble properly; if one
device uses a protocol with different steps, the two devices
will not be able to communicate with each other 5
 Protocols in a layered architecture
 Protocols that work together to provide a layer or layers of the
model are known as a protocol stack or protocol suite, e.g.
TCP/IP
 Each layer handles a different part of the communications
process and has its own protocol
 Data Communication Standards
 Standards are essential for interoperability
 Data communication standards fall into two categories
 De facto standards: that have not been approved by an
organized body; mostly set by manufacturers
 De jure standards: those legislated by an officially recognized
body such as ISO, ITU, ANSI, IEEE

6
Network (Reference) Models
 Layers and Services
 Within a single machine, each layer uses the services
immediately below it and provides services for the layer
immediately above it
 Between machines, layer x on one machine communicates with
layer x on another machine
 Two important network models or architectures
 The ISO OSI (Open Systems Interconnection) Reference Model
 The TCP/IP Reference Model
a. The OSI Reference Model
 Consists of 7 layers
 Was never fully implemented as a protocol stack, but a good
theoretical model
 Open – to connect open systems or systems that are open for
communication with other systems
7
layers, interfaces, and protocols in the OSI model
8
 Media (lower) Layers
 Physical: Physical characteristics of the media
 Data Link: Reliable data delivery across the link
 Network: Managing connections across the network
or routing
 Transport: End-to-end connection and reliability (handles
lost packets); TCP (connection-oriented),
UDP (connectionless), etc.
 Session: Managing sessions between applications
(dialog control and synchronization); rarely
supported
 Presentation: Data presentation to applications; concerned
with the syntax and semantics of the
information transmitted
 Application: Network services to applications; contains
protocols that are commonly needed by
users; FTP, HTTP, SMTP, ...
Host (upper) Layers
9
a typical message as it appears on the network

10
b. The TCP/IP Reference Model
 TCP/IP - Transmission Control Protocol/Internet Protocol
 Used by ARPANET and its successor the Internet
 Design goals
 The ability to connect multiple networks (internetworking) in a
seamless way
 The network should be able to survive loss of subnet
hardware, i.e., the connection must remain intact as long as
the source and destination machines are properly functioning
 Flexible architecture to accommodate requirements of
different applications - ranging from transferring files to real-
time speech transmission
 These requirements led to the choice of a packet-switching
network based on a connectionless internetwork layer
 Has 4 (or 5 depending on how you see it) layers: Application,
Transport, Internet (Internetwork), Host-to-network (some split it
into Physical and Data Link)
11
 OSI and TCP/IP Layers Correspondence

12
 Layers involved in various hosts (TCP/IP)
 When a message is sent from device A to device B, it may pass
through many intermediate nodes
 The intermediate nodes usually involve the first three layers

13
 Middleware Protocols
 A middleware is an application that contains general-purpose

protocols to provide services

 Example of middleware services

 Authentication and authorization services

 Distributed transactions (commit protocols; locking
mechanisms)
 Middleware communication protocols (calling a procedure or
invoking an object remotely, synchronizing streams for real-time
data, multicast services)
 Hence an adapted reference model for networked
communications is required

14
an adapted reference model for networked communication

15
 2.2 Remote Procedure Call
 The first distributed systems were based on explicit message
exchange between processes through the use of explicit send
and receive procedures; but do not allow access transparency
 In 1984, Birrel and Nelson introduced a different way of handling
communication: RPC
 It allows a program to call a procedure located on another
machine
 Simple and elegant, but there are implementation problems
 The calling and called procedures run in different address
spaces
 Parameters and results have to be exchanged; what if the
machines are not identical?
 What happens if both machines crash?

16
 Conventional Procedure Call, i.e., on a single machine
 e.g. count = read (fd, buf, bytes); a C like statement, where
fd is an integer indicating a file
buf is an array of characters into which data are read
bytes is the number of bytes to be read
Stack pointer

Stack pointer

parameter passing in a local procedure the stack while the called

call: the stack before the call to read procedure is active

 Parameters can be call-by-value (fd and bytes) or call-by

reference (buf) or in some languages call-by-copy/restore 17
 Client and Server Stubs
 RPC would like to make a remote procedure call look the same
as a local one; it should be transparent, i.e., the calling procedure
should not know that the called procedure is executing on a
different machine or vice versa

principle of RPC between a client and server program

 When a program is compiled, it uses different versions of library
functions called client stubs
 A server stub is the server-side equivalent of a client stub 18
 Steps of a Remote Procedure Call
1. Client procedure calls client stub in the normal way
2. Client stub builds a message and calls the local OS (packing
parameters into a message is called parameter marshaling)
3. Client's OS sends the message to the remote OS
4. Remote OS gives the message to the server stub
5. Server stub unpacks the parameters and calls the server
6. Server does the work and returns the result to the stub
7. Server stub packs it in a message and calls the local OS
8. Server's OS sends the message to the client's OS
9. Client's OS gives the message to the client stub
10. Stub unpacks the result and returns to client
 Hence, for the client remote services are accessed by making
ordinary (local) procedure calls; not by calling send and receive

 server machine vs server process; client machine vs client process

19
 Parameter Passing
1. Passing Value Parameters
 e.g., consider a remote procedure add(i, j), where i and j are
integer parameters

steps involved in doing remote computation through RPC

20
 The above discussion applies if the server and the client machines
are identical
 But that is not the case in large distributed systems
 The machines may differ in data representation (e.g., IBM
mainframes use EBCDIC whereas IBM PCs use ASCII)
 There are also differences in representing integers(1’s complement
or 2’s complement) and floating-point numbers
 Byte numbering may be different (from right to left in Pentium called
little endian and left to right in SPARC, big endian)
 e.g.
 Consider a procedure with two parameters, an integer and a
four-character string; each one 32-bit word (5, “JILL”)
 The sender is Intel and the receiver is SPARC

21
 Original message on the Pentium
(the numbers in boxes indicate the address of each byte)

The message after receipt on the SPARC; wrong integer (224+226 =

83,886,080), but correct string

22
 one approach is to invert the bytes of each word after receipt

the message after being inverted (correct integer but wrong string)

 Additional information is required to tell which is an integer

and which is a string

23
2. Passing Reference Parameters
 Assume the parameter is a pointer to an array
 Copy the array into the message and send it to the server
 The server stub can then call the server with a pointer to this
array
 The server then makes any changes to the array and sends it
back to the client stub which copies it to the client
 This is in effect call-by-copy/restore
 Optimization of the method
 One of the copy operations can be eliminated if the stub
knows whether the parameter is input or output to the server
 If it is an input to the server (e.g., in a call to write), it need not
be copied back
 If it is an output, it need not be sent over in the first place; only
send the size

24
 Parameter Specification and Stub Generation
 the caller and the callee need to use the same protocol (format of

messages) and the same steps; with such rules the client and
server stubs can assemble, communicate, and interpret messages
correctly
 consider the following example; the procedure foobar has 3

parameters: a character, a floating point number, and an array of 5

integers

 Assume a word is 4 bytes

 One possibility is to transmit the character in
the rightmost byte, a float as a whole word, and
an array as a group of words equal to the array
length preceded by a word giving the length
 This way both client stub and server stub can
understand outgoing and incoming messages
the corresponding message
25
 Other issues that need the agreement of the client and the server
 How are simple data structures like integers (e.g. 2’s
complement), characters (e.g. 16-bit Unicode), Booleans, ...
represented?
 Endianess
 Which transport protocol to use - the connection-oriented TCP or
the unreliable connectionless UDP

26
Asynchronous RPC
 A shortcoming of the original model: no need of blocking for the

client in some cases

 two cases
1. if there is no result to be returned
 e.g., inserting records in a database, ...
 the server immediately sends an ack promising that it will
carryout the request
 the client can now proceed without blocking

a) the interconnection between client and server in a traditional RPC

b) the interaction using asynchronous RPC 27
2. if the result can be collected later
 e.g., prefetching network addresses of a set of hosts, ...
 the server immediately sends an ack promising that it
will carryout the request
 the client can now proceed without blocking
 the server later sends the result

a client and server interacting through two asynchronous RPCs

28
  the above method combines two asynchronous RPCs
and is sometimes called deferred synchronous RPC
 variants of asynchronous RPC
 let the client continue without waiting even for an ack,
called one-way RPC
 problem: if reliability of communication is not guaranteed

29
 DCE (Distributed Computing Environment) RPC
 a middleware and an example RPC system developed by
OSF (Open Software Foundation), now The Open Group
 it is designed to execute as a layer of abstraction between
existing OSs and distributed applications
 The Open Group sells the source code and vendors
integrate it into their systems
 it uses the client-server programming model and
communication is by means of RPCs
 services
 distributed file service: a worldwide file system that
provides a transparent way of accessing files
 directory service: to keep track of the location of all
resources in the system (machines, printers, data,
servers, ...); a process can ask for a resource without
knowing its location
 security service: for protecting resources; access is only
through authorization 30
 distributed time service: to maintain clocks on different
machines synchronized

31
 Binding a Client to a Server in DCE RPC
 for a client to call a server, the server must be registered (1
& 2 in the following figure)
 the registration allows the client to locate the server and
bind to it
 the DCE daemon maintains a table (server, endpoint) and the
protocols the server uses
 the directory server maintains the locations of all resources
in the system (machines, servers, data,, ...)
 two steps for server location
 locate the server’s machine (3)
 locate the server process on that machine (which has
what is called an endpoint or port) (4)

32
33
 2.3 Message-Oriented Communication
 RPCs are not adequate for all distributed system applications
 The provision of access transparency may be good but they
have semantics that is not adequate for all applications
 Example problems
 They assume that the receiving side is running at the time of
communication
 A client is blocked until its request has been processed

34
 Persistence and Synchronicity in Communication
 assume the communication system is organized as a
computer network shown below

general organization of a communication system in which hosts are

connected through a network
35
 communication can be
 persistent or transient
 asynchronous or synchronous
 persistent: a message that has been submitted for
transmission is stored by the communication system as long
as it takes to deliver it to the receiver
 e.g., email delivery, snail mail delivery
 transient: a message that has been submitted for
transmission is stored by the communication system only as
long as the sending and receiving applications are executing
 asynchronous: a sender continues immediately after it has
submitted its message for transmission
 synchronous: the sender is blocked until its message is
stored in a local buffer at the receiving host or delivered to the
receiver

36
 Message-Oriented Transient Communication
 many applications are built on top of the simple message-
oriented model offered by the transport layer
 standardizing the interface of the transport layer by
providing a set of primitives allows programmers to use
messaging protocols
 they also allow porting applications

1. Berkley Sockets
 an example is the socket interface as used in Berkley
UNIX
 a socket is a communication endpoint to which an
application can write data that are to be sent over the
network, and from which incoming data can be read

37
Primitive Meaning Executed by
Socket Create a new communication endpoint both

Bind Attach a local address to a socket; e.g., IP

address with a known port number
Listen Announce willingness to accept connections
Accept Block caller until a connection request arrives
Actively attempt to establish a connection; the servers
Connect client is blocked until connection is set up
Send Send some data over the connection
Receive Receive some data over the connection
Close Release the connection
socket primitives for TCP/IP

38
connection-oriented communication pattern using sockets

39
2. The Message-Passing Interface (MPI)
 Sockets were designed to communicate across networks using
general-purpose protocol stacks such as TCP/IP
 MPI is designed for parallel applications and tailored for
transient communication
 MPI assumes communication takes place within a known
group of processes, where each group is assigned an identifier
(group ID)
 Each process within a group is also assigned an identifier
(process ID)
 A (group ID, process ID) identifies the source or destination of
a message, and is used instead of a transport-level address

40
 Message-Oriented Persistent Communication
 there are message-oriented middleware services, called
Message-Queuing Systems or Message-Oriented
Middleware (MOM)
 they support persistent asynchronous communication
 they have intermediate-term storage capacity for messages,
without requiring the sender or the receiver to be active
during message transmission
 unlike Berkley sockets and MPI, message transfer may take
minutes instead of seconds or milliseconds
 Message-Queuing Model
 applications communicate by inserting messages in
specific queues
 it permits loosely-coupled communication
 the sender may or may not be running; similarly the
receiver may or may not be running, giving four possible
combinations
41
four combinations for loosely-coupled communications using queues
42
  General Architecture of a Message-Queuing System
 messages can be put only into queues that are local to the
sender (same machine or on a nearby machine on a LAN)
 such a queue is called the source queue
 messages can also be read only from local queues
 a message put into a local queue must contain the
specification of the destination queue; hence a message-
queuing system must maintain a mapping of queues to
network locations; like in DNS

the relationship between queue-level addressing and network-level addressing 43

 messages are managed by queue managers
 they generally interact with the application that sends and
receives messages
 some also serve as routers or relays, i.e., they forward
incoming messages to other queue managers
 however, each queue manager needs a copy of the queue-
to-location mapping, leading to network management
problems for large-scale queuing systems
 the solution is to use a few routers that know about the
network topology

44
the general organization of a message-queuing system with routers
45
 Message Brokers
 how can applications understand the messages they receive
 each receiver can not be made to understand message
formats of new applications
 hence, in a message-queuing system conversations are
handled by message brokers
 a message broker converts incoming messages to a format
that can be understood by the destination application based
on a set of rules

the general organization of a message broker in a message-queuing system 46

 2.4 Stream-Oriented Communication
 until now, we focused on exchanging independent and
complete units of information
 time has no effect on correctness; a system can be slow or fast
 however, there are communications where time has a critical
role
 Multimedia
 media
 storage, transmission, interchange, presentation,
representation and perception of different data types:
 text, graphics, images, voice, audio, video, animation, ...
 movie: video + audio + …
 multimedia: handling of a variety of representation media
 end user pull
 information overload and starvation
 technology push
 emerging technology to integrate media 47
 The Challenge
 new applications
 multimedia will be pervasive in few years (as graphics)
 storage and transmission
 videos are extremely large, even after compressed
(actually encoded)
 continuous delivery
 e.g., 30 frames/s (NTSC), 25 frames/s (PAL) for video
 guaranteed Quality of Service
 admission control
 search
 can we look at 100… videos to find the proper one?

48
 Types of Media
 two types
 discrete media: text, executable code, graphics, images;
temporal relationships between data items are not
fundamental to correctly interpret the data
 continuous media: video, audio, animation; temporal
relationships between data items are fundamental to
correctly interpret the data
 a data stream is a sequence of data units and can be applied
to discrete as well as continuous media
 stream-oriented communication provides facilities for the
exchange of time-dependent information (continuous media)
such as audio and video streams

49
 timing in transmission modes
 asynchronous transmission mode: data items are
transmitted one after the other, but no timing constraints;
e.g. text transfer
 synchronous transmission mode: a maximum end-to-end
delay defined for each data unit; it is possible that data can
be transmitted faster than the maximum delay, but not slower
 isochronous transmission mode: maximum and minimum
end-to-end delay are defined; there is a fixed gap b/n the
transmission. also called bounded delay jitter; applicable for
distributed multimedia systems
 simple stream: consists of a single sequence of data; e.g.,
mono audio, video only (only visual frames)
 complex stream: consists of several related simple streams,
called substreams, that must be synchronized; e.g., stereo
audio, video consisting of audio and video (may also contain
subtitles, translation to other languages, ...)
50
movie as a set of simple streams

51
A general architecture for streaming stored multimedia data over a
network

52
 Quality of Service (QoS)
 QoS requirements describe
what is needed from the
underlying distributed
system and network to
ensure acceptable delivery;
e.g. viewing experience of a
user
 it refers to flow characteristics
 Reliability
 lack of reliability means losing a packet or

acknowledgement, which entails retransmission for some

media types
 Delay
 source-to-destination delay

 Bandwidth
 requirements of applications
53
 different applications have different flow characteristic
requirements

54
 Techniques to improve Network QoS (some are useful for
multimedia)
 the easiest (but impractical) solution: overprovisioning -
provide enough router capacity, buffer space, and bandwidth
for all packets; very expensive
 five common methods: buffering, traffic shaping, scheduling,
admission control, and resource reservation
1. Buffering - Client Side
 buffer flows on the receiving side (client machine) before
delivery (playback)
 it smoothes jitter (for audio and video on demand since
jitter is the main problem) - does not affect reliability or
bandwidth, increases delay

55
  how long to delay is difficult - commercial Web sites that
stream audio or video use players that buffer for about 10
seconds before starting to play
 may not always be possible, e.g., videoconferencing

2. Traffic Shaping - Server Side

 to control the amount and the rate of the traffic sent to a
network
 nonuniform output is common if a server is handling many
streams at once and also allows other actions - such as
fast forward and rewind, user authentication, ...; this may
create congestion
 two techniques: Leaky Bucket and Token Bucket

56
a. Leaky Bucket
 Shapes bursty traffic into fixed-rate traffic by averaging the data
rate
 May drop packets if the bucket is full
 The input rate may vary, but the output rate remains constant

57
b. Token Bucket
 leaky bucket is very restrictive; does not credit an idle host
 token bucket allows idle hosts to accumulate credit for the
future in the form of tokens
 for each tick of the clock, the system sends n tokens to the
bucket
 the system removes one token for every cell of data sent

58
 e.g., if n = 100 and the host is idle for 100 ticks, the bucket
collects 10,000 tokens; now the host can consume all these
tokens in one tick with 10,000 cells, or the host takes 1000
ticks with 10 cells per tick
 that means the host can send bursty data as long as the
bucket is not empty
 token bucket allows bursty traffic at a regulated maximum
rate

59
3. Scheduling - at the Routers
 to avoid an aggressive sender from utilizing most of the
capacity of a router; 3 commonly used scheduling methods
a.FIFO Queuing
 packets wait in a buffer; if the queue fills up, new
packets are discarded

60
b. Priority Queuing
 packets are first assigned to a priority class
 there is a queue for each priority class
 e.g., multimedia can be given higher priority
 better QoS than FIFO, but packets in a low-priority queue
may starve

61
c.Weighted Fair Queuing
 A modification of priority queuing to avoid starvation; packets
are still assigned to priority classes
 Queues are assigned weights (higher priority means a higher
weight);
 The system processes packets in each queue in a round-
robin fashion with the number of packets selected from each
queue based on the corresponding weight
 A better scheduling method

62
4. Resource Reservation
 Qos is improved if resources (buffer, bandwidth, CPU time, ...)
are reserved along the path before hand
 there is a specific route for a flow

5. Admission Control
 A mechanism used by a router to accept or reject a flow based
on predefined parameters called flow specifications
 Before it accepts a flow, it checks the flow specification to see if
its capacity (buffer size, bandwidth, CPU speed, ...) and its
previous commitments to other flows can handle the new flow

63
 Stream Synchronization
 How to maintain temporal relations between streams, e.g., lip
synchronization
 Two approaches
1. Explicitly by operating on the data units of simple streams; the
responsibility of the application

the principle of explicit synchronization on the level of data units 64

2. Through a multimedia middleware that offers a collection of
interfaces for controlling audio and video streams as well as
devices such as monitors, cameras, microphones, ...

the principle of synchronization as supported by high-level interfaces

65
 2.5 Multicast Communication
 Multicasting: delivery of data from one host to many destinations;
for instance for multimedia applications
 A one-to-many relationship
1. Application-Level Multicasting
 Nodes are organized into an overlay network and information is
disseminated to its members (routers are not involved as in
network-level routing)
 How to construct the overlay network
 Nodes organize themselves as a tree with a unique path
between two pairs of nodes or
 Nodes organize into a mesh network and there will be
multiple paths between two nodes; adv: robust
2. Gossip-Based Data Transmission
 Use epidemic protocols where information is propagated among
a collection of nodes without a coordinator
66