VXLAN and EVPN for Data Center Network

Transformation
Talvinder Singh Varun Jain Satish Babu G
Project Associate (DC & IS) Joint Director (DC & IS) Principal Technical Officer (DC&IS)
CDA C, No ida, India CDAC, No ida, India CDAC, No ida, India
+91-8860593787 +91- 9015110101 +91-8826579401
[email protected] [email protected] [email protected]

Abstract —Data Center architectu res are continually evolving private) can be used to build a cloud. This paper presents the
to address the growing need to handle thousands of servers, study about the current and emerging technologies used in inter
Exabyte of storage, Tbps of traffic and multi-tenant and intra DC networking architectures that applies to virtual
environment. It has to be more flexible & scalable to cater to private cloud services. Also, an overview of datacenter
the need of Cloud services to provide Scalable compute, network architecture with different requirements is analyzed
storage & network, Efficient resource utilization, Cost and the packet networking technologies used for
effectiveness, Reliable service and Security. The resources are implementation have been discussed in detail in section II. In
interconnected through routers and switches that cater the section III the overview of existing networking technologies
multi-tenant virtual networking need within and across Data and its enhancements viz., Virtual Bridging & VPLS are
Center for which the WAN connectivity should be transparent explored. To overcome the lacunae in these technologies the
in the Layer 2. At present the DC Interconnect (DCI) are evolving new areas like VXLAN [2] & EVPN are elaborated in
through IP/MPLS Virtual Private Network (VPN) over which Section IV. The transformation required for existing
the L2 transparency is achieved majorly by the technologies technologies and the adaptability of new technologies in terms
like VPLS & EoMPLS. This paper focuses majorly on the of virtualized implementations in the cloud environment
latest evolving technology to extend VXLAN across DCs with namely Network Virtualization (NV) & Software Defined
EVPN control plane and concludes with its benefits like faster Networking (SDN) are covered in Section V. The need for the
convergence, efficient, flexible & scalable DCI realized by the new technologies and the benefits that it provides are detailed in
implementations of MP-BGP which optimizes unicast the conclusion at Section VI.
flooding, MAC Address pu rging& apply fine grain policy.
II. DATA CENTER NETWORKING ARCHITECTURE
Keywords—VXLAN; SDN; EVPN; Cloud Networking; Virtualization, at DC level, provides foundation for offering
Datacenter networks application services and resources to multiple tenants over same
infrastructure. A cloud may compose of several DCs. A virtual
I. INTRODUCTION machine (VM ) is launched on a server (bare metal) with
dedicated CPUs, memory, storage, and I/O bandwidth. M ultiple
Data Center requirements for scaling of storage, computing VM s can be launched on the same physical server. In cloud,
and bandwidth are shooting up due to the rise of internet, video when a VM is created, it may launch on any server configured
content, e-commerce, analysis applications and tailored cloud- in cluster on basis of its favorability. A tenant may be
based services. The type of services offered by Data Center provided a group of VM s that may reside on servers spread
(DC) after the evolution of Cloud Technology includes across DCs or within a DC. Some of the DCs may be private
applications, platforms, compute and storage infrastructure tenant DCs connected through the cloud IP or a multiprotocol
services through internet or virtual private networks. The label switching (M PLS) network to corresponding tenant virtual
demand of computing, storage, connectivity and bandwidth private datacenters.
may vary for all such services. Depending on the type of DC
the services and their mode of providing services may vary. For DC has large number of tenants as virtual services are
instance, Private DCs may support multi-tenancy within provisioned to them, it requires networking ability among
enterprise. Enterprise-dedicated or virtual private network is compute and storage resources allotted to them. Through
setup for such purpose. Generally, services within private DC traditional VLAN and IP routing methods, it is difficult to
are private to enterprise but could be partially connected to manage and orchestrate the resources. Virtualization of network
internet through secure gateways [9]. On other hand, Public is next obvious step – decoupling the virtual network from
datacenters provide service to their users through internet that physical network makes it easier to manage, automate, and
includes multi tenant services as well as commercial public orchestrate.
cloud services. DC that is distributed across multiple sites and clouds, a multi-
Virtual private Data Centers are built on common datacenter vendor network requires to innovate and evolve their
infrastructure provided by cloud service providers in form of architecture. Network challenges include increase agility,
Infrastructure as a Service (IaaS). The tenant network isolation reduce costs, enhance network response, and reduce risk.
and privacy depends on the underlying network architecture. With certain challenges network infrastructure must support:
Particular type of Data Center (i.e public, private, or virtual
• Ability to provide overlapped virtual -MAC and IP address III. ENHANCEM ENTS IN THE EXISTING DATA
spaces CENTERS AND CLOUD NETWORKING
• VM mobility TECHNOLOGIES
• Bandwidth growth
There are some limitations associated with existing standard
• Efficient utilization of resources
technologies to address the network virtualization, such as
• Traffic engineering with performance constraints.
limitations on scale in number of services. Enhancements are
being carried out in the existing technologies and also Newer
technologies have emerged, to address these limitations.Few
new technologies are in the process of being standardized for
deployment. Below sections provide overview of existing
technologies in context of DC architecture.
IEEE 802.1q: VIRTUAL BRIDGING
VLAN bridging and isolation among tenants is provided by
IEEE 802.1q. VLAN creates a logical separation among
different tenants. It can support up to 4096 VLANs. Having a
12-bit VLAN ID may put constraint on number of tenants that
can be supported by it. Additionally, M AC forwarding tables
may also impose limitation when the number of VM s connected
to particular VLAN grows. Given the requirement of number of
tenants in cloud, scalability becomes intrinsic problem in IEEE
802.1q.
IEEE 802.1AH/AQ/QBP: PROVIDER BACKBONE BRIDGING
Provider backbone bridging (PBB) extends L2 Ethernet
FIG1. TYPICAL DATACENTER ARCHITECTURE switching to provide enhanced scalability and QoS features.
PBB is defined in IEEE 802.1ah. PBB has 24-bit service I-SID
Cloud service providers (DCs) are generally connected to one which supports over 16 million VLAN instances. I-SID can be
or more internet service providers (ISPs) to provide assigned to a tenant virtual network. PBB provides separation
connectivity to users that are accessing the DC services from between customers M AC addresses (C-MACs) and provider-
their private sites or the Internet, or to connect multiple DCs. domain bridge M AC addresses (B-MACs). C-MACs
M ultiple DCs are connected through a leased fiber or private corresponding to hosts, servers, and VM s are only learned at the
line service provided by ISPs. The solutions usually in place
make use of L3 VPN or L2 Ethernet VPN service. Cloud
tenants may be provided with services like firewall (FW),
application Load Balancing (LB), and Network Address
Translation (NAT) as part of their virtual network. Each tenant
is can be provided with instance of these services on physical or
virtual appliances (in VM s). The DC depicted in Fig. 1 includes
top of the rack (ToR) switches placed in each rack (typically a
hardware based element), virtual switches (vSW) [5], a
software based switch running inside each server and core
switches that may act as aggregation switches. DC gateway,
usually a L3 device such as router provides connectivity to
internet and interconnection between DCs. Virtual routing and
switching capabilities are provided by DC GW that are
connected to service provider edge (PE) devices. Inter-DC
FIG2. UNICAST PACKET FLOW IN PBB
connectivity is provided over IP/M PLS network and/or an
independent network can also be used for the purpose. VM s PBB backbone edge bridges (BEBs) providing tenant access
instantiated on physical server appears to be an IP host to the [10]. A BEB encapsulates C-M AC frames with a B-M AC
vSW. VLAN services and IP routing within DC infrastructure is header. As a result, only B-M ACs are learned by PBB
supported by ToR. Depending on the size of DC there could be backbone core bridges (BCBs) interconnecting BEBs. PBB
two or more core switches. As we move from server to DC core separates Ethernet as a service from Ethernet as infrastructure.
the Ethernet link rates increases. Commonly ToR links varies As shown in Fig.2, the MAC address learning and forwarding at
from 1GbE to 10GbE. For DC core and interconnections across BEB will happen based on both C-MAC and B-MAC. It builds
WAN 40GbE and 100GbE have been achieved and some a mapping from C-M AC to B-MAC to forward packets from
phases are in test bed. CE to BEB. As packet reaches BEB, I-SID is added to it along
with bridge VLAN identifier. Now when packet travels in
The following sections discuss about the current and evolving
bridge core it knows only about B-M ACs.
packet technologies to realize these Network Architectures.
With reference to the typical DC architecture given in Fig. 1, Below section discusses the technologies that remove the
DC core device can be used for providing BCB switching on limitations mentioned.
BMACs, BEB can be realized through ToR. If the topology is
loop-free then the B-MAC forwarding tables and tenant virtual VXLAN
networks (VNs) (e.g., I-SIDs, VLANs) can be configured by
As highlighted in the above section, traditional VLAN IDs
management plane, which is based on IEEE 802.1q or PBB’s
are 12-bit long, which limits the total number of VLANs to
control plane. The loops can also be prevented by multi-
4094. . VXLAN is often described as overlay technology. The
spanning tree (M ST) but it inefficiently utilizes the available
link capacity. Loops can be prevented and M ST can be L2 connections are allowed to stretch over a existing L3
network by the encapsulated (tunneled) Ethernet Frames in the
improved in utilizing available capacity through the usage of
VXLAN packet having IP addresses. The virtual network
equal cost trees by Shortest path bridging (SPB), then the
identifier (VNI) used in VXLAN is 24-bit long which unlike
network design will become complex. The link utilization can
VLANs provides more number of identifiers to isolate the
be further improved and network architecture design &
operation can be eased by SPB with IEEE 802.1Qbp, equal cost network logically for cloud networks which are large in nature
due to the concentration of virtual machines. Virtual Tunnel
multiple paths (ECM P), but during network convergence transit
End Points (VTEPs) are those devices which support VXLANs.
loops may occur. There are few deployments of SPB and PBB
A number of fields like outer IP source address, outer IP
which are reputed in nature in DCs which use Ethernet Fabric.
destination address, outer M AC source address, outer M AC
This has led the joint efforts of the companies in the
virtualization and network domain, which are driven by the destination address, outer UDP header and finally 24-bit
VXLAN network identifier (VNI), the VXLAN header are
architecture principles and need, to the new technology called
added by VTEP for encapsulating an Ethernet Frame. The
VXLAN (Virtual Extensible LAN).
VXLAN packet encapsulation is shown in Fig-3.
Traditionally, data centers have used virtual private network
VXLAN overlays offer a number of benefits:
(VPN) technologies for carrying the IP based or Ethernet based
traffic across different Service Providers with core networks • Elimination of Spanning Tree Protocol (STP)
• Increased scalability
using IP/M PLS using the technologies like tunneling which also
• Improved resiliency
maintains the isolation of customers. The cloud services can
• Fault containment and Enables you to migrate VM s between
also use the same technologies. Transparent LAN Service can
hosts which exists in different L2 domains by way of tunneling
be provided by VPLS (Virtual private LAN service) over the
shared packet switched network through IP / M PLS. M PLS the traffic over the existing L3 network. This allows dynamic
allocation of resources within or among DCs and not being
provides high performance forwarding mechanism for building
restricted by L2 boundary.
private LAN where we can interconnect multiple LANs which
The M AC learning in VXLAN can happen at data plane and
are based at multiple sites or data centers. It simply maps onto
control plane (Fig4).
L2 protocol and provide common fast efficient transport
method over PSN. A large quantity of service instances can be
support by it. In reference to Fig1, a VPLS instances can be
provided by a DC gateway across WAN for extending the
tenant Ethernet VN to the cloud provider of another DC or at a
customer site. The customer site is provided connectivity by
connecting DC GW to the service provider’s PE providing
customer site VPN. Depending on the technology (e.g.,
NVGRE, VXLAN, 802.1ah and 802.1q) the VPN services
access at the DC Gateway and within DC will vary.
FIG3. VXLAN PACKET FORMAT
IV. EVOLVING NETWORKING TECHNOLOGIES FOR
DATA CENTERS AND CLOUD EVPN
Data centers have used L2 technologies such as Spanning
In the traditional data center, network architects use VLANs Tree Protocol (STP), multichassis link aggregation group (M C-
to create L2 logical networks that provide security by LAG), and Transparent Interconnection of Lots of Links
segregating users and applications, and they improve (TRILL) for compute and storage connectivity. M ost network
performance by limiting broadcast traffic. However, this designers prefer DCI over WAN be performed at L3, without
architecture limits the scaling of tenants as mentioned in above. spanning L2 between DCs. However, some application
requirements may need L2 interconnection between DCs such
Design objectives for next generation data centers are: as high availability clustering services, VM migration,
a) Scalability of Layer-2 and Layer-3 networks irrespective of replication, and storage virtualization. As the design of these
underlying physical topology. data centers evolves to scale out multitenant networks, new data
b) Location independent addressing, center architecture is needed that creates tenant overlay network
c) Preserving L-2 semantics for services and allowing them to with VXLAN over the existing underlay network. Using a
retain their addresses as they move across and within data Layer-3 IP-based underlay coupled with a VXLAN-EVPN
centers, and overlay, data center and cloud operators can deploy much larger
d) Providing broadcast isolation as workloads move around networks than are otherwise possible with traditional L2
without burdening the control plane of network. Ethernet-based architectures. With overlays, endpoints (servers
or virtual machines) can be placed anywhere in the network and may contain thousands of hosts or VM s running behind it. If
remain connected to the same logical L2 network, enabling the link between the hostsand leaf switch LS3 goes down, LS3 will
virtual topology to be decoupled from the physical topology. advertise that ESI (0:1:1:1:1:1:1:1:1:1) is not reachable and LS1
will simply withdraw its route for that ESI, resulting in faster
Ethernet VPN (EVPN) started as L2 VPN solution for MPLS
convergence.
core. It helps to improve network efficiency by reducing
unknown-unicast flooding due to control-plane M AC learning TABLE1. TYPE-2 ROUTE INFORMATION PACKET
and also reduce ARP flooding due to IP-to-MAC binding in MAC/IP
control plane. It achieves faster convergence when link to dual- Route Type Advertisement
homed server fails and re-convergence when VM moves.
Type-2
L2 switch (leaf switch) learns MAC address of VM attached to
Route
it through traditional L2 learning. Optionally, it may also learn …
IP-to-MAC binding through DHCP or ARP Distinguisher
Ethernet Segment
0:1:1:1:1:1:1:1:1:1
Identifier (ESI)
NLRI
Ethernet Tag ID VNID
MAC of H2
MAC Address
(5:5:5:5:5:1)
IP of Host H2
IP Address
(10.10.10.22)
MPLS Label 1 +
VNID
MPLS Label 2
Next –hop Loopback of LS3

FIG4. ST RET CHING VXLAN ACROSS DC WITH EVPN Stretching VXLAN between Data Centers
Enterprises might want to stretch VXLAN tunnels between
spoofing. In traditional flood-and-learn network, leaf-switch in DCs, example, for live migration of VM between DCs.
different Ethernet would not learn the M AC of VM until either Traditionally, enterprises with multiple DCs have L3 solutions
VM has sent traffic to it or leaf-switch has received ARP for data center interconnectivity. EVPN provides flexibility
request from leaf-switch of other Ethernet say, at different DC. with easy integration of L3VPNs and L2VPNs for Data Center
Interconnect (DCI). Fig4 shows abstraction of how L2
With EVPN, as soon leaf-switch locally learns VM M AC information is learned from one DC to other using EVPN
address it immediately advertises this information via Type-2 control plane.
route (IP/M AC Binding Advertisement) to all its M P-BGP
peers having same VXLAN VNID. This is primary benefit of Stretching VXLAN across DCs
EVPN control plane. So, far EVPN introduced five types of MPLS based technologies have not been conventionally
routes. We have discussed only Type-2 route, as it provides deployed in DC operational environment. The VPN
remote M AC learning over control plane. technologies described for the WAN environment could also be
used within a DC to provide Ethernet bridging and IP Virtual
We assume that LS2 and LS3 (Fig5) have both learned host Network services at scale. However, EVPN and IP-VPN data
H2’s M AC address. Leaf switch LS1 receives Type-2 route planes are being targeted for implementation on virtual switches
advertisement for H2 M AC along with associated ESI as shown in servers with GRE or VXLAN tunneling over the DC core.
in Table1.

V. NETWORK VIRTUALIZATION OVER LAYER 3

(NVO3) & SOFTWARE DEFINED NETWORKING

The goal of data center network virtualization overlay

(NVo3) networks is to decouple the communication among
tenant systems from DC physical infrastructure networks and
allow one physical network infrastructure. NV is an
economical way to provide traffic sep aration and multiple
virtualized networks can be overlaid on a single physical
FIG5. EVPN T YPE-2 ADVERTISMENT infrastructure. In cloud environment, traffic separation between
different tenants is required. Internet Engineering Task Force
The same advertisement for H2 is received from LS2. H2 is (IETF) is currently working on NVo3 to develop an architecture
reachable through both LS2 and LS3. Since, same host H2 is framework, address the issues that cloud applications face in
connected to two leaf switches LS2 and LS3, link aggregation date center, and determine gaps in existing solutions. VXLAN
is performed and single ESI is generated. ESI plays important and NVGRE triggered the NVo3 work.
role in faster convergence. In DC networks the leaf switches
Software Defined Networking (SDN) is essentially makes a increasing the operational efficiency providing effective
network programmable by decoupling control from the business continuity.
underlying hardware and assigning it instead to software based
controller. High performance: The adoption of cloud has made DCs to
rethink their network architecture both at intra and inter-DC
Software defined networking relies on a logically centralized level. To overcome the bandwidth limitation and network delay
network controller (control plane) to provide an abstraction problems, the DC networks should support multi-pathing and
layer that provides control of the network behavior requested by control plane learning to optimize network traffic flow and
applications. The controller has global view of all networking bandwidth utilization. It requires faster convergence and
devices below it and has a way to communicate packet flow aggregation of links which are the available by using a Layer-3
instruction to them. SDN brings the flexibility and economy of IP-based underlay coupled with a VXLAN-EVPN overlay.
software to DC hardware.
In brief, the benefits of using EVPN include multi-tenancy,
In DCs, SDN can offer benefits in dealing with big data, integrated routing & bridging and support for VXLAN and
supporting cloud-based traffic – which requires on-demand NVGRE. VXLAN is itself not designed as L2 DCI technology
capacity and self-service, managing traffic to many IP but uses EVPN acts as a control plane. EVPN as L2 solution
addresses and VM s, making infrastructure scalable, agile, for DCI provides faster convergence when VM moves from one
managing policy and security. DC to another and also provides remote M AC learning which
thereby reduces unknown-unicast flooding. The M AC learning
An SDN controller can be programmed to disseminate the VM in VXLAN is more efficient and scale well as it happens in
reach ability information to the servers, switching and routing control plane. Also some light has been thrown on NV &SDN
nodes. Protocols such as OpenFlow or an API can be used for which helps enterprises to separate the control plane from the
providing communication between SDN controller and network
forwarding plane, in turn enabling better control of the network,
entities it is controlling. It involves programmability of network increased programmability options, and better agility and
elements, virtual or physical, on servers and DC GWs and flexibility; all this in combination leads to reduction of capital
between DC GW and PE. As virtual applications, storage and and operating expenditure of the network. VXLAN has already
compute resources are created it is needed to interconnect these been accepted as the de-facto standard overlay technology for
resources through virtual networks, and connect them to
deployment of next-gen DCs.
corresponding tenant virtual and private data sites or Internet. It
is expected that such resources and networks are established VII. LIM ITATIONS AND FUTURE SCOPE
within seconds or minutes. Use of network overlay
technologies, such as VPN or VXLAN tunneling, SDN This paper is based on the comparative study among the limited
controller may program DC core with forwarding information. technologies used at present and some latest emerging
Having a SDN controller enables to have control over the entire technologies as per the requirement analysis for the industry
network and provides ease to manage it. based on our professional expertise. The existing technologies
are analyzed with its present implementation at our
VI. CONCLUSION environment and to overcome the limitations of those, the new
technologies are studied, which are yet to be implemented.
With the evolution of Cloud Technology and vast benefit of
multi-tenancy, the basic requirement of network architecture ACKNOWLEDGEM ENTS
has altogether changed from a Single DC perspective to multi
DC across locations in order to achieve seamless business We would like to show our gratitude to M r. V.K Sharma,
continuity. This has necessitated to enhance the existing Director, Centre for Development of Advanced Computing,
technologies and also to implement new technologies with lots Noida for providing us opportunity and support to explore and
of new features. write this paper to share our cloud implementation experience.
Trends that are pushing DCs to re-envision the network have REFERENCES
three goals in mind:
[1] “Cisco global cloud index: Forecast and methodology,
Scalability: The increased dependency on the network isolation 2014–2019,” White Paper, Cisco, San Jose, CA, USA, Oct.
in a multi-tenant environment has envisaged the need to scale 2015. [Online]. Available: http://www.cisco.com/c/en/us/
the VLANs and the cloud implementation across
solutions/collateral/serviceprovider/global-cloud-indexgci/Clou
geographically separated DCs necessitated the technologies like d_Index_White_Paper.pdf
VXLAN. It decouples the tenants’ state from the state of an
underlying network by tunneling it over an underlay network. [2] M . M ahalingam, D. Dutt, "Virtual eXtensible Local Area
Operational efficiency: As enterprise expands their operations Network (VXLAN): A Framework for Overlay Virtualized
across the globe, the problems that raise due to physical Layer 2 Networks over Layer 3 Networks," RFC7348 (2014),
distance between DCs have to be addressed. The DC network IETF, [Online]. Available: https://tools.ietf.org/html/rfc7348
must support application mobility; applications must migrate
seamlessly within DC and between DCs for business continuity. [3] Steve Cosgrove. (2016). "Teaching Software Defined
The new technologies like EVPN reduces the load on both the Networking: It’s not just coding." 2016 IEEE International
data & control planes and improves the efficiency by the Conference on Teaching, Assessment, and Learnin g for
implementation of features like remote M AC learning thereby Engineering (TALE).
 [4] "IEEE 802.1ah Provider Backbone Bridging," Alcatel-
Lucent. [Online]. Available: https://infoproducts.alcatel-
lucent.com/html/0_add-h-f/93-0076-10-
01/7750_SR_OS_Services_Guide/services_PBB.html

[5] "Design decisions in Open vSwitch" Document, Open

vSwitch, [Online]. Available: https://infoproducts.alcatel-
lucent.com/html/0_add-h-f/93-0076-10-
01/7750_SR_OS_Services_Guide/services_PBB.html

[6] "EVPN Control Plane and VXLAN Data Plane Feature

Guide", Documentation, Juniper Networks Inc. Sunnyvale,
California 94089 USA. [Online]. Available:
https://www.juniper.net/documentation/en_US/junos/topics/con
cept/evpns-overview.html
[7] "Juniper Networks EVPN Implementation for Next -
Generation Data Center Architectures, 2016" White Paper,
Juniper Networks Inc, California, U SA. [Online]. Available:
https://www.juniper.net/assets/us/en/local/pdf/whitepapers/2000
606-en.pdf
[8] Dennis Cai, Sai Natrajan. (2013). "The Evolution of Carrier
Cloud Networking" 2013 IEEE Seventh International
Symposium on Service-Oriented System Engineering
[9] Nabil Bitar, Steven Gringeri & Tiejun J. Xia. (2013).
"Technologies and Protocols for Data Center and Cloud
Networking" IEEE Communications M agazine
[10] "802.1ah - Provider Backbone Bridges", [Online].
Available: http://www.ieee802.org/1/pages/802.1ah.html

View publication stats