Data Center

A data center is a facility that provides shared access to

applications and data using a complex network, compute,
and storage infrastructure.
Industry standards exist to assist in designing, constructing, and
maintaining data center facilities and infrastructures to ensure
the data is both secure and highly available.
 Traditional Data Center
A traditional data center is a facility housing IT equipment, such
as servers and routers.
Data center hardware components and technical elements
include:
Compute, the memory and processing power
Storage to house data
Networking to connect the components and to access the
internet
Power and Cooling Infrastructure
 From the history of IT, traditional data centers tended to
be on-premises, often in conjunction with a main corporate
office.
Today, businesses can also select colocations (facilitates run
by third-party data center operators) that operate in a
traditional or virtualized manner.
A physical, on-premises data center has various advantages:
If you’re starting from scratch, you can build it to suit your own needs. When
upgrades are required.
You can restrict access to the facility to individuals from within your own
organization, as well as trusted providers should you so choose.
Traditional data centers located on-premises can deliver low-latency access
to applications.
Disadvantages
Construction and ongoing data center infrastructure management
are typically more expensive for traditional data centers.
Traditional data centers are less energy efficient and require more
cooling.
A traditional data center relies on static IP addresses.
Some organizations suffer from hardware vendor lock-in
Traditional Data Center Architecture
Servers mounted on 19’’ rack
cabinets

Racks are placed in single rows forming

corridors between them.

• Src: the datacenter as a computer – an introduction to the design of warehouse-scale machines

Virtualized Data Center
Data center virtualization uses virtualized servers and virtual machines
(VMs).

A single physical server can house multiple virtual servers. Repeat such
virtualization over all the servers in an environment and you have data
center virtualization.

Data center virtualization is central to cloud computing, including private

and public clouds. It also makes infrastructure as a service (IaaS) possible.
Advantages of Data Center Virtualization
Reduce dependency on a single hardware
Better Disaster Recovery
Better Cooling
Cost Saving
Less Data Center Footprint
Faster deployment of new resources
 Main components of a datacenter

src: The Datacenter as a Computer – Barroso, Clidaras, Holzle

 Main components of a datacenter
Power subsystem Instead of generating the large hyper-scale 32MW/h
Uninterruptable power supplies (UPS) datacenters, they moved to creating smaller units of
Ventilation and cooling systems datacenters that could be composed to create larger units, but
Backup generators also more agile when deploying elsewhere. In addition, that
Cabling to connect to external network enabled them to become even more efficient.
operators
Gen 7, was deployed until last year, they simplified the design
Routers, switches, security devices, storage the electrical system, by intelligent monitoring and software
systems, servers, application delivery throttling of power usage, etc.
controllers, and more.
Reliability, efficiency, security, and constant
evolution of a datacenter are typically a top
priority.

src: The Datacenter as a Computer – Barroso, Clidaras, Holzle

Inside a modern data center
§ Today’s DC use shipping containers packed with 1000s
servers each.

§ For repairs, whole containers are replaced.

Costs for operating a data center
§ DCs consume 3% of global electricity supply
(416.2 TWh > UK’s 300 TWh)
Monthly cost = $3’530’920

§ DCs produce 2% of total greenhouse gas Servers

emissions 4%
13% Networking Equipment

§ DCs produce as much CO2 as The Netherlands

or Argenti Power Distribution &
18% Cooling
57% Power

31% power 8%
Other Infrastructure

45,978 servers, 3yr server & 10 yr infrastructure amortization

45,978 servers, 3yr server & 10 yr infrastructure amortization
Evolution of datacenter design
§ Gen 6: scalable form factor (2017)
- Reduced infrastructure, scale to demand
- 1.17-1.19 PUE

§ Gen 7: Ballard (2018)

- Design execution efficiency
- Flex capacity enabled
- 1.15-1.18 PUE

§ Gen 8: Rapid deploy datacenter (2020)

- Modular construction and delivery
- Equipment skidding and preassembly
- Faster speed to market

§ Project Natick (future) – 1.07 PUE or less

Src: Inside Azure Datacenter Architecture with Mark Russinovich

1. Project Natick: (future) rapid deployment,

close to population centers, high energy
Challenge 1: Cooling data centers

Cooling plant at a Google DC in Oregon

Challenge 2: Energy Proportional Computing
§ Average real world DC and servers are too
inefficient. Sub-system power usage in an x86 server as the compute
- waste 2/3+ of their energy load varies from idle to full (reported in 2012).

§ Energy consumption is not proportional

to the load
- CPUs are not so bad but the other
components are
- CPU is the dominant energy consumer in
servers – using 2/3
of energy when active/idle.

§ Try to optimize workloads

§ Virtualization and consolidation. • src: “The Datacenter as a Warehouse Computer”
Challenge 3: Servers are idle most of the time
§ For non-virtualized servers 6-15% utilization

§ Server virtualization can boost to

an average 30% utilization

§ Need for resource pooling and application

and server consolidation

§ Need for resource virtualization

src: Luiz Barroso, Urs Hölzle “The Datacenter as a Computer”

Challenge 4: Efficient monitoring
§ Even with virtualization and software defined DC,
resource utilization can be poor.

§ Need for efficient monitoring (measurement) and cluster

management.

§ Goal to meet SLOs and SLIs.

§ Job’s tail latency matters!

src: “Heterogeneity and dynamicity of clouds at scale: Google trace analysis” SoCC’12
Improving resource utilization
§ Hyper-scale system management software
• Treat the datacenter as a warehouse scale computer
• Software defined datacenters
• System software that allows DC operations to manage the entire DC infrastructure
• Compose a system using pooled resources of
compute, network, and storage based on workload requirement

§ Dynamic resource allocation

• Virtualization is not enough to improve efficiency
• Need the ability to dynamically allocate CPU resources across servers and racks, allowing admins to
quickly migrate resources to address the shifting demand
• Drive 100-300% better utilization for virtualized WLs, and 200-600% for bare-metal WLs.
 Software-defined datacenter (SDDC)

Create a virtual version of devices like network, storage device,

operating system, and even a server.
The SDDC brings the data center and virtualization together
across infrastructure with not only virtualized servers but also
software-defined storage and software-defined networking.
 Software-defined networking (SDN)
• SDN (software-defined networking) is the separation of control and data
planes
• The separation allows control topology to be independent of physical network
topology

Why would anyone want to do this?

That question has a lot of answers…

Logically centralized control plane

e.g. OpenFlow

Data Plane
 SDN definition and concept
• Definition
• A technology to networking which allows centralized,
programmable control planes so that network operators can
control and manage directly their own virtualized networks
• Basic concept
• (1) separation of control and data planes,
• (2) centralized, programmable control planes of network
equipment, and
• (3) support of multiple, isolated virtual networks
Basic SDN concept
Control Plane Control Plane
Network OS
Network OS Separation
Data Plane
(Specialized Hardware)
Data Plane
(Forwarding Abstraction)
a.1 control and data planes tightly-coupled b.1 control and data planes separated

SDN
Apps SDN
Languages
Controller
(NetworkOS)

Server
Client

a.2 distributed control of network equipments

Server
Client
b.2 centralized, programmable control of network equipments

e.g., Content Routing Mobilty

a.3 single physical network

Energy efficient forwarding
b.3 Multiple, isolated, virtual networks

a. Current networking technology b. SDN networking technology

Software Defined Network
A network in which the control plane is
physically separate from the data plane.

and

A single (logically centralized) control plane controls

several forwarding devices.
Software Defined Network (SDN)

Control Control Control

Program Program Program

Global Network Map

Control Plane

Control
Packet
Forwarding Control
Packet
Forwarding
Control
Packet
Control Forwarding
Packet
Forwarding Control
Packet
Forwarding
The Networking “Planes”
• Data plane: processing and delivery of packets with local
forwarding state
• Forwarding state + packet header à forwarding decision
• Filtering, buffering, scheduling

• Control plane: computing the forwarding state in routers

• Determines how and where packets are forwarded
• Routing, traffic engineering, failure detection/recovery, …

• Management plane: configuring and tuning the network

• Traffic engineering, ACL config, device provisioning, …
Data and Control Planes
control plane
data plane Processor

Line card Line card

Switching
Line card Line card
Fabric

Line card Line card

26
Data Plane
• Streaming algorithms on packets
• Matching on some header bits
• Perform some actions

• Example: IP Forwarding

1.2.3.4 1.2.3.7 1.2.3.156 5.6.7.8 5.6.7.9

host host ... host host host ... host

LAN 1 LAN 2
router router router
WAN WAN

1.2.3.0/24
5.6.7.0/24

forwarding table
Control Plane
• Compute paths the packets will follow
• Populate forwarding tables
• Traditionally, a distributed protocol

• Example: Link-state routing (OSPF, IS-IS)

• Flood the entire topology to all nodes
• Each node computes shortest paths
• Dijkstra’s algorithm
 Software-Defined Networking

Virtualizing the Network

§ Two major network architectures have been developed to support network virtualization:
• Software Defined Networking (SDN) - A network architecture that virtualizes the network.
• Cisco Application Centric Infrastructure (ACI) - A hardware solution for integrating cloud computing
and data center management.
§ These are some other network virtualization technologies, some of which are included as
components in SDN and ACI:
• OpenFlow - The OpenFlow protocol is a basic element in building SDN solutions.
• OpenStack - This approach is a virtualization and orchestration platform available to build scalable
cloud environments and provide an infrastructure as a service (IaaS) solution.
 Software-Defined Networking

SDN Architecture
§ In a traditional router or switch architecture, the control plane and data plane functions occur in
the same device. Routing decisions and packet forwarding are the responsibility of the device
operating system.
§ Software defined networking (SDN) is a network architecture that has been developed to
virtualize the network. SDN can virtualize the control plane. SDN moves the control plane from
each network device to a central network intelligence and policy-making entity called the SDN
controller.
 Software-Defined Networking

SDN Architecture
§ The SDN controller enables network administrators to manage
and dictate how the data plane of virtual switches and routers
should handle network traffic.
§ The SDN controller uses northbound APIs to communicate with
the upstream applications. These APIs help network
administrators shape traffic and deploy services.

§ The SDN controller also uses southbound APIs to define the

behavior of the downstream virtual switches and routers.
§ An API is a set of standardized requests that define the proper way
for an application to request services from another application.
§ OpenFlow is the original and widely implemented southbound
API.
 SDN Types
§ To better understand APIC-EM, it is helpful to take a
broader look at the three types of SDN:
• Device-based SDN - The devices are programmable by
applications running on the device itself or on a server
in the network. Cisco OnePK is an example of a device-
based SDN. It enables programmers to build
applications to integrate and interact with Cisco
devices.
• Controller-based SDN - Uses a centralized controller
that has knowledge of all devices in the network. The
applications can interface with the controller
responsible for managing devices and manipulating
traffic flows throughout the network. The Cisco Open
SDN Controller is a commercial distribution of
OpenDaylight.
 SDN Types
• Policy-based SDN - Similar to controller-based SDN
where a centralized controller has a view of all devices
in the network. Includes an additional Policy layer. Uses
built-in applications that automate advanced
configuration tasks via a guided workflow and user-
friendly GUI. No programming skills are required.
Cisco APIC-EM is an example of this type of SDN.
§ Policy-based SDN is the most robust, providing for a
simple mechanism to control and manage policies across
the entire network.
Traditional Control and Data Planes

Control Plane
• Protocols: BGP, OSPF, RIP
• RIB: Collection of Link/Path Attributes
Routing Table • Northbound Configuration Interface
Control Plane
(RIB) − e.g., Cisco CLI

Data Plane
• Protocols: IP
Forwarding Table • FIB: Optimized for Fast Lookup
Data Plane
(FIB) • Northbound Control Interface
− Historically Private/Internal
 SDN Control and Data Planes

Control Control Control Control

App App App ... App

Global
Network
Network OS Map
Control Plane
Data Plane
Flow Rules
 Management, Control and Data Planes
Network topology
request

Request stored
and acknowledged
MANAGEMENT
PLANE

Desired State
Calculate data
CONTROL plane state
PLANE

Discovered State Translated State

DATA
PLANE Identify data plane
resources

Realized State
 Traffic Engineering
Network Policies

Controller

Datacenter Datacenter

Datacenter
 Datacenter Switching Fabric
Internet

Leaf-Spine Topology
• Leaf Switches = Top-of-Rack (ToR)
Spine Spine Spine • Optimized for East-West Traffic
• Built-in Redundancy (not shown)
• Scale with additional layers

Well-Established in Commodity Clouds

• Bare-Metal Switches
Leaf Leaf Leaf Leaf • Control Plane running in the cloud
 Leaf-Spine Switching Fabric

Spine Spine Spine Trellis Design

• Intra-Rack: L2 Domain within L3 Subnet
• Inter-Rack: L3 Routing between Subnets
• Segment Routing across Fabric
Trellis Features
Leaf Leaf Leaf Leaf • VLANs / QinQ
• End-to-End L2 Tunnels
• IPv4 / IPv6 Routing
• Multicast (with IGMP)
• ARP (IPv4) / NDP (IPv6)
• DHCPv4 / DHCPv6
• High Availability
 Inband Network Telemetry (INT)
Add Switch ID, arrival time, Generate report with
departure, queue delay, etc. switch metadata
Header
Header Metadata S1
Metadata S1 Metadata S2

Header Payload S2 Payload Header

Payload Payload

S1 S3 S5
Metadata S1
Metadata S2
Fine-Grain Telemetry
Metadata S5
• Flow Rule(s) that matched S4
• Queuing delays of individual packets
• Other flows being buffered
• … Log, analyze,
replay, visualize
Uses
• Verify correct behavior
• Identify micro-bursts
• …
 SDN Challenges

• Scale
• Stability & Correctness
• Timeliness
• Inter-domain
 SDN Additional Reading :

https://study-ccna.com/cisco-sdn-software-defined-networking/
https://networklessons.com/cisco/ccna-routing-switching-icnd2-200-105/introduction-to-sdn-
software-defined-networking