WHITE PAPER

10 Gigabit Ethernet Application Overview

Executive Summary

In the past 25 years as the demand for ever-faster networking speed has grown, Ethernet has gone through many changes, starting as a half-duplex shared-media 10-Mbps local-area network (LAN) technology to become a full-duplex switched 10/100/1000/10000-Mbps LAN technology. During this evolution Ethernet has always maintained backward compatibility with the past while displaying the same familiar ease of use and low cost that have been key to its success. It has also grown beyond the limited boundaries of LANs to encompass the realm of long-distance point-to-point links in metropolitan-area networks (MANs).
The huge amount of effort dedicated to evolving Ethernet-based technologies over the years has proven most beneficial to bring Ethernet networks from the campus all the way up to the MAN. Virtual LANs (VLANs) and Layer-3 security features, class of service (CoS), quality of service (QoS), voice-over-IP support, and so on have enabled a whole new generation of applications to tap into the almost unlimited potential of switched Ethernet networks, bringing users a level of speed and flexibility not previously available. Now is the moment that culminates a long and strenuous evolution of an idea, conceived almost three decades ago in a Palo Alto research facility, about the growing importance of computer networks. An idea that is bearing fruits well beyond the expected. Today an ever-increasing number of applications is converging toward an all-Ethernet infrastructure that can provide each user with a single high-speed connection over which data, voice, and video streams can be unicast or multicast at wire-speed while guaranteeing indispensable QoS and security services. The latest standard incarnation of this evergreen technology, 10 Gigabit Ethernet (10 GbE), is nearing completion at the IEEE 802.3ae task force. The draft standard specifies that 10 Gigabit Ethernet will operate only in full-duplex mode. Therefore a 10 Gigabit Ethernet collision detection protocol will no longer be necessary. The draft standard goes to great lengths to extend Ethernet beyond its natural boundaries. It expands the distance for single-mode fiber from the 5 km that Gigabit Ethernet supports to 40 km. It also enables Ethernet packets to travel across SONET links with very little inefficiency. In every other respect, however, 10 Gigabit Ethernet remains true to the original Ethernet model.

Cisco Systems All contents are Copyright 19922001 Cisco Systems, Inc. All rights reserved. Important Notices and Privacy Statement. Page 1 of 10

Introduction

This white paper provides a general overview of some of the very powerful applications of the upcoming 10 Gigabit Ethernet standard to readers interested in cutting-edge network architectures that can support the convergence of storage, data, and multimedia applications. It briefly describes the reasons behind the ever-increasing bandwidth demand and then illustrates a host of different applications that today require 10 Gbps (or larger) data pipes to be able to deliver the required level of network performance.
Need for Speed?

Todays accelerating demand for network bandwidth is associated with several factors and must be considered in a broader scope where other key requirements play an important role. First of all, in a closed system (for example, the earth) the amount of traffic switched across any interconnecting network node is a function of the average traffic demand at the ends of the network. Therefore, it is important to consider the evolution of user applications and environments to better understand the future requirements that the core of a network may need to meet. Researchers have observed that the burstiness in the traffic at the edges of the Internet tends to disappear at the core: the traffic on a high-capacity link becomes more random, regular, smooth, and manageable when the numbers of users and simultaneous computer-to-computer connections go up. It is also well understood by the engineering community that the recent years over-provisioning of network bandwidth has greatly contributed to the overall increase in network efficiency and reliability. For these reasons, it is important to have high-speed data connections both in the core and at the edge of the network to gain better control over the traffic spikes (edge) and to better manage the overall aggregate utilization (core) generated by user demand. Statistical changes in the number of users or in the traffic pattern can be better predicted and easily dealt with in networks that display a limited degree of utilization. Therefore some people (perhaps mistakenly) believe that the current infrastructures are more than sufficient to cope with any future user demands and can graciously scale in the coming years. However, this does not take into account the ongoing technological shift of many households from low-speed access technologies to much higher speed Ethernet-based or cable-based lines, nor does it consider the advances in computer architecture that will likely soon bring about a new crop of high-performance multimedia-ready digital appliances. Any significant technological evolution in a bandwidth-starved marketplace may only mean a wave of new highly demanding applications that can likely tax even the most powerful of todays computers. In the past, over-provisioned core networks and medium-speed uplinks have allowed home users to reliably access the Internet at low or moderate speeds (analog modem, ISDN, and DSL) mainly for Web browsing and file sharing, namely asynchronous applications with very low multimedia content. In contrast, corporate users have enjoyed relatively higher speeds both within their intranets and across their extranets, even if they have just started using new digital-era applications such as voice-over-IP and video conferencing. However, this scenario is rapidly changing, driven by hardware advances and by emerging new powerful applications. Analog modems, ISDN, and low-speed DSL lines are slowly being replaced by high-speed DSL and cable modems in the United States, whereas in Europe several providers are already deploying 10 or 100 Mbps fiber or copper Ethernet connections to the customers homes in metropolitan areas where a lot of dark fiber is readily available. All in all, even if slowly, it is clear now that end-user technologies in the ballpark of 10 Mbps have started to make substantial inroads into todays access networking arena.

Cisco Systems All contents are Copyright 19922001 Cisco Systems, Inc. All rights reserved. Important Notices and Privacy Statement. Page 2 of 10

Enterprise users are beginning to gain access to new PCs and workstations sporting 10/100/1000 network interface cards (NICs) instead of the more common 10/100 cards. Besides, affordable multi-processor systems with faster I/O systems and disks are just around the corner, ready to again push the network utilization close to or beyond its acceptable limits. In data centers, in order to achieve wire-speed transfer rates over single or multi-gigabit links, existing high performance servers employ NICs that utilize an embedded CPU or even a hardware-accelerated TCP/IP stack to boost the network performance of the system. This type of high-end hardware wont take long to become part of the commodity market when the demand for it becomes a differentiation factor for networking equipment vendors. The rapid growth of storage is resulting in the need for storage networks to support ever-growing throughput requirements. Besides, new IP storage technologies such as SCSI over IP (iSCSI), Fibre Channel over IP (FCIP), and Network Data Management Protocol (NDMP) enable enterprises to extend storage consolidation, disaster recovery, and backup solutions to meet exploding storage demands. Therefore, whether they be file-oriented network attached storage (NAS) or block-oriented storage-area networks (SANs), storage networks need to be able to scale well beyond the limits of the current Gigabit Ethernet and Fibre Channel deployments. In addition to the above factors, existing PC and workstation hardware is slowly making it possible to run new high-speed applications that once belonged exclusively to the realm of research institutions and universities. Video and voice applications embedded in conferencing and instant messaging programs, high-speed peer-to-peer file sharing, three-dimensional medical imaging, and distributed computing applications are just a few examples of the sheer potential of a whole new breed of Internet applications that can greatly contribute to stress the limits of the existing networking technologies. In the near future the upgrade from an old generation of access technologies towards the newer Ethernet-based or cable-based lines can potentially bring an overall ten- to hundred-fold increase in the aggregate bandwidth available to residential users if the applications create the necessary demand. In addition, the ever-growing bandwidth need within enterprises and U.S. government institutions is already giving signs of a dramatically new generation of applications that is about to fuel yet another increase in the nations productivity: voice-over-IP and related services, high-definition multimedia streams, massively distributed supercomputing, inter-application high-performance messaging, and so forth. The time has arrived to take another step up the speed ladder of the most widely deployed networking technology in the world: Ethernet. And Cisco Systems, the leader in terms of installed base of Ethernet switch ports, is spearheading the effort to spread the word about yet another technological advance: 10 Gigabit Ethernet.
10 Gigabit Ethernet for the Next-generation Multi-purpose Data Highway

Per IEEE 802.3ae objectives, 10 Gigabit Ethernet meets several key criteria for efficient and effective use in very high-speed networks: Easy migration for existing 10/100/1000 deployments to higher performance levels (ten-fold increase) without disruption Lower cost of ownership versus current alternative technologies Familiar management tools and feature set Ability to support old and new types of applications Overall investment protection due to the flexibility and proven interoperability of the technology In addition, 10 Gigabit Ethernet is paving the way for another exciting new development of this never-ending technological saga: the foundation for another ten-fold increase in speed up to 100 Gbps, and beyond. Thanks to the still infant but very promising 10 Gigabit Ethernet technology, the industry is already discussing the different approaches available to break the 10 Gbps barrier and make inroads into the 40 Gbps and 100+ Gbps realms.

Cisco Systems All contents are Copyright 19922001 Cisco Systems, Inc. All rights reserved. Important Notices and Privacy Statement. Page 3 of 10

QoS, CoS, Layer-4-to-7 intelligence, security and policy-based networking capabilities are just part of the rich feature set that todays Ethernet products can offer. By utilizing these features and the same familiar Layer-2 transport media to stretch over LAN, MAN, and also WAN distances, network managers can deploy 10 Gigabit Ethernet as a cornerstone of efficient and scalable network architectures that support the convergence of voice, video, and data streams from the wiring closet up to the core of the network.
10 Gigabit Ethernet Network Applications

Todays main applications of 10 Gigabit Ethernet in network architectures primarily leverage the following key advantages of the new technology: Long Reach and Extended Long Reach fiber connections Optimum link utilization for high-speed applications Low latency for delay-sensitive applications Support of existing Ethernet features such as QoS, security, multicast, link aggregation, and so on.
High-performance Computing Application

The characteristics listed above are a perfect fit for example for what research institutions and universities recently are striving to achieve. Through advanced programs, they are trying to enable new computational science research, spawn new high-performance computer architectures and networks, and foster computer science research in algorithms, visualization, system software, and tools. In this context, the need for an enormous distributed computing power and high-speed communication links is key to addressing many very demanding applications, such as: Predicting violent storms (hurricanes and tornadoes) Atmospheric modeling to guide environmental policy Designing materials with improved structural and electronic properties (e.g., semiconductors and high-temperature superconductors) Designing intelligent manufacturing systems Performing accurate three-dimensional medical imaging Examining turbulence around surface, airborne, and space vehicles Delineating the basic building blocks of matter and their interactions Understanding the development of the universe and the largest structures within it Uncovering the structure-function relations of viruses Mining large archives of experimental and observational data Modeling the global economy Analyzing critical national infrastructures and their response to adverse conditions

Cisco Systems All contents are Copyright 19922001 Cisco Systems, Inc. All rights reserved. Important Notices and Privacy Statement. Page 4 of 10

To gather all the necessary computing power to run such demanding tasks research facilities can connect isolated high-speed CPUs across the country and use 10 Gigabit Ethernet or 10 Gigabit EtherChannel connections to carry the huge amount of inter-processor communication required to ensure the optimum performance of a massively distributed supercomputer cluster (Figure 1).
Figure 1
Distributed High-performance Computer Architecture

10 GbE

10 GbE 10 GbE

10 GbE 10 GbE 10 GbE 10 GbE

10 GbE

Cisco Systems All contents are Copyright 19922001 Cisco Systems, Inc. All rights reserved. Important Notices and Privacy Statement. Page 5 of 10

Advanced Enterprise Network Architecture

For enterprises, 10 Gigabit Ethernet-based networks can allow them to aggregate the computing power into dedicated sites, such as remote server-farm facilities, that host a variety of high-performance filers, resource and financial management application servers, database servers, data mining systems, and so on. For example, financial institutions can use 10 Gigabit Ethernet to seamlessly and efficiently interconnect remote sites that are dedicated to totally different functions: trading floors in server-less buildings, one or more centralized server farms and a site dedicated to backups and disaster recovery (Figure 2). Delay-sensitive multicast streams can be efficiently delivered from the central data center to the trading floors by exploiting the intrinsic low latency of the 10 Gigabit Ethernet technology and by using QoS to properly classify the different traffic flows.
Figure 2
High-speed Enterprise Network Architecture

Disaster Recovery Site

10 GbE 10 GbE

Up to 50 km Server-less Building Server Farm

Cisco Systems All contents are Copyright 19922001 Cisco Systems, Inc. All rights reserved. Important Notices and Privacy Statement. Page 6 of 10

Next Generation POP Architecture

Service providers are looking for a new platform to do aggregation of OC-12 POS or faster links. A 10 Gigabit Ethernet-based architecture will be able to aggregate the service providers customer links (OC12 and faster) and it will need to use 10 Gigabit Ethernet uplinks from the edge devices toward the core LAN switches. A core LAN switch is a Layer-2 or Layer-3 switch that aggregates uplinks from edge platforms within a point of presence (POP). This architecture will require 10 Gigabit Ethernet interfaces whenever any edge platforms have 10 Gigabit Ethernet uplinks. Usually, edge platforms have multiple Gigabit Ethernet uplinks to core LAN switch and therefore Layer-2 or Layer-3 load balancing support is required. Often, service provider network design favors the use of Gigabit Ethernet interfaces for intra-POP connectivity. However, as the capacity of edge platforms increases, the solution of using multiple Gigabit Ethernet uplinks to core LAN switches, and from core LAN switches to core WAN routers, becomes less and less scalable, and therefore it becomes necessary to adopt a higher speed technology (Figure 3).
Figure 3
High-speed POP Architecture

10 GbE Core

Aggregates OC-12 or Faster

Higher Degree of Scalabililty for TLS Providers

Transparent LAN services (TLS) providers are becoming very popular. They offer a raw Layer-2 data pipe to their customers, enhanced by basic Layer-2 QoS and security services. When the customers connection is a Gigabit Ethernet or a Gigabit EtherChannel link (or sometimes a fraction of their bandwidth), the core of the TLS cloud may need to aggregate multi-gigabit traffic into a highly efficient and redundant network. 10 Gigabit Ethernet and 10 Gigabit EtherChannel technologies represent the ideal choice to offer the necessary link efficiency and redundancy at extremely high speeds (Figure 4). Rate limiting and scheduling of the inbound traffic are two very useful features that can allow the provider to offer fractional speeds to the customers and to better manage the bandwidth at the entry points of the TLS cloud. QoS-enabled 10 Gigabit Ethernet core interconnections can handle the traffic prioritization and can offer very-low-latency delivery of the high priority traffic.

Cisco Systems All contents are Copyright 19922001 Cisco Systems, Inc. All rights reserved. Important Notices and Privacy Statement. Page 7 of 10

Figure 4

High-speed Transparent LAN Services (TLS) Network Architecture

Customer A L2 or L3 Core

Customer D

10 GbE Links Aggregates Single or Multiple GbE Links Customer B Customer C

Aggregation of Gigabit Ethernet Traffic in Server Farms

The sheer demand of data bandwidth from modern file and data servers is astonishing: a single high-performance filer can host several Gigabit Ethernet NICs and thanks to its optimized system software and accelerated NIC hardware it can easily consume all of the cards available bandwidth. This huge processing power, whether it is for NFS transfers or transaction processing or mission-critical applications, strains the capacity of any high-performance network to its limits. Only by deploying a very-high speed technology like 10 Gigabit Ethernet in the distribution and core layers of the server farms it is possible to make their growth path much smoother and deal with the rapidly increasing demand of cheap raw bandwidth for high-profile applications (Figure 5).

Cisco Systems All contents are Copyright 19922001 Cisco Systems, Inc. All rights reserved. Important Notices and Privacy Statement. Page 8 of 10

The Cisco Catalyst 6500 series today supports a 16-port fabric-enabled 10/100/1000Base-T Ethernet module that can deliver the level of performance required by high-end servers and bandwidth-intensive applications in the data center. To maintain the same level of performance in the distribution and core layers, the network designer can use 10 Gigabit Ethernet uplinks to efficiently aggregate the access-layer traffic towards the 10 Gigabit Ethernet-enabled Internet routers.
Figure 5
High-performance Server Farm

File/Application Servers

Catalyst 6500 Provides 10/100/1000 Mb Connections for Servers

Catalyst 6500 Provides HighDensity Gigabit Ethernet Interconnections for Distribution and Access Switches Catalyst 6500 Provides HighSpeed 10 Gigabit Ethernet Interconnections for Distribution and Core Switches Internet

Conclusion

Significant deployment of dark fiber in the metropolitan areas is enabling service providers to offer higher amounts of cheap bandwidth to end users. At the same time, it is important for both service providers and end users to be able to benefit from an ever-growing variety of services. For this purpose, Ethernet is becoming the ideal target for the much-awaited media convergence revolution, when almost every service will be delivered through an omnipresent Ethernet connection to digital-media-enabled devices: application data transfers, file sharing, messaging, asynchronous backup transfers, remote storage, isochronous and/or media-rich application streams, music or video on demand downloads, wireless base station connections, high-speed distributed computing interprocessor communication (IPC), and so forth. Figure 6 represents a model of network whose entire infrastructure is based on Ethernet.

Cisco Systems All contents are Copyright 19922001 Cisco Systems, Inc. All rights reserved. Important Notices and Privacy Statement. Page 9 of 10

Figure 6

Enabling the Network Media Convergence

Storage Router

GbE

Tape 10 GbE 10 GbE DWDM GbE

GbE

10 GbE

FC NAS GbE ESCON SAN NAS = Networked Attached Storage SAN = Storage-Area Network FC = Fiber Channel DWDM = Dense Wavelength Division Multiplexing ESCON = Enterprise Systems CONnection V V IP Phones Wireless Base Stations

Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100

European Headquarters Cisco Systems Europe 11, Rue Camille Desmoulins 92782 Issy-les-Moulineaux Cedex 9 France www-europe.cisco.com Tel: 33 1 58 04 60 00 Fax: 33 1 58 04 61 00

Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA www.cisco.com Tel: 408 526-7660 Fax: 408 527-0883

Asia Pacific Headquarters Cisco Systems Australia, Pty., Ltd Level 9, 80 Pacific Highway P.O. Box 469 North Sydney NSW 2060 Australia www.cisco.com Tel: +61 2 8448 7100 Fax: +61 2 9957 4350

Cisco Systems has more than 200 offices in the following countries and regions. Addresses, phone numbers, and fax numbers are listed on the

Cisco Web site at www.cisco.com/go/offices

Argentina Australia Austria Belgium Brazil Bulgaria Canada Chile China PRC Colombia Costa Rica Croatia Czech Republic Denmark Dubai, UAE Finland France Germany Greece Hong Kong SAR Hungary India Indonesia Ireland Israel Italy Japan Korea Luxembourg Malaysia Mexico The Netherlands New Zealand Norway Peru Philippines Poland Portugal Puerto Rico Romania Russia Saudi Arabia Scotland Singapore Slovakia Slovenia South Africa Spain Sweden Switzerland Taiwan Thailand Turkey Ukraine United Kingdom United States Venezuela Vietnam Zimbabwe
Copyright 2001, Cisco Systems, Inc. All rights reserved. Printed in the USA. AccessPath, AtmDirector, Browse with Me, CCIP, CCSI, CD-PAC, CiscoLink, the Cisco Powered Network logo, Cisco Systems Networking Academy, the Cisco Systems Networking Academy logo, Fast Step, Follow Me Browsing, FormShare, FrameShare, GigaStack, IGX, Internet Quotient, IP/VC, iQ Breakthrough, iQ Expertise, iQ FastTrack, the iQ logo, iQ Net Readiness Scorecard, MGX, the Networkers logo, Packet, RateMUX, ScriptBuilder, ScriptShare, SlideCast, SMARTnet, TransPath, Unity, Voice LAN, Wavelength Router, and WebViewer are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, Discover All Thats Possible, and Empowering the Internet Generation are service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, the Cisco IOS logo, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Enterprise/Solver, EtherChannel, EtherSwitch, FastHub, FastSwitch, IOS, IP/TV, LightStream, MICA, Network Registrar, PIX, Post-Routing, Pre-Routing, Registrar, StrataView Plus, Stratm, SwitchProbe, TeleRouter, and VCO are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries. All other trademarks mentioned in this document or Web site are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0106R) 08/01 LW2590