Brian W.

Hugenbruch, CISSP z/VM Development Team, IBM: Endicott, NY, USA

Link Aggregation for the z/VM Virtual Switch

2010 IBM Corporation

Trademarks
The following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.
Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market. Those trademarks followed by are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.

For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:

*, AS/400, e business(logo), DBE, ESCO, eServer, FICON, IBM, IBM (logo), iSeries, MVS, OS/390, pSeries, RS/6000, S/30, VM/ESA, VSE/ESA, WebSphere, xSeries, z/OS, zSeries, z/VM, System i, System i5, System p, System p5, System x, System z, System z9, BladeCenter

The following are trademarks or registered trademarks of other companies.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.
* All other products may be trademarks or registered trademarks of their respective companies. Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

2 2

2010 IBM Corporation

Agenda

Overview of Virtual Switch and Virtual NICs Link Aggregation Requirements Configuration and Commands LACP Active vs. Inactive Benefits

Questions?

2010 IBM Corporation

2010 IBM Corporation

Whats a switch anyway?

Cisco Corp

A box that creates a LAN It can be remotely configured E.g. Turn ports on and off Similar to a home router

2010 IBM Corporation

z/VM Virtual Switch

A special-purpose Guest LAN Ethernet Built-in IEEE 802.1q bridge to outside network IEEE VLAN capable Each Virtual Switch has up to 8 separate OSA-Express connections associated with it Created in SYSTEM CONFIG or by CP DEFINE VSWITCH command
z/VM 5.3

guest

guest

guest

CP

Router

AIX

2010 IBM Corporation

Guest LAN vs. Virtual Switch

Virtual router is required Different subnet External router awareness Guest-managed failover

No virtual router Same subnet Transparent bridge CP-managed failover

2010 IBM Corporation

z/VM Virtual Switch VLAN unaware

Linux

VM TCP/IP

VSE

z/OS
Virtual QDIO adapter

Virtual Switch Guest LAN CP OSA-Express Ethernet LAN Same LAN segment and subnet

Access port

2010 IBM Corporation

A VLAN-aware switch: An inside look

2 2 2 2 3 3

4 4 4 4

2010 IBM Corporation

Trunk Port vs. Access Port

Access port carries traffic for a single VLAN

T 2 4 2 4 3 2 4 4 3 2 4

Host not aware of VLANs

Trunk port carries traffic from all VLANs Every frame is tagged with the VLAN id
10

T 2

2 4

4 3

2 4

3 3

3 4

2010 IBM Corporation

Physical Switch to Virtual Switch

T 2

4 2

4 3

2 4

4 3

2 4

Trunk port carries traffic between CP and switch

Each guest can be in a different VLAN

2 4 2 3 3 4

CP

Virtual Switch

11

2010 IBM Corporation

z/VM Virtual Switch VLAN aware

Linux

VM TCP/IP

VSE

z/OS
Virtual QDIO adapter

Virtual Switch Guest LAN CP OSA-Express Ethernet LAN IEEE 802.1q transparent bridge Multiple LANs

Trunk port

12

2010 IBM Corporation

Network with VSWITCH

LPAR 1 web web web web LPAR 2

z/VM
web app app app

z/OS DB2

VSWITCH

To internet

With 1 VSWITCH, 3 VLANs, and a multi-domain firewall

13 2010 IBM Corporation

Virtual Switch Attributes

Name Associated OSAs Access List One or more controlling virtual machines (minimal VM TCP/IP stack servers)
User pre-configured DTCVSW1 and DTCVSW2 Controller not involved in data transfer Starts, stops, and monitors OSAs Do not ATTACH or DEDICATE

Similar to Guest LAN

Owner SYSTEM Type QDIO Persistent Restricted
14 2010 IBM Corporation

Create a Virtual Switch

SYSTEM CONFIG or CP command:

DEFINE VSWITCH name [RDEV NONE | cuu.Pnn [cuu.Pnn [cuu.Pnn]] ] [CONNECT | DISCONNECT] [CONTROLLER * | userid] [IP IPTIMEOUT 5 NONROUTER | ETHERNET] [NOGroup | GROup groupname] [VLAN UNAWARE | VLAN def_vid NATIVE native_vid] [PORTTYPE ACCESS | PORTTYPE TRUNK]

Example: DEFINE VSWITCH SWITCH12 RDEV 1E00 1F04 CONNECT

15

2010 IBM Corporation

Change the Virtual Switch access list

Specify after DEFINE VSWITCH statement in SYSTEM CONFIG to add users to access list

MODIFY VSWITCH name GRANT SET

userid [VLAN vid1 vid2 vid3 vid4] [PORTTYPE ACCESS | TRUNK] [PROmiscuous | NOPROmiscuous]

SET

VSWITCH name REVOKE userid

Examples: MODIFY VSWITCH SWITCH12 GRANT LNX01 VLAN 3 7 105 CP SET VSWITCH SWITCH12 GRANT LNX02 PORTTYPE TRUNK VLAN 4-20 22-29 CP SET VSWITCH SWITCH12 GRANT LNX03 PRO

CP QUERY LAN or VSWITCH to display current access list (see DETAILS option)
16 2010 IBM Corporation

17

2010 IBM Corporation

Virtual Network Interface Card (NIC)

A simulated network adapter OSA-Express QDIO HiperSockets Must match LAN type 3 or more devices per NIC More than 3 to simulate port sharing on 2nd-level system or for multiple data channels Provides access to Guest LAN or Virtual Switch
Guest LAN or virtual switch

Virtual Machine

Created by directory or CP DEFINE NIC command

18 2010 IBM Corporation

Virtual NIC - User Directory

May be automated with USER DIRECT file:

NICDEF vdev [TYPE HIPERS | QDIO] [DEVices devs] [LAN owner name] [CHPID xx] [MACID xxyyzz] Example:

Combined with VMLAN MACPREFIX to create virtual MAC

NICDEF 1100 LAN SYSTEM SWITCH1 CHPID B1 MACID B10006

19

2010 IBM Corporation

Virtual NIC - CP Command

May be interactive with CP DEFINE NIC and COUPLE commands:

CP DEFINE NIC vdev [[TYPE] HIPERsockets|QDIO] [DEVices devs] [CHPID xx] CP COUPLE vdev [TO] owner name Example: CP DEFINE NIC 1200 TYPE QDIO CP COUPLE 1200 TO SYSTEM CSC201

20

2010 IBM Corporation

21

2010 IBM Corporation

Traditional VSWITCH Setup

controller

guest

guest

guest

CP

OSA

OSA

Up to 8 OSAs per VSWITCH Define a VSWITCH with 3 RDEVs

22

Use one for data transfer, keep two as back-up devices Automatic failover
2010 IBM Corporation

Traditional VSWITCH Setup

controller

guest

guest

guest

CP

OSA

OSA X

If OSA dies or stalls, controller will detect it and switch to backup OSA
Failover to a back up OSA causes a brief network outage Improved from release to release, but a brief outage is still technically an outage
2010 IBM Corporation

23

Motivation for Link Aggregation

But why arent you using my back up OSA card?

24

2010 IBM Corporation

Link Aggregation
Group two or more ports together to form a logical fat pipe between two switches

S W I T C H

S W I T C H

IEEE 802.3ad

Cascading Switches

25

2010 IBM Corporation

VSWITCH Link Aggregation Specifications

Group multiple active QDIO VSWITCH real OSA connections as a single logical group Up to 8 OSA ports (within a group or as backup devices) Synchronized conversations over the same OSA link Only one aggregate group per VSWITCH No support for aggregation of virtual NICs 802.3ad compliance for Layer 2 ETHERNET VSWITCH only MAC level implementation transparent to all connected NICs or protocols
26 2010 IBM Corporation

VSWITCH Link Aggregation Specifications

Port group management

Dynamic (LACP ACTIVE) Static (LACP INACTIVE)

Near seamless failover

Port failover to another port within the group Group failover to a single backup port (existing failover support)

Minimal link selection overhead Ability to distribute single guest port traffic across multiple OSA connections. External controls using existing commands and a new SET PORT Command

27

2010 IBM Corporation

Hardware Requirements

Dedicated OSA Express2 or Express3 Ports

Same type of NIC card (10, 100,1000 and 10000 mbps) Point to point connection to the same switch Support of IEEE 802.3ad by both switches Full duplex mode (send and receive paths) VLAN considerations

All member OSA ports within the group must be trunk links to provide the virtual LAN connectivity in which to flow tagged traffic Aggregated link should be viewed as one logical trunk link containing all the VLANs required by the LAN segment

28

2010 IBM Corporation

Hardware Features
Exclusive Port Mode
Single QDIO Connection
The ability to establish an exclusive QDIO connection on an OSA port . Once the connection is established, the port can no longer be shared within this or any other LPAR. Any attempt to establish another connection on the port will be prevented as long as the exclusive QDIO connection is active.

Automatic Port Disablement / Enablement

When an exclusive QDIO connection leaves the QDIO Active state, the OSA port will be automatically disabled until the next QDIO connection is established. By disabling the OSA port, the connected switch port is notified the link is no longer operational. This provides a signal to the partner switch to route future traffic to another port within the group.

29

2010 IBM Corporation

Simple Virtual Switch LAN Segment

Create a simulated Layer 2 or Layer 3 switch device Virtual machine access control and VLAN authorization Create ports and connect NIC to virtual switch (LAN Segment) Provides full MAC address management (generation and assignment)
Linux
NIC

(VSWITCH)
Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

Port 65

Port 66

Port 67

Port 68

Port 69

Port 70

Virtual Switch

Forwards traffic between Guest Ports by either IP or MAC address 1-n VSWITCHs per z/VM image

z/VM System z LPAR

Example
Create VSWITCH from PRIVCLASS B User ID DEF VSWITCH VSWITCH1 ETHERNET SET VSWITCH VSWITCH1 GRANT {user ID} From Linux Virtual Machines DEF NIC 100 TYPE QDIO COUPLE 100 SYSTEM VSWITCH1
Physical Switch

30

2010 IBM Corporation

Cascading a Virtual to a Physical Switch

Linux Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

Start VM TCPIP Controllers

NIC

VM TCPIP VM TCPIP Controller Controller

XAUTOLOG DTCVSW1 XAUTOLOG DTCVSW2

Port 65 Port 66 Port 67 Port 68 Port 69 Port 70

Virtual Switch
Port 1

z/VM System z LPAR

QDIO Connection (3 Devices)

Connect the Real Switch SET VSWITCH VSWITCH1 RDEV 100

OSA

Read Control Device Write Control Device Data Device

Port 1

Physical Switch

31

2010 IBM Corporation

Adding a Failover Device

Linux Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

Issue the SET VSWITCH command and include the new RDEV

VM TCPIP VM TCPIP Controller Controller

NIC

Port 65

Port 66

Port 67

Port 68

Port 69

Port 70

Virtual Switch
Port 1 Port 2

z/VM System z LPAR

OSA

Example SET VSWITCH VSWITCH1 RDEV 100 500 SET VSWITCH VSWITCH1 CONNECT

OSA
Port 1

Backup Physical Switch

Port 1

Physical Switch

32

2010 IBM Corporation

Port Failover
Linux Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

VM TCPIP VM TCPIP Controller Controller

Port Error

NIC

Port 65

Port 66

Port 67

Port 68

Port 69

Port 70

Virtual Switch
Port 1 Port 2

z/VM System z LPAR

QDIO connection terminated on the primary OSA device and is established and activated on the BACKUP device Only one QDIO Connection is active at any point in time

OSA OSA
Port 1

Backup Physical Switch

Port 1

Physical Switch

33

2010 IBM Corporation

Defining Port Groups Two step process to create a LinkAG port configuration 1. Create a port group using new SET PORT CP Command 2. Associate a port group with an ETHERNET type VSWITCH
Create a Port Group SET PORT GROUP ETHGRP JOIN 500 600 700 800 SET PORT GROUP ETHGRP LACP INACTIVE

Display INACTIVE Port Groups Q PORT GROUP INACTIVE

Group: ETHGRP VSWITCH <none> RDEV: 0500 RDEV: 0600 RDEV: 0700 RDEV: 0800

Inactive

LACP Mode: Inactive Interval: 300

Display ACTIVE Port Groups Q PORT GROUP

34

HCPSWP2837E No active groups found.

2010 IBM Corporation

SET or MODIFY PORT GROUP

Use the SET or MODIFY PORT command to define or change the OSA Express2 devices that make up a link aggregation group and to set the attributes of a link aggregation group.
Privilege Class: B >>---SET-PORT-GROup groupname -++++| ++------+ | |(1) v | JOIn --+---rdev --------+--->< LEAve -+ | DELete -----------------+ LACP -+- ACTive -----+--+ +- INActive ---+ | INTerval--+- nnnn -+----+ +- OFF --+
(2)

Note:

(1) You can specify a maximum of 8 real device numbers (2) Operands that may be specified while the group is ACTIVE

35

2010 IBM Corporation

QUERY PORT GROUP CP Command

Use the QUERY PORT command to display information about link aggregation groups or devices that have been defined for virtual switches on the system.
Privilege Class: B +-ALL--ACTive-------+ >--Query--PORT--+-GROup--+-------------------+-+--+---------+----->< | | +-ACTive---+ | | +-DETails-+ | +-ALL--+----------+-| | | | +-INActive-+ | | | +-groupname---------+ | '-+------------+---------------+ +-RDEV--rdev-+

36

2010 IBM Corporation

Display Routing Table

Query PORT GROup name DETails

Group: ETHGRP Active LACP Mode: Active VSWITCH SYSTEM SWITCH1 Interval: 300 Group: ETHGRP Active LACP Mode: Active GROUP Information: VSWITCH SYSTEM SWITCH1 Interval: 300 PORT Information - Total Frames per Interval: GROUP Information: Device Status PORT Information - Previous Total Frames per Interval: 0510 Active 7 Device Status 11 Previous 0520 7 7 0510 Active Active 11 11 ROUTING Information - Frame Distribution per Interval: 0520 Active 11 7 MAC Previous Current ROUTING Device Information - Frame Distribution per Interval: 0 MAC 0510 0 Current Device 0 Previous 1 0 0520 0 0 0 0 0510 2 1 0510 0 0 0 0 0520 3 2 0520 0 0 0 0 0510 4 3 0510 0 0 0 0 0520 5 4 0520 0 0 0 0 0510 6 5 0510 0 0 0 0 0520 7 6 0520 0 0 0 0 0510 7 0520 0 0

37

2010 IBM Corporation

LACP INACTIVE LinkAG Group

Associate a port group with an ETHERNET type VSWITCH
Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

VM Controllers

Disconnect the Physical Switch

Port 65 Port 66 Port 67 Port 68 Port 69 Port 70

SET VSWITCH VSWITCH1 DISCON

Load Balance - Distributor/Collector

Actor

Setup Partner Switch for a LACP INACTIVE port

Virtual Switch
Port 1 Port 2 Port 3 Port 4

z/VM System z LPAR

Associate the Port Group SET VSWITCH VSWITCH1 GROUP ETHGRP

OSA

OSA

OSA

OSA

Static Logical Port Group

Connect the Port Group SET VSWITCH VSWITCH1 CONNECT

Port 1

Port 2

Port 3

Port 4

Partner
Load Balance - Distributor/Collector

Physical Switch

38

2010 IBM Corporation

LACP ACTIVE LinkAG Group

Create a Dynamically Managed LinkAG Port Group
Disconnect the Physical Switch SET VSWITCH VSWITCH1 DISCON
Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

VM Controllers

Port 65

Port 66

Port 67

Port 68

Port 69

Port 70

Load Balance - Distributor/Collector

LACP and Marker Protocol

Port 1 Port 2 Port 3 Port 4

Actor Virtual Switch z/VM System z LPAR

Setup Partner Switch for a LACP ACTIVE port Make Port Group LACP ACTIVE
LACP Packets

OSA

OSA

OSA

OSA

SET PORT GROUP ETHGRP LACP ACTIVE

Active Port Group Formed After LACP Negotiations

Connect the Port Group SET VSWITCH VSWITCH1 CONNECT

LACP and Marker Protocol

Port 1

Port 2

Port 3

Port 3

Partner
Load Balance - Distributor/Collector

Physical Switch

39

2010 IBM Corporation

Switch Failover to Traditional Backup Device

LinkAG group can be setup to failover to a single port on another switch
Select another physical switch on the same LAN segment Add the BACKUP device SET VSWITCH VSWITCH1 RDEV 100
LACP Packets
Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

Linux
NIC

VM Controllers

Port 65

Port 66

Port 67

Port 68

Port 69

Port 70

Load Balance - Distributor/Collector

LACP and Marker Protocol

Port 1 Port 2 Port 3 Port 4

Actor Virtual Switch

Port 5

z/VM System z LPAR

OSA OSA OSA OSA OSA

Port 1

Switch Error
VM automatically establishes and activates the QDIO connection on the BACKUP device
40

Physical Switch

LACP and Marker Protocol

Port 1

Port 2

Port 3

Port 3

Load Balance - Distributor/Collector

Partner Physical Switch

2010 IBM Corporation

Advantages of a LACP ACTIVE Port Group (Recommended)

Ports can be added or removed dynamically within the LinkAG group Changes made on one switch are automatically made on the other switch Immediate packet rerouting Fast nearly seamless failover to another port within the group Adding or removing capacity is not disruptive LACP Protocol provides a heartbeat mechanism Marker Protocol allows greater flexibility to dynamically move work from one port to another within the group Automatic fail-back from the backup device to a port group

41

2010 IBM Corporation

Virtual Switch Link Aggregation

IEEE 802.3ad compliant including support of active LACP (Link Aggregation Control Protocol (switch to switch only) No support for aggregation of virtual NICs. Deploy up to 8 OSA adapters. OSA Adapters that are part of the aggregated group are not sharable with other hosts on z/VM or LPAR. Non-disruptive failover Communications will continue if a hardware link in the group experiences a non-recoverable failure. Improved bandwidth over link aggregate group Workload balanced across aggregated links

42

2010 IBM Corporation

Motivation for Link Aggregation

Why only nearly seamless?

43

2010 IBM Corporation

Exploring an Always Available Solution

Link Aggregation still had challenges. Physical switch redundancy was not addressed. A physical switch failure would result in a drastic decrease in band width as up to 7 OSA paths would have to fail over to 1.

Possible solution: Use Cisco VSS1440 (or Cross Stack EtherChannel on the Catalyst 3750) solution to spread a single Link Aggregation group across two physical switches Also available on Juniper Switches with Virtual Chassis

44

2010 IBM Corporation

Always Available
CISCO VSS1440 A Feature With 6500 Switches
6500

Guest1
9.60.86.23 02-00-00-00-00-01

Guest2
9.60.86.24 02-00-00-00-00-02

Guest3
9.60.86.25 02-00-00-00-00-03

OSA 4500 OSA 4600 10Gig Links 6500 OSA 6600 OSA 6700 OSA 7700 OSA 7A00 OSA A000 OSA B000

VSWITCH SWT1

Virtual Switching System 1440

Group SPG01

z/VM5.4

* CISCO Catalyst 3750 With Cross-Stack Ether-Channel Can Provide This Function.
45 2010 IBM Corporation

Always Available

Guest1
9.60.86.23 02-00-00-00-00-01

Guest2
9.60.86.24 02-00-00-00-00-02

Guest3
9.60.86.25 02-00-00-00-00-03

6500

X
OSA 4500 10Gig Links 6500

X X X X

X OSA 4600 X OSA 6600 X OSA X 6700

OSA 7700 OSA 7A00

VSWITCH SWT1

OSA A000 OSA B000

Virtual Switching System 1440

X Group
SPG01

z/VM5.4

X = Lost Function or Path Remains Active

46

2010 IBM Corporation

Cisco VSS 1440 solution

zVM Linux Linux Linux Linux Linux Linux Linux Linux VM Controller NIC NIC NIC NIC NIC NIC

Port 65

Port 66

Port 67

Port 68

Port 69

Port 70

Load Balancer Aggregator/Multiplexer

LACP

Port 1

Port 2

Port 3

Port 4

Virtual Switch

OSA

OSA

OSA

OSA

System z LPAR

Port 1

Port 2
Physical Switch A

Port 1

Port 2
Physical Switch B

Switch Error
2010 IBM Corporation

47

Questions?

48

2010 IBM Corporation

Useful diagnostic commands

CP QUERY VMLAN to get global VM LAN information (e.g. limits) to find out what service has been applied

CP QUERY LAN ACTIVE to find out which users are coupled to find out which IP addresses are active

CP QUERY NIC DETAILS to find out if your adapter is coupled to find out if your adapter is initialized to find out if your IP addresses have been registered to find out how many bytes/packets sent/received CP QUERY PORT GROUP To determine the members of a particular groupname To determine which groups are active or inactive
49 2010 IBM Corporation

References
Publications: z/VM CP Planning and Administration z/VM CP Command and Utility Reference z/VM TCP/IP Planning and Customization z/VM Connectivity Planning, Administration and Operation

Links: http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps9336/prod_ white_paper0900aecd806ee2ed.html http://www.ibm.com/servers/eserver/zseries/os/linux/ http://www.linuxvm.org/ http://www.vm.ibm.com/virtualnetwork/

50

2010 IBM Corporation

Contact Information
Speaker: Brian Hugenbruch E-mail: bwhugen at us.ibm.com Web: http://www.vm.ibm.com/devpages/hugenbru Phone: USA 607.429.3660 Web sites: http://www.vm.ibm.com/networking/ -- zVM Virtual Networking http://www.vm.ibm.com/related/tcpip/ -- zVM TCPIP http://ibm.com/vm/techinfo/listserv.html -- List of Mailing Lists Via mailing lists: [email protected] [email protected] [email protected] Special Thanks to: Tracy Adams, Alan Altmark

51

2010 IBM Corporation