Technology Overview: Why Fibre Channel Over Ethernet?: Storage Area Network
Technology Overview: Why Fibre Channel Over Ethernet?: Storage Area Network
Technology Overview:
Why Fibre Channel over Ethernet?
Fibre Channel over Ethernet (FCoE) is not a replacement for
conventional Fibre Channel (FC), but is an extension of Fibre
Channel over a different link layer transport.
STORAGE AREA NETWORK Technical Overview
CONTENTS
Introduction........................................................................................................................................................................................................................3
The Success of Fibre Channel......................................................................................................................................................................................3
FCoE Standards Initiative...............................................................................................................................................................................................4
Maintaining the Channel...................................................................................................................................... 5
Avoiding Packet Loss............................................................................................................................................ 6
Redundant Pathing and Failover ......................................................................................................................... 8
Mapping Fibre Channel to Ethernet .................................................................................................................... 7
FCoE, iSCSI, and FCIP......................................................................................................................................................................................................9
Summary.........................................................................................................................................................................................................................10
INTRODUCTION
Over the past ten years, Fibre Channel (FC) has become the technology of choice for Storage Area
Networks (SANs) worldwide. In the process, it has generated a wide range of new storage solutions,
including higher-performance block transport, high-availability storage access, streamlined data center
operations for backup and data protection, and higher-level storage services based on virtualization
and advanced management utilities. Other technical innovations, such as InfiniBand, Network-
Attached Storage (NAS), and iSCSI, however, periodically spark temporary debate about Fibre
Channel’s future and its viability as the next generation SAN transport. As of mid-2008, the Fibre
Channel installed base is estimated to be well over 10 million ports.
Without compelling economic or functional advantages, new technologies rarely displace successful
incumbents. Due to its economies of scale, for example, Ethernet displaced Token Ring for Local Area
Network (LAN) transport despite Token Ring’s higher performance (16 Mbit/sec versus 10 Mbit/sec at
the time) and more robust operation. Asynchronous Transfer Mode (ATM), on the other hand, was
unable to displace Ethernet to the desktop primarily due to its inability to co-opt Ethernet’s large
installed base. ATM’s LAN Emulation (LANE) was simply too problematic. And although InfiniBand has
demonstrated its end-user value for high-performance server clustering, it has been unable to
compete as a transport for either local or storage area networking. Because only a few smaller
vendors have introduced InfiniBand storage arrays, Fibre Channel continues to be the connectivity of
choice for data center storage applications. In addition, InfiniBand’s inability to demonstrate value as
a LAN transport and its unique cabling scheme and limited distance support at high speeds have
discouraged IT administrators from fork-lifting their existing networks.
Fibre Channel has also introduced a set of higher-level services for scaling reliable and highly available
fabrics. Fabric routing protocols, policy-based routing, hardware-based trunking, Virtual Fabrics, fabric
security, and fault isolation have been built on top of a foundation of stable transport. New fabric-
based application services for storage virtualization and data protection are further enhancing
simplification and automation of storage administration. Collectively, Fibre Channel standards and
standards-compliant products are optimized to deliver maximum performance and maximum
availability of storage data. As a consequence, Fibre Channel SANs are now powering every significant
enterprise and institution worldwide.
FCoE enables a simplified connection on the server (initiator) side. Storage systems (targets), however, will for
the foreseeable future remain native Fibre Channel. The storage network, therefore, can incorporate different
link layer transports where appropriate, with Brocade® FCoE-enabled switches and directors providing the
conversion between FCoE and Fibre Channel fabrics. Because all storage transactions from servers to storage
are based on the Fibre Channel protocol, the inherent performance, stability and high availability features of
Fibre Channel are maintained on both CEE and native FC links. It is therefore more accurate to describe this
new architecture not as network unification but as network convergence, or the point at which separate SAN
and LAN protocols converge on a common transport for simplified server connectivity. The new Data Center
Fabric (DCF) will consist of traditional FC and FCoE supported on Brocade directors and the Brocade DCX
Backbone platform, while the conventional outward-facing network for messaging and data communications
will continue to be supported on LAN and WAN technologies.
Over the years, Fibre Channel has evolved higher-level functions tailored to storage requirements:
• The Simple Name Server (SNS) hosted by every fabric switch, for example, provides device
discovery for initiators seeking target resources.
• Zoning (based on port World Wide Name or Domain, Port identification) enables segregation of
storage relationships and prevents unauthorized servers from communicating with designated
storage assets.
• Registered State Change Notifications (RSCNs) provide a means to alert servers to the arrival or
departure of storage systems on the fabric.
• The Fabric Shortest Path First (FSPF) protocol establishes optimum paths in a multi-switch fabric
and allows multiple trunked links to increase bandwidth between switches.
• Fabric routing with fault isolation provides sharing of resources between autonomous SANs.
To preserve the channel attributes and storage-centric services of conventional Fibre Channel, FCoE
requires significant enhancements to conventional Ethernet networking and integrated controllers to
provide device discovery, notifications, security, and other advanced storage services. Assuming that
Ethernet can be hardened for data center use, FCoE would be a fairly straightforward means to wrap
FC frames in Ethernet for frame translation between FCoE initiators and FC targets. To be viable for
customer implementation, however, the rich set of Fibre Channel advanced fabric services must be
preserved.
FCoE Switches
FCoE switches or Fibre Channel Forwarders (FCFs) provide the connectivity between FCoE initiators
and conventional Fibre Channel fabrics. FCFs therefore offer both CEE ports and native FC ports for
both device and switch-to-switch fabric connections. For CEE connectivity, FCoE device ports are
VF_Ports (corresponding to Fibre Channel F_Ports), while switch-to-switch ports are VE_Ports
(corresponding to conventional Fibre Channel E_Ports). Brocade FCF switches can be embedded as
blade server FCoE switch modules, installed as rack-mount FCFs, or as FCoE blades inserted into the
Brocade DCX Backbone. Customers will thus have the flexibility to deploy both native Fibre Channel
and FCoE that aligns with their server strategy and platform mix over time.
The conventional use of IEEE 802.3x PAUSE operates at the port level. This could prove problematic
for FCoE, however, if other protocols on the same CEE port are the actual source of congestion.
Consequently, successful implementation of PAUSE frames for storage environments must be more
granular and operate on a per-application virtual channel basis. Because Ethernet does provide the
ability to set priority bits in the Ethernet header, it is possible to combine 802.1Q prioritization with the
802.3x PAUSE mechanism. The priority-based flow control (PFC) CEE initiative is being worked in the
802.1Qbb workgroup and will provide the means to fine tune congestion control to the application
layer.
In addition, 802.1Qbb can leverage prioritization to establish bandwidth allocation on a per-
application basis. Time-sensitive applications such as inter-process communications (IPC) can be
given a higher percentage of available bandwidth as needed while other applications are assured
portions of the remaining available bandwidth. The enhanced transmission selection (ETS) algorithm
will strengthen the ability of FCoE to reliably use Ethernet as a transport layer and minimize the
chance of link congestion and frame loss.
Ethernet Congestion Management (ECM) uses another technique for implementing reliable flow
control on an end-to-end basis. The IEEE 802.1Qau Congestion Notification Group is developing ways
to mitigate congestion by reflecting frames back to their source when a congestion point occurs. When
a host sees its own frames being reflected by a downstream switch, it will slow its own frame
transmission until no more reflected frames are seen. Analogous to Backward Explicit Congestion
Notification (BECN) in WAN technologies, ECM can slow the pace of issued frames until a congestion
point is cleared.
FCoE frames are native Layer 2 Ethernet frames with conventional six-byte destination and source
MAC (media access control) addresses. The MAC addresses, however, are storage agnostic and are
used only to switch frames from source to destination. The FCoE content of the frame retains Fibre
Channel addressing required for storage transactions, and so some means is required to map Fibre
Channel IDs (FCIDs) to Ethernet MAC addresses. The FCoE Initialization Protocol (FIP) allows for either
fabric-provided MAC addresses or server-provided MAC addresses and FCoE switches (known as Fibre
Channel Forwarders or FCF) maintain the mapping between Ethernet MAC addresses and the
corresponding Fibre Channel addresses.
from the interface to application memory. Dedicated 10 Gbit/sec iSCSI adapters add significant
cost per server attachment compared to 8 Gbit/sec FC HBAs, however, and undermine the value
proposition of iSCSI at 1 Gbit/sec.
Although iSCSI-to-FC gateways enable iSCSI initiators to access FC storage targets, the requisite
protocol conversion is considerably more complex than FCoE streamlined frame mapping to Fibre
Channel. For iSCSI gateways, a complete address translation is required between iSCSI and FC
address conventions. In addition, the gateway must proxy virtual FC initiators and virtual iSCSI targets
and terminate sessions within the gateway between the two protocols. If the objective is to have
Ethernet-attached servers access FC SAN targets, FCoE will require less protocol overhead and
processing latency to span between Ethernet and Fibre Channel transports.
SUMMARY
The large installed base and maturity of Fibre Channel technology has generated a wide spectrum of
storage-specific features and management tools to facilitate robust deployment of shared storage in
the data center. The Converged Enhanced Ethernet (CEE) initiative will enable customers to combine
storage, messaging traffic, VoIP, video, and other data on a common data center Ethernet
infrastructure. FCoE is the component technology that enables highly efficient block storage over
Ethernet for consolidating server network connectivity. By enabling customers to deploy a single server
interface for multiple data types, FCoE and CEE will simplify both deployment and management of
server network connectivity, while maintaining the high availability and robustness required for storage
transactions. FCoE is thus not a replacement for, but an extension of, Fibre Channel and is intended to
coexist with existing FC SANs.
Because FCoE takes advantage of further enhancements to CEE, its development will require close
coordination of both Fibre Channel and Ethernet technologists and standards bodies. Although link
layer issues, such as flow control and the limitations of Ethernet spanning tree protocols, present
significant challenges, the more arduous task will be to create FCoE solutions that preserve the rich
set of Fibre Channel advanced services that customers are deploying productively today. Even at
10 Gbit/sec, today’s Ethernet technology will require substantial work to be suitable for data center
storage applications. As a pioneer of Fibre Channel fabric technology, Brocade is bringing its expertise
to the FCoE initiative to streamline the server network interface while preserving data center
performance, reliability, and the proven benefits of advanced storage services.
© 2008 Brocade Communications Systems, Inc. All Rights Reserved. 06/08 GA-TB-044-01
Brocade, the Brocade B-weave logo, Fabric OS, File Lifecycle Manager, MyView, SilkWorm, and StorageX are registered
trademarks and the Brocade B-wing symbol, SAN Health, and Tapestry are trademarks of Brocade Communications Systems,
Inc., in the United States and/or in other countries. FICON is a registered trademark of IBM Corporation in the U.S. and other
countries. All other brands, products, or service names are or may be trademarks or service marks of, and are used to identify,
products or services of their respective owners.
Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning
any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes
to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes
features that may not be currently available. Contact a Brocade sales office for information on feature and product availability.
Export of technical data contained in this document may require an export license from the United States government.