Lan, San, Man, Wan: Making An Intelligent Choice For Your Storage
Lan, San, Man, Wan: Making An Intelligent Choice For Your Storage
Agenda
Storage architectures DAS: Direct Attached Storage NAS: Network Attached Storage SAN: Storage Area Network Network Architectures Ethernet FC Sonet DWDM Networked Storage Comparison Scaling Congestion Control
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.
Networked Storage
NAS (Network Attached Storage)
Storage accessible at the file system level through: NFS CIFS/SMB IP/Ethernet network Main application: Engineering
SCSI History
SCSI iSCSI Parallel SCSI Fibre Channel Fibre Channel TCP IP Ethernet
IP Ethernet
4
Storage issues
SCSI has a lot of baggage from the past
It assumes the old bus based architecture It is not efficient in recovering from packet loss Not an issue in bus architecture Drivers are still based on old SCSI standards and they have been retrofitted with the network
Storage Latency
Latency budget for SAN should be less than storage response time
3 possible technology + 1
Plus one
DWDM
An Historical perspective
Metcalfe in 1976 presents Ethernet to the National Computer Conference
1980 Digital, Intel and Xerox had released a de facto standard for a 10 Mbps in 1991 10Mbps on UTP In 1995 100Mbps In 1998-1999 1Gps In 2002 10Gb/s Ethernet
"The diagram ... was drawn by Dr. Robert M. Metcalfe in 1976 to present Ethernet ... to the National Computer Conference in June of that year.
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.
Ethernet: characteristics
10
100Mbps
CSMA/CD CSMA/CD Reconciliation Sublayer Reconciliation Sublayer
PCS 100BASE -T4 PCS 100BASE -T2 PCS 100BASE-T4 PCS 100BASE -T2
100BASE-T4 100BASE-T4
100BASE-T2 100BASE-T2
1 Gbps
CSMA/CD CSMA/CD Reconciliation Sublayer Reconciliation Sublayer
1000BASE-T 1000BASE-T 1000BASE-SX 1000BASE-SX 1000BASE-X 1000BASE-X 1000BASE-LX 1000BASE-LX 1000BASE-CX 1000BASE-CX
10GBASE- X 10GBASE-X (8B10B) (8B10B)
10 Gbps
MAC (opzionale) MAC (opzionale) Reconciliation Sublayer Reconciliation Sublayer
10GBASE-R (64B/66B) 10GBASE-R (64B/66B) WAN Interface Sublayer(WIS) WAN Interface Sublayer (WIS) 10GBASE-R 10GBASE-R
10GBASE-LX4 10GBASE-SR 10GBASE-LR 10GBASE-ER 10GBASE-S W 10GBASE-LW 10GBASE- EW 10GBASE-LX4 10GBASE-SR 10GBASE-LR 10GBASE-ER 10GBASE-SW 10GBASE- LW 10GBASE-EW 10GBASE-R 10GBASE-W
11
Ethernet and IP get married J Everything over IP implies everything over Ethernet
12
When
1988 1995 In 1995 Ethernet 100 Mb/s 1 Gb/s in HW without loosing frames Ad Hoc network NIH syndrome IETF was basic Internet January 1994, RFC 1577 Classical IP and ARP over ATM
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.
13
Hub
14
FC Port Types
ISL
Node
NL_Port FL_Port E_Port
Fabric Switch
E_Port
Node
NL_Port
FC Switch
Node
NL_Port
F_Port
N_Port
Node
F_Port
N_Port
Node
15
Domain ID
Area ID
Port ID (AL_PA)
Domain ID
Identifies the switch
Area ID
Identifies different loops connected to the same switch
16
Low volumes (nature of the beast) Resurrects few years ago on storage needs
Today it is the totality of the SAN market
Improved interoperability
FC-PI, FC-FS, FC-MI, FC-DA, FC-SW3, FC-GS4
Added
4 Gbps 10 Gbps
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.
17
Higher level protocols may see Sonet has a synchronous point-to-point link without loss
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.
18
PDH e SDH
PDH SDH
Europe
US
US: SONET
Europe: SDH
E1 2.048 Mbps
E3 34.368 Mbps
E4 139.26 Mbps
Gains some momentum Widespread adoption of fiber increases the momentum First to reach 10Gb/s Popular at OC-3 (155 Mbps) and OC-12 (622 Mbps)
20
GFP/G.7041
Generic Framing Procedure Frame-Mapped GFP:
Ethernet PPP
Transparent GFP:
Fibre Channel Ficon Escon Transparent Gb Ethernet
21
GFP architecture
22
DWDM/CWDM
Site #3 Site #2
DWDM Ring
Site #1
FC/FICON/ESCON GigE/10GigE/ SONET/SDH/ Legacy
Site #4
6500
7xxx
4xxx
Data Data
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.
Storage Storage
23
DWDM Principles
Routers ATM
OEO
Optical Amplifier
DWDM Mux (Filter)
OEO
OA
OA
Storage
OEO
MM/SM 850nm/1310nm
24
850/1310nm FC/2GFC/GE
25
Storage Interconnections
1994 OC3
TDM
SONET/SDH
1996
OC12
TDM
OC48 1999
TDM
50%
~ ~ ~
OC192
50%
2001
Cisco Proprietary and Confidential- Do not distribute
~ ~ ~
100% ?
DWDM
26
Packet Loss
In IP/Ethernet
Its part of the game! Used by TCP/IP to handle congestions
In SCSI/Fibre Channel
Will throw you out of the market! There is no congestion control!
27
ACBCA Drop
C
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.
28
29
Fabric
TX Buf
Recipient
RX Buf
R_RDY
ACK
R_RDY
TX Buf
ACK
RX Buf
R_RDY
BB_C EE_Credit
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.
BB_C
30
FCIP Fibre-Channel-over-IP used to transport Fibre Channel frames within TCP/IP connections
IP TCP FCIP FC SCSI Data
31
iSCSI
A SCSI transport protocol that operates on top of TCP
Encapsulates SCSI CDBs and data into TCP/IP byte-streams Allows IP hosts to access IP based SCSI targets
IP
TCP
iSCSI
SCSI
Data
32
FCIP
FCIP is a mechanism that allows SAN islands to be interconnected over IP networks. Each interconnection is called a FCIP Link and can contain one (1) or more TCP connection(s). Each end of a FCIP Link is associated to a Virtual E_Port (VE_Port). VE_Ports communicate between themselves just like normally interconnected E_Ports by using SW_ILS: ELP, ESC, BF, RCF, FSPF, etc. The result is a fully merged Fibre Channel fabric.
IP
Cisco Proprietary and Confidential- Do not distribute
TCP
FCIP FC
SCSI Data
33
FCIP
FC Fabric
FCIP
Backup Servers
FC Fabric
FCIP
FCIP
Corporate HQ
Cisco Proprietary and Confidential- Do not distribute
Remote Sites
2002, Cisco Systems, Inc. All rights reserved.
FC Fabric
34
FCIP
Metro Ethernet
Gateway
FCIP
SAN
FCIP
SONET
FCIP
SAN
Low speed (T1 DS3) Higher latency Longer distance Mainly asynchronous
Remote Datacenter
FCIP
SAN
IP Routed WAN
Gateway
FCIP
SAN
35
TOEs
It is difficult to implement TCP in HW
At 10Gb/s TCP is tough !!!
The few TOEs that work are aliens in the OS Overall performance is required
True Zero Copy
RDMA
Significant OS and application changes Never took off
36
Comparing IP with FC
FC is limited
Size Congestion
37
Size limitation
Asynchronous operation
Journal in the network
38
39
40
Congestion
TCP reacts to congestion differently from FC
It scales to the Internet Van Jacobson taught us about windows and RED Congestion is signaled by packet loss TCP slows down in the presence of congestion
41
Neither of them guarantees low latency and high throughput in the presence of congestion Should we rethink the solution and add traffic engineering concepts?
The telephone network has used it with success The IETF has had some success with MPLS
42
43