0% found this document useful (0 votes)
80 views43 pages

Lan, San, Man, Wan: Making An Intelligent Choice For Your Storage

LAN, SAN, MAN, WAN: Making an Intelligent Choice for your Storage Silvano Gai Cisco / Andiamo Fellow. NAS (Network Attached Storage) storage accessible at the file system level through: NFS CIFS / SMB IP / Ethernet network Main application: Engineering SAN (Storage Area Network) storage accessible at the block level through: SCSI Fibre Channel Fibre Channel TCP IP Ethernet FCIP TCP SAN Cisco Proprietary and Conf

Uploaded by

Vicky Jain
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
80 views43 pages

Lan, San, Man, Wan: Making An Intelligent Choice For Your Storage

LAN, SAN, MAN, WAN: Making an Intelligent Choice for your Storage Silvano Gai Cisco / Andiamo Fellow. NAS (Network Attached Storage) storage accessible at the file system level through: NFS CIFS / SMB IP / Ethernet network Main application: Engineering SAN (Storage Area Network) storage accessible at the block level through: SCSI Fibre Channel Fibre Channel TCP IP Ethernet FCIP TCP SAN Cisco Proprietary and Conf

Uploaded by

Vicky Jain
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 43

LAN, SAN, MAN, WAN: Making an Intelligent Choice for your Storage

Silvano Gai Cisco/Andiamo Fellow

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

Agenda
Storage architectures DAS: Direct Attached Storage NAS: Network Attached Storage SAN: Storage Area Network Network Architectures Ethernet FC Sonet DWDM Networked Storage Comparison Scaling Congestion Control
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.

Networked Storage
NAS (Network Attached Storage)
Storage accessible at the file system level through: NFS CIFS/SMB IP/Ethernet network Main application: Engineering

SAN (Storage Area Network)


Storage accessible at the block level through SCSI Fibre Channel or IP/Ethernet networks Main Application: Database The topic of this tutorial
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.

SCSI History
SCSI iSCSI Parallel SCSI Fibre Channel Fibre Channel TCP IP Ethernet

FCIP TCP SAN


Cisco Proprietary and Confidential- Do not distribute

IP Ethernet
4

2002, Cisco Systems, Inc. All rights reserved.

Storage issues
SCSI has a lot of baggage from the past
It assumes the old bus based architecture It is not efficient in recovering from packet loss Not an issue in bus architecture Drivers are still based on old SCSI standards and they have been retrofitted with the network

Applications are designed to cope with the above


Pipeline is hardly used

Applications need to commit to stable storage


When you send status(OK) you own the data and you cannot lose it
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.

Storage Latency

Storage response time is:


Few milliseconds for disks Sub-millisecond for caches

Latency budget for SAN should be less than storage response time

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

3 possible technology + 1

There are 3 possible technology for SAN:


Ethernet FC Sonet

Plus one
DWDM

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

An Historical perspective
Metcalfe in 1976 presents Ethernet to the National Computer Conference
1980 Digital, Intel and Xerox had released a de facto standard for a 10 Mbps in 1991 10Mbps on UTP In 1995 100Mbps In 1998-1999 1Gps In 2002 10Gb/s Ethernet

Fibre Channel initial development in 1988


In 1994, the first Fibre Channel standard was approved (FC-PH) In 1995 1 Gb/s based products are deployed In 2003 10Gb/s Fibre Channel

Sonet is developed in 1985 by Bellcore


In 1988 first ITU standard (G.707) In 2000 10 Gb/s OC-192
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.

Ethernet: the origin

"The diagram ... was drawn by Dr. Robert M. Metcalfe in 1976 to present Ethernet ... to the National Computer Conference in June of that year.
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.

Ethernet: characteristics

Simple Single MAC design Broad range of speeds


From 10 Mbps to 10 Gbps

High volumes/Low costs Only survivor No guaranteed delivery


+/- of loosing frames

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

10

Ethernet: the standards


10Mbps
IEEE 802.2/ISO 8802.2 Logical Link Control IEEE 802.2/ISO 8802.2 Logical Link Control CSMA/CD CSMA/CD Physical Layer Signaling (PLS) Physical Layer Signaling (PLS) 10BASE-T 10BASE-T 10BASE2 10BASE2 100BASE5 100BASE5 10BASE-F 10BASE-F

100Mbps
CSMA/CD CSMA/CD Reconciliation Sublayer Reconciliation Sublayer
PCS 100BASE -T4 PCS 100BASE -T2 PCS 100BASE-T4 PCS 100BASE -T2

PCS 100BASE-X PCS 100BASE-X 100BASE-TX 100BASE-TX 100BASE-FX 100BASE-FX

100BASE-T4 100BASE-T4

100BASE-T2 100BASE-T2

1 Gbps
CSMA/CD CSMA/CD Reconciliation Sublayer Reconciliation Sublayer
1000BASE-T 1000BASE-T 1000BASE-SX 1000BASE-SX 1000BASE-X 1000BASE-X 1000BASE-LX 1000BASE-LX 1000BASE-CX 1000BASE-CX
10GBASE- X 10GBASE-X (8B10B) (8B10B)

10 Gbps
MAC (opzionale) MAC (opzionale) Reconciliation Sublayer Reconciliation Sublayer
10GBASE-R (64B/66B) 10GBASE-R (64B/66B) WAN Interface Sublayer(WIS) WAN Interface Sublayer (WIS) 10GBASE-R 10GBASE-R

10GBASE-LX4 10GBASE-SR 10GBASE-LR 10GBASE-ER 10GBASE-S W 10GBASE-LW 10GBASE- EW 10GBASE-LX4 10GBASE-SR 10GBASE-LR 10GBASE-ER 10GBASE-SW 10GBASE- LW 10GBASE-EW 10GBASE-R 10GBASE-W

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

11

Ethernet: the evolution

Ethernet kills all other LANs


Token Ring, FDDI, etc. (except 802.11)

IP kills all other network architectures


IPX, NetBeui, Decnet, AppleTalk

Ethernet and IP get married J Everything over IP implies everything over Ethernet

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

12

Fibre Channel: the origin


Why
SCSI needed to get out of the parallel bus

When
1988 1995 In 1995 Ethernet 100 Mb/s 1 Gb/s in HW without loosing frames Ad Hoc network NIH syndrome IETF was basic Internet January 1994, RFC 1577 Classical IP and ARP over ATM
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.

13

Fibre Channel Topologies


Point-to-Point Switched Fabric Arbitrated Loop

Hub

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

14

FC Port Types

ISL
Node
NL_Port FL_Port E_Port

Fabric Switch
E_Port

Node

NL_Port

FC Switch

Node

NL_Port

F_Port

N_Port

Node

F_Port

N_Port

Node

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

15

Fibre Channel ID Format


8 bits 8 bits 8 bits

Domain ID

Area ID

Port ID (AL_PA)

Domain ID
Identifies the switch

Area ID
Identifies different loops connected to the same switch

Port ID (or AL_PA)


Identifies the port on the switch (for N_Ports) or the specific node on the loop (for NL_Ports)
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.

16

FC: the evolution


Stagnated on
Poor protocol design Poor Interoperability

Low volumes (nature of the beast) Resurrects few years ago on storage needs
Today it is the totality of the SAN market

Improved interoperability
FC-PI, FC-FS, FC-MI, FC-DA, FC-SW3, FC-GS4

Added
4 Gbps 10 Gbps
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.

17

Sonet/SDH: the Origin


Telco flavor
Isochronous traffic High Availability/Resiliancy Distance NEBS compliant

Higher level protocols may see Sonet has a synchronous point-to-point link without loss
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.

18

PDH e SDH
PDH SDH

Europe

US

US: SONET

Europe: SDH

E1 2.048 Mbps

T1 - DS1 1.544 Mbps

OC-1c / STS-1c 51.84 Mbps

E3 34.368 Mbps

T3 - DS3 44.736 Mbps

OC-3c / STS-3c 155.52 Mbps

STM-1 155.52 Mbps

E4 139.26 Mbps

OC-12c / STS-12c 622.08 Mbps

STM-4 622.08 Mbps

PDH: Plesiochronous Digital Hierarchy SDH: Synchronous Digital Hierarchy


Cisco Proprietary and Confidential- Do not distribute

OC-48c / STS-48c 2.4 Gbps

STM-12 2.4 Gbps


19

2002, Cisco Systems, Inc. All rights reserved.

Sonet: the evolution

Grows on Telco money


Enabling Technology for ATM backbone, later dead Now used for IP over WAN

Gains some momentum Widespread adoption of fiber increases the momentum First to reach 10Gb/s Popular at OC-3 (155 Mbps) and OC-12 (622 Mbps)

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

20

GFP/G.7041
Generic Framing Procedure Frame-Mapped GFP:
Ethernet PPP

Transparent GFP:
Fibre Channel Ficon Escon Transparent Gb Ethernet

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

21

GFP architecture

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

22

DWDM/CWDM
Site #3 Site #2
DWDM Ring

Site #1
FC/FICON/ESCON GigE/10GigE/ SONET/SDH/ Legacy

Site #4

6500

7xxx

4xxx

Data Data
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.

Storage Storage
23

DWDM Principles

Routers ATM

OEO

Optical Amplifier
DWDM Mux (Filter)

OEO

OA

OA

Storage

OEO

Wavelength Multiplexed Signals

Optically Amplified Wavelengths


OA

MM/SM 850nm/1310nm

ITU-T grid 15xxnm Transponders

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

24

FC over Optical Todays Solutions


FC over DWDM DWDM 10GE/2.5G
850/1310nm FC/2GFC/GE

FC over SONET SONET

850/1310nm FC/2GFC/GE

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

25

Storage Interconnections
1994 OC3
TDM

SONET/SDH

1996

OC12

TDM

SONET/SDH SONET/SDH DWDM

OC48 1999

TDM

50%

~ ~ ~
OC192

50%

2001
Cisco Proprietary and Confidential- Do not distribute

~ ~ ~

100% ?

DWDM
26

2002, Cisco Systems, Inc. All rights reserved.

Packet Loss

In IP/Ethernet
Its part of the game! Used by TCP/IP to handle congestions

In SCSI/Fibre Channel
Will throw you out of the market! There is no congestion control!

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

27

Why frames get dropped


Not for transmission error Nor for collision Queue overflow due to congestion

ACBCA Drop

C
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.

28

To Drop or NOT TO Drop?


Queue in front of the link Dropping or crediting? No drop FC native over Sonet or DWDM No TCP Credits Drop TCP needs to recover Transport SCSI over TCP/IP

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

29

Flow Control and Credits


Initiator
RX Buf TX Buf R_RDY RX Buf TX Buf

Fabric
TX Buf

Recipient
RX Buf

R_RDY

ACK
R_RDY

TX Buf

ACK

RX Buf

R_RDY

BB_C EE_Credit
Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.

BB_C

30

SCSI over TCP


SCSI over TCP provides solution to carry storage traffic over Intranet/Internet Uses TCP, a reliable transport for delivery Can be used for local data center and long haul applications Two primary protocols:
iSCSI IP-SCSI - used to transport SCSI CDBs and data within TCP/IP connections
IP TCP iSCSI SCSI Data

FCIP Fibre-Channel-over-IP used to transport Fibre Channel frames within TCP/IP connections
IP TCP FCIP FC SCSI Data
31

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

iSCSI
A SCSI transport protocol that operates on top of TCP
Encapsulates SCSI CDBs and data into TCP/IP byte-streams Allows IP hosts to access IP based SCSI targets

Standards status update


Draft RFC gone to ratification (passed workgroup last call) Current draft : draft-ietf-ips-iSCSI-20 Expected to be ratified as a standard in Q2-03

IP

TCP

iSCSI

SCSI

Data
32

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

FCIP
FCIP is a mechanism that allows SAN islands to be interconnected over IP networks. Each interconnection is called a FCIP Link and can contain one (1) or more TCP connection(s). Each end of a FCIP Link is associated to a Virtual E_Port (VE_Port). VE_Ports communicate between themselves just like normally interconnected E_Ports by using SW_ILS: ELP, ESC, BF, RCF, FSPF, etc. The result is a fully merged Fibre Channel fabric.
IP
Cisco Proprietary and Confidential- Do not distribute

TCP

FCIP FC

SCSI Data
33

2002, Cisco Systems, Inc. All rights reserved.

Example FCIP Environment


Joining SAN islands over the WAN Transparent bridging of FC over TCP/IP
IP Network

FCIP
FC Fabric

FCIP
Backup Servers

FC Fabric

FCIP

FCIP

Corporate HQ
Cisco Proprietary and Confidential- Do not distribute

Remote Sites
2002, Cisco Systems, Inc. All rights reserved.

FC Fabric

34

Potential FCIP Environments


Near wire-rate (1Gbps) Relatively low latency Mainly asynchronous Suitable for some synchronous apps Typical OC3 / OC12 Relatively low latency Mainly asynchronous Suitable for some synchronous apps
SAN Local Datacenter Gateway Gateway SAN Local Datacenter Gateway Remote Datacenter

FCIP

Metro Ethernet

Gateway

FCIP
SAN

Short distance ~ <= 60km Remote Datacenter

FCIP

SONET

FCIP
SAN

Medium distance ~ <= 160km

Low speed (T1 DS3) Higher latency Longer distance Mainly asynchronous

Local Datacenter Gateway

Remote Datacenter

FCIP
SAN

IP Routed WAN

Gateway

FCIP
SAN

Long distance > 160km


Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.

35

TOEs
It is difficult to implement TCP in HW
At 10Gb/s TCP is tough !!!

The few TOEs that work are aliens in the OS Overall performance is required
True Zero Copy

RDMA
Significant OS and application changes Never took off

At same performance/same efficiency, same cost of HBAs


Cisco Proprietary and Confidential- Do not distribute
2002, Cisco Systems, Inc. All rights reserved.

36

Comparing IP with FC

FC is limited
Size Congestion

while IP is not, or is it?

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

37

Size limitation

239 switches in 100 KM radius


256 available Domain IDs minus 17 reserved Is it a limitation?

Above 100 KM, light is slow


100 KM = 200KM round trip, 1 ms

Asynchronous operation
Journal in the network

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

38

B2B Congestion Control

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

39

TCP Congestion Control

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

40

Congestion
TCP reacts to congestion differently from FC
It scales to the Internet Van Jacobson taught us about windows and RED Congestion is signaled by packet loss TCP slows down in the presence of congestion

but slow down implies


Reduced throughput Increased latency

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

41

Ethernet/IP vs Fibre Channel

Neither of them guarantees low latency and high throughput in the presence of congestion Should we rethink the solution and add traffic engineering concepts?
The telephone network has used it with success The IETF has had some success with MPLS

Traffic engineering for Storage?

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

42

Cisco Proprietary and Confidential- Do not distribute

2002, Cisco Systems, Inc. All rights reserved.

43

You might also like