PLNOG16: Coping with Growing Demands – Developing the Network to New Bandwidths, Arne Heitman
0 likes156 views
This document discusses the growing demands on data center networks and the transition to higher bandwidth technologies like 25GbE, 50GbE and 100GbE. It covers drivers of higher bandwidth demand like increased virtual machine density and higher resolution video formats. It then discusses technology solutions for higher bandwidth including new silicon, port architectures, transceiver standards and cable types. Examples of network architectures using these technologies are presented.
PLNOG16: Coping with Growing Demands – Developing the Network to New Bandwidths, Arne Heitman
- 1. Arne Heitmann | Sr. System Engineer | EMEA PLNOG | Warsaw | February 29th 2016 Coping with Growing Demands Developing the Network to New Bandwidths
- 2. © 2015 Mellanox Technologies 2 Entering The Era of 25GbE, 50GbE And 100GbE Copper (Passive, Active) Optical Cables (VCSEL) Silicon Photonics 100GbE Adapter (10 / 25 / 40 / 50 / 56 / 100GbE) Multi Host Solution 32 100GbE Ports, 64 25/50GbE Ports (10 / 25 / 40 / 50 / 56 / 100GbE) Throughput of 6.4Tb/s
- 3. © 2015 Mellanox Technologies 3 Demand - More Virtual Machines Per Server Interconnect Bandwidth Determines VM Density 10GbE adapter card Mellanox 40GbE adapter card 20 VMs + Vs. + 60 VMs
- 4. © 2015 Mellanox Technologies 4 Demand - The World of Bandwidth is changing International bandwidth growth (projected 2012-2019) Global IP traffic by type in petabytes/month Source: Ars Technica, 2012Source: TeleGeography/ITU
- 5. © 2015 Mellanox Technologies 5 Description Hres Vres Colour depth (bits) Pixels RGB FPS RAW BW (MB/sec) RAW BW (Gbits/sec) 8Gb FC lanes 16Gb FC lanes No. 10Gb lanes No. 40Gb lanes No. 56Gb lanes No. 100Gb lanes Storage GB/sec Strg 90 min Movie (TB) HD Video - Low FPS 1920 1080 16 3 30 373.25 2.99 1 1 1 1 1 1 0.37 2.02 HD Video (US) 1920 1080 16 3 50 622.08 4.98 1 1 1 1 1 1 0.62 3.36 HD Video (EMEA) 1920 1080 16 3 60 746.50 5.97 1 1 1 1 1 1 0.75 4.03 2K Video (US) 2048 1080 16 3 50 663.55 5.31 1 1 1 1 1 1 0.66 3.58 2K Video (EMEA) 2048 1080 16 3 60 796.26 6.37 2 1 1 1 1 1 0.80 4.30 4K UHD (Std FPS) 3840 2160 16 3 30 1492.99 11.94 2 1 2 1 1 1 1.49 8.06 4K UHD (3D FPS) 3840 2160 16 3 60 2985.98 23.89 4 2 4 1 1 1 2.99 16.12 4K Cinema (Std FPS) 4096 2160 16 3 30 1592.52 12.74 3 2 2 1 1 1 1.59 8.60 4K-Full Cinema (Std FPS) 4096 3112 16 3 30 2294.42 18.36 4 2 3 1 1 1 2.29 12.39 4K Cinema (3D FPS) 4096 2160 16 3 60 3185.05 25.48 5 3 4 1 1 1 3.19 17.20 5K Cinema (Std FPS) 5120 2700 16 3 30 2488.32 19.91 4 2 3 1 1 1 2.49 13.44 5K Cinema (3D FPS) 5120 2700 16 3 60 4976.64 39.81 7 4 6 2 1 1 4.98 26.87 8K UHD (Std FPS) 7680 4320 16 3 30 5971.97 47.78 8 4 7 2 1 1 5.97 32.25 8K UHD (3D FPS) 7680 4320 16 3 60 11943.94 95.55 16 8 14 3 2 2 11.94 64.50 Super Hi-Vision 7680 4320 16 3 120 23887.87 191.10 32 16 28 6 4 3 23.89 128.99 Demand: Media/Entertainment – Acceleration already happening Data rates and storage are exploding, due to high pixel counts and frame rates 10GbE Ethernet is not going to provide the necessary BW going forward
- 6. © 2015 Mellanox Technologies 6 Moving to 25GbE, 50GbE And 100GbE Compute Nodes 150% Higher Bandwidth Storage Nodes 25% Higher Bandwidth Network 150% Higher Bandwidth 100GbE 25GbE 50GbE Same Connectors Similar Infrastructure Better Cost / Power Compute Nodes Storage Nodes 40GbE Network 40GbE 10GbE
- 7. © 2015 Mellanox Technologies 7 Silicon - SerDes Silicon is connected to board SerDes • Serializer / Deserializer • May work at • ~10Gb/s • ~14Gb/s • ~25Gb/s • Addresses a certain number of ports Can be bundled: • i.e. 4x10G for a 40Gb/s link • i.e. 4x14G for a 56Gb/s link • i.e. 4x25G for a 100G link
- 8. © 2015 Mellanox Technologies 8 Silicon – Port Architecture Example 100 GigE 50 GigE 50 GigE 40 GigE 40 GigE 10 GigE 10 GigE 40 GigE OR OR OR OR 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 20Gig 20Gig 20Gig 20Gig 10Gig 10Gig 10Gig 10Gig 10Gig 10Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig 25Gig n x 100Gig Port Basic UnitPort Options ASIC
- 9. © 2015 Mellanox Technologies 9 Silicon - Many Port Configuration Options for n*25GbE SFP28 SFP+ 25GbE (1x25Gb/s) 10GbE (1x10Gb/s) SFP+10GbE (1x10Gb/s) SFP+10GbE (1x10Gb/s) SFP+10GbE (1x10Gb/s) SFP2825GbE (1x25Gb/s) SFP2825GbE (1x25Gb/s) SFP2825GbE (1x25Gb/s) QSFP40GbE (4x10Gb/s) QSFP28 50GbE (2x25Gb/s) QSFP28 50GbE (2x25Gb/s) QSFP28100GbE (4x25Gb/s) 135 watts
- 10. © 2015 Mellanox Technologies 10 Transceiver – some Numbers What comes next? 50G over a single Lane • IEEE 802.3 50 Gb/s Ethernet Over a Single Lane and Next Generation 100 Gb/s and 200 Gb/s Ethernet Study Group IEEE P802.3by 25 Gb/s Ethernet Task Force • Standard to come IEEE P802.3bs 400 GbE Task Force • Adopted timeline says Standard for 2018??
- 11. © 2015 Mellanox Technologies 11 Transcerver - The Evolution between 10Gb, 25Gb, 40Gb, 50Gb & 100Gb IEEE 802.3bm
- 12. © 2015 Mellanox Technologies 12 Transceiver - Pluggable Module Standards CFP The CFP MSA defines hot-pluggable optical transceiver form factors to enable 40 Gbit/s and 100 Gbit/s applications. CFP modules use the 10-lane CAUI-10 electrical interface. CFP2 CFP2 modules use the 10-lane CAUI-10 electrical interface or the 4-lane CAUI-4 electrical interface. CFP4 CFP4 modules use the 4-lane CAUI-4 electrical interface. QSFP28 QSFP28 modules use the 4-lane CAUI-4 electrical interface. CPAK Cisco has the CPAK optical module that uses the four lane CEI-28G-VSR electrical interface. CXP There are also CXP and HD module standards. CXP modules use the CAUI-10 electrical interface.
- 13. © 2015 Mellanox Technologies 13 Cables - Optical Connector types for Parallel and single fiber infrastructures MPO Optical Connectors 4 Transmit 4 Receive 12-fiber Optical Connector (4-unused fibers in middle) Duplex LC 2-fiber Optical Connector Typically for Single-Mode Can be used for Multi-mode (SR4) or Single-mode (PSM4) Also called MTP or MPO/MTP Single-mode (LR4)
- 14. © 2015 Mellanox Technologies 14 Cables - Solutions for data center applications 14 Data Center Fabrics Link Length (m) 10 100 500150 300 1000 2000 10 25 50 3 51 20 DataRateperLane(Gbs) 10000500020 30 50 752 Single mode fiber OM4OM3 Copper Multi-mode fiber Silicon Photonics Direct Attach Copper • Zero power • Demo’s 8m at 100G • Best fit 3m VCSELsDACs Active Optical Cables • VCELs or SiP • Reaches to 200m • Best fit for 5-20m VCSEL Transceivers • Reaches to 100m • Best fit for MMF SiP Transceivers • Reaches to 2km • Best fit for SMF • Parallel or WDM
- 15. © 2015 Mellanox Technologies 15 Cables/Transceivers - 100GbE Products 100G SR4 Ethernet-Only Transceiver 100G Copper DAC InfiniBand & Ethernet 100G AOC InfiniBand & Ethernet For lowest-cost optical 100G switch-to-switch links . For Breakouts to 25G / 50G servers + storage with breakout fibers. For low-cost, 100G links up to 100m. Lowest-cost, 100G-to-Quad- 25/50G Breakout cables. For Linking Servers + Storage to ToR Switch & NICs.
- 16. © 2015 Mellanox Technologies 16 Possible Scenarios
- 17. © 2015 Mellanox Technologies 17 Multi-Mode Optics 3m-100m Where Interconnects are Being Used in Data Center DAC Server/ToR-to-ToR SR4 For structured cabling Short Reaches PSM4 DAC WDM4 8-Fiber MPOAOC: 3-50m AOC ToR-Leaf/Spine “DAC In the Rack” 3m Quad 25G SFP breakout Dual 50G Breakout 25G SFP Quad 25G SFP breakout Dual 50G Breakout 25G SFP For Structured Cabling Long Reaches 2-Fiber LC Single-Mode Optics Up to 2Km Optical Patch Panel
- 18. © 2015 Mellanox Technologies 18 Small/Medium Cloud Deployment 10GbE endpoints (48 + 48)x 10/25GbE active HA (48 + 48)x 10/25GbE active HA 15x Racks Full L2 solution 1440x 10GbE ports WAN access 2x 10/40GbE uplink to router 2x SN2700 Spines To Spine mLAG 2x SN2410 ToRs 400G 960G Per Rack Pure L2 Network; full HA and no-SPoF Phase-1: Start with as small as 1 Rack and 2x ToR Phase-2: Add 2x spines (32x100G) and build up-to 15x Racks in pure L2 domain ToR to spine uplinks with 50GbE to ensure (2+2)x link bundle; mitigate cable failure (4+4)x 100/40GbE or (8+8)x 50GbE ports available per rack for high performance/”fat” storage nodes 48+48x 10GbE for compute/hyper- converged infrastructure 50GbE 100GbE
- 19. © 2015 Mellanox Technologies 19 Small/Medium Cloud Deployment 25GbE endpoints (48 + 48)x 10/25GbE active HA (48 + 48)x 10/25GbE active HA 7x Racks Full L2 solution 672x 25GbE ports WAN access 2x 10/40GbE uplink to router 2x SN2700 Spines To Spine mLAG 2x SN2410 ToRs 800G 2400G Per Rack Pure L2 Network; full HA and no-SPoF Ideal for small/medium private cloud Phase-1: Start with as small as 1 Rack and 2x ToR Phase-2: Add 2x spines (32x100G) and build up-to 7x Racks in pure L2 domain mLAG on ToRs and spines for full active- active HA (2+2)x 100GbE or (4+4)x 50GbE ports available per rack for high performance/”fat” storage nodes 48+48x 25GbE for compute/hyper- converged infrastructure 100GbE
- 20. © 2015 Mellanox Technologies 20 1024-port 1:1 100GbE, 2048-port 1:1 50GbE 32x Spines 64x Leafs 16x 100GbE 16x 100GbE 16x 100GbE 1024x 100GbE 1:1 non- blocking network All leaf-spine links are 50GbE 32x 50GbE uplinks Same Concept can be used for 2048 port 50GbE 1:1 Can be used as Spine for 3-level networks
- 21. Thank You
- 22. © 2015 Mellanox Technologies 22 References www.ieee802.org/3/ad_hoc/bwa/BWA_Report.pdf http://www.ieee802.org/3/50G/public/adhoc/ http://www.ethernetalliance.org/wp-content/uploads/2013/04/Ethernet-Alliance-Technology- Roadmap-FINAL.pdf https://en.wikipedia.org/wiki/Terabit_Ethernet http://www.ieee802.org/3/bs/ http://www.open-ethernet.com/ http://25gethernet.org/ http://www.mellanox.com/ethernet/