Building small and medium SANs

with IBM FlashSystem, for Flash and

NVMe
ATG Storage Template 2021

Tim Jeka Brian Larsen

Broadcom Senior FAE Idirector, Partner Business Development, Broadcom
[email protected] [email protected]

© Copyright IBM Corporation 2021

Accelerate with IBM Storage Technical Webinar Series

Advanced Technology Group – Storage experts cover a variety of technical topics.

Audience: Clients who have or are considering acquiring IBM Storage solutions. Business Partners and IBMers are also welcome.

To automatically receive announcements of upcoming Accelerate with IBM Storage webinars, Clients, Business Partners and IBMers are
welcome to send an email request to [email protected].

Accelerate with IBM Storage Support Site: ATG – Storage Media Center Channel:
https://www.ibm.com/support/pages/node/1125513 https://ibm.biz/BdfEgQ

2021 Upcoming Webinars:

March 9 - Say "WOW" to the FlashSystem 5200
Register Here: https://ibm.biz/Bdf4TC

March 18 - IBM Block Storage, OpenShift and IBM Cloud Paks, the rest of the story.
Register Here: https://ibm.biz/Bdf4VE

April 1 - Spectrum Virtualize 8.4 Technical Update

Register Here: https://ibm.biz/Bdf4Vq

April 6 – Four Things You Probably Don’t Know That TDMF (Transparent Data Migration Facility) Can Do
Register Here: https://ibm.biz/BdfidP

© Copyright IBM Corporation 2021

IBM ATG – Customer Workshops, Demos & Test Drives and Studies

IBM ATG DS8000 Offerings

Customer Workshops
• DS8000 Advanced Functions Virtual Workshop
• IBM Storage Cyber Resiliency Virtual Workshop

Demo / Test Drives for IBM/Business Partner skill development and for remote customer demonstrations
• Business Resiliency with Copy Services Manager Demonstration and Test Drive
• DS8900F Demonstration GUI Demonstration and Test Drive
• DS8900F Safeguarded Copy Test Drive
• DS8900F Safeguarded Copy SME lead Demonstration

Studies
• DS8000 Global Mirror Bandwidth Sizing – estimate required network bandwidth for target Recovery Point Objective (“RPO”)
• DS8880 Transparent Cloud Tiering Savings Analysis – calculate CPU savings from TCT cloud offload of HSM host cycles
• DS8000 Safeguarded Copy Sizing – calculate the capacity needed to meet Logical Corruption Protection ("LCP") requirements

© Copyright IBM Corporation 2021 3

Accelerate with IBM Storage Technical Webinar Series - Survey

Please take a moment to share your feedback with our team!

You can access this 5-question survey via Menti.com with code 15 75 27 5 or

Direct link https://www.menti.com/mkg7a2x6q8

Or

QR Code

© Copyright IBM Corporation 2021

Agenda

• The IBM FlashSystem Portfolio

• IBM b-type SAN Switch Portfolio

• IBM b-type Switch Added Value

• Brocade’s EZSwitchSetup

• Brocade’s Webtools V9.0 Overview

• Deploying NVMe-0F IBM B-type switches

• ESG Paper -IBM FlashSystem with Brocade Gen 7

Storage Networking(SWITCHES), and Emulex
Fibre Channel Technology
© Copyright IBM Corporation 2021
Legal Disclaimer
All or some of the products or offerings detailed in this presentation may still be under development and certain specifications, including but not
limited to, release dates, prices, and product features, may change. The products may not function as intended and a production version of the
products may never be released. Even if a production version is released, it may be materially different from the pre-release version discussed in
this presentation.

Nothing in this presentation shall be deemed to create a warranty of any kind, either express or implied, statutory or otherwise, including but not
limited to, any implied warranties of merchantability, fitness for a particular purpose, or non-infringement of third-party rights with respect to any
products and services referenced herein.

Copyright © 2021 Brocade Communications Systems LLC. All Rights Reserved. Brocade and the stylized B logo are among the trademarks of Brocade Communications
Systems LLC. Broadcom, the pulse logo, and Connecting everything are among the trademarks of Broadcom. The term “Broadcom” refers to Broadcom Inc. and/or its
subsidiaries.

© Copyright IBM Corporation 2021

IBM FlashSystem Portfolio
Simple, innovative, flexible, safe and fast
NVMe

NVMe
HIGH-END
FlashSystem 9200 FlashSystem 9200R

NVMe
MIDRANGE
New FlashSystem 7200

NVMe
ENTRY New
SAS
FlashSystem 5200

New FlashSystem 5035

SAS

FlashSystem 5015
IBM FlashSystem Family
FlashSystem FlashSystem FlashSystem FlashSystem
FlashSystem 7200
5015 5035 5200 9200
Software IBM Spectrum Virtualize Software
2x Broadwell @ 2.2GHz 2x Broadwell @ 2.2GHz 2x Skylake @ 2.3GHz 4x Cascade Lake 4x Cascade Lake
CPU (2 cores) (6 cores) (8 cores) @ 2.1GHz (8 cores) @ 2.3GHz (16 cores)
16Gb FC 16Gb FC 16/32Gb FC / NVMeoF
16/32Gb FC / NVMeoF 16/32Gb FC / NVMeoF
25Gb iSCSI 25Gb iSCSI 25Gb iSER / iSCSI
Connectivity 10Gb iSCSI 10Gb iSCSI 10Gb iSCSI
25Gb iSER / iSCSI 25Gb iSER / iSCSI
10Gb iSCSI 10Gb iSCSI
12Gb SAS 12Gb SAS 12Gb SAS
Cache 32GB or 64GB 32GB or 64GB 64GB to 512GB 256GB to 1.5TB 256GB to 1.5TB
Capacity Up to 392 drives (12 PB) Up to 504 drives (15 PB) Up to 748 drives (23 PB) Up to 760 drives (23PB) Up to 760 drives (23PB)
Advanced SW
Optional Optional Included Included Included
Features
Encryption -- Yes Yes Yes Yes
Compression -- SW HW-assist HW-assist HW-assist
Deduplication -- Yes Yes Yes Yes
Clustering -- Yes (2-way) Yes (4-way) Yes (4-way) Yes (4-way)
Ext. Virtualization -- -- Yes Yes Yes
3-year
1-year
1-5 year 3-year warranty warranty
3-year warranty 3-year warranty warranty
Service & Support 9x5, CSU Install 9x5, CSU Install
IBM Storage Expert Care 9x5, CSU Install
24x7, IBM
24x7, IBM
Basic or Advanced (ECS optional) Install
Install
(ECS)
Power AC or DC (800W PSU) AC or DC (800W PSU) AC (1200W PSU) AC (2000W PSU) AC (2000W PSU)
Max Performance 400k IOPS 1.2M IOPS 1.5M IOPS 2.3M IOPS 4.5M IOPS
8.3.1* 8 GB/s 12 GB/s 21 GB/s 35 GB/s 45 GB/s
What tools are available to simplify management of the IBM b-type Gen6 and
GEN7 Switches?

IBM SAN64B-7 Switch

• 1U, 24 to 56 64Gb/s ports
IBM SAN128-6 Switch • 2x performance
IBM SAN64B-6 Switch • 2U, 48 to 128 32Gb/s ports • 50% lower latency
• 1U, 24 to 64 32Gb/s ports • Enterprise bundle or • Autonomous SAN
individual features licensed Features
IBM SAN24B-6 Switch • Enterprise bundle or
• 1U, 8 to 24 32Gb/s ports individual features licensed separately • Traffic Optimizer
• Enterprise bundle or individual separately • Flow Learning
features licensed separately Version 2 switches (F65/N65 and F97/N97) are • Congestion Notify
FEATURE & FUNCTION IDENTICAL • All optional software
to the Version 1 switches (F64/N64 and F96/N96) licenses included

Performance and Functionality

© Copyright IBM Corporation 2021
IBM b-type SAN64B-7 Switch vs. SAN64B-6 Switch
A much better option for a very modest price premium
• Full Gen 7 feature set
– Traffic optimizer, congestion management

• Enterprise Bundle – and all other software – included

– Fabric Vision, ISL Trunking, Integrated Routing, FICON CUP,
Extended Fabrics

• Native NVMe-over-FC instrumentation and metrics

• Up to 50% lower latency compared to G620

IBM b-type SAN64B-7 Switch
• 64G speed
Latest generation Brocade
mid-range switch • Able to upgrade to FOS 10.x*

• Much longer service life until End-of-Support

* Entitlement subject to having current maintenance contract

© Copyright IBM Corporation 2021
Brocade 64G SWL SFP+ Optic
Brocade delivers the industry’s first 64G transceiver
• Accelerate data access with 64G links to unleash the full performance of
next-generation storage
– Backward-compatibility support for connectivity to 16/32G Fibre
Channel devices
– Supported in Gen 7 products only

• OM3/OM4/OM4+/OM5 operating distances across 16/32/64G Fibre

Channel
– OM3 50 µm (2000 MHz x km) Brocade 64G Transceiver
– 16G Distance: 0.5m to 100m
– 32G Distance: 0.5m to 70m
64G optics part number:
– 64G Distance: 0.5m to 70m • XBR-000462 (1-pack)
– OM4, OM4+, OM5 50 µm (4700 MHz x km) • XBR-000463 (8-pack)
– 16G Distance: 0.5m to 125m
– 32G Distance: 0.5m to 100m
– 64G Distance: 0.5m to 100m
© Copyright IBM Corporation 2021
IBM b-type Gen 7 64G Portfolio
The industry’s first 64G switches, adapters, and transceivers
Servers Network IBM Storage
Emulex HBAs Brocade SAN switches, SAN storage components
directors, and analytics (Emulex HBAs, RAID and
I/O controller chips)

• 64G Performance: 2X speed and 50% lower latency

• Innovation: End-to-end monitoring and diagnostics
• Future-Ready Now: Autonomous SAN technology

Investment protection for the next-generation of storage

© Copyright IBM Corporation 2021
Brocade’s EzSwitchSetup V9.0.x Overview

© Copyright IBM Corporation 2021 13

Simple Deployment

• EZSwitchSetup enables fast configuration and deployment

Brocade EZSwitchSetup deploys a switch faster

• Easy-to-use GUI for setting up and managing a
switch
• Simple steps to deploy a switch in minutes vs.
hours
• Intuitive device management
• Eliminates configuration errors

© Copyright IBM Corporation 2021

 Introduction to EZSwitchSetup

• EZSwitchSetup is an easy-to-use graphical user interface application setting up and managing

your switch.
It has the following components:
• EZSwitchSetup wizard downloadable from the MyBroadcom portal
• EZSwitchSetup switch configuration wizard
The following figure illustrates the high-level workflow of EZSwitchSetup.
• EZSwitchSetup Components
• The Setting Up Your Switch, describes how to use the EZSwitchSetup wizard and EZSwitchSetup
switch configuration wizard to set up and configure your switch for the first time.
• NOTE Although your switch might have advanced capabilities, EZSwitchSetup is for a single-switch
fabric with FC ports only. To configure and manage other features on your switch, use the command
line interface, Web Tools V9.0 or IBM SANav.

© Copyright IBM Corporation 2021

Downloading and Installing the EZSwitchSetup File

• You can run EZSwitchSetup from a computer that you are currently using for SAN
administration, or you can use a different computer that is not part of the SAN, such as a laptop.
• The computer that runs EZSwitchSetup is called the setup computer. You can download the
EZSwitchSetup software and documentation files from MyBroadcom for online installation.
• To install and run EZSwitchSetup, perform the following steps:
1. Go to http://www.broadcom.com/mybroadcom/, click LOGIN, and enter your username and
password. If you do not have an account, click REGISTER to set up your account

2. Click Support link on the top of the page and select the Customer Support Portal under the
Documents, Downloads and Support division. The Customer Support Portal page displays.

3. Click Brocade Products and select Downloads & Documentation

16
© Copyright IBM Corporation 2021
Accessing EzSwitchSetup Downloads
Enter the product name or the software version number in the Search box and press Enter.
In the result window, uncheck the Documents check box so that only software files are shown.
• The search results display, as shown in the following figure: EZSwitchSetup Search Downloads Page:

© Copyright IBM Corporation 2021

 Accessing EzSwitchSetup Downloads
• 5. Select the EZSwitchSetup check box, to download the software for Windows system
• The search results display, as shown in the following figure: EZSwitchSetup Search Downloads Page:

6. Click the Download Software icon under Actions to start with the download, or click the desired EZSwitchSetup
release link
7. Review and click Accept in the Broadcom License Agreement page.
8. Select a download method in the Please select a download method dialog.
9. Save the file, and extract the ZIP file.
10.Run the executable file. On Windows, the default path is usually
ezswitchsetup(version)\CDROM_Installers\Windows \Disk1\InstData\VM\install.exe.
© Copyright IBM Corporation 2021
EZSwitchSetup Installation Process
Click Next. The License Agreement window appears

© Copyright IBM Corporation 2021

Launching the EZSwitchSetup Wizard
• If the EZSwitchSetup wizard does not start automatically after installation,
• Go to Start > Programs > EZSwitchSetup and click EZSwitchSetup to run the EZSwitchSetup
wizard manually from your PC. EZSwitchSetup Introduction Window

• Use an Ethernet connection to the LAN on the same sub-net as the switch:
• Use a Serial cable to connect direct to the switch:
© Copyright IBM Corporation 2021
Click Next. The Connect Cables window displays.
Connecting Cables Windows displays

21
© Copyright IBM Corporation 2021
Discovering the Switch

• Click Next. EZSwitchSetup attempts to discover the switch.

• If the switch discovery fails, see Switch Discovery Failure information for details about how to
recover the switch.

© Copyright IBM Corporation 2021

Discovering the Switch

• Locate the WWN for your switch on the switch ID pull-out tab located on the
bottom of the port side of the switch.

© Copyright IBM Corporation 2021

 Second Discover Switch Window

• Select an option for assigning the IP address and click Next.

• The options vary depending on the configuration of your switch:

© Copyright IBM Corporation 2021

 Set Switch IP Address Window
• If you are setting up the switch for the first time, the addresses shown are not valid. If you click Next with
these addresses in place, EZSwitchSetup returns an error message.
• Click Next. EZSwitchSetup attempts to log in using default credentials.

© Copyright IBM Corporation 2021

Confirm IP Address Window/Continue Configuration

Check the displayed addresses carefully to ensure they are correct.

© Copyright IBM Corporation 2021

Content page, Port Information tab

© Copyright IBM Corporation 2021

 Brocade Webtools V9.0 Overview

© Copyright IBM Corporation 2021 28

IBM SANnav Management Portal and Global View

IBM SANnav Global and Management Port Models (MTM) SANnav Management Portal
Subscription Period

Product Name Description 1-Year 3-Year 5-Year Next-generation SAN management

Up to 600 SAN
IBM SANnav Ports.
application, architected from the ground
Management Portal
(Base)
No Director
Support.
9239- 9240- 9241-
B01 B03 B05
up with a focus on streamlining
Up to 15,000 SAN common workflows, such as
IBM SANnav Ports.
Management Portal Full Director 9239- 9240- 9241- configuration, zoning, deployment, and
(Enterprise) Support. E01 E03 E05
troubleshooting.
IBM SANnav Up to 20 SANnav 9239- 9240- 9241-
Global View Portal instances G01 G03 G05

SANnav Global View

Quickly visualize the health,
performance, and inventory of all the
fabrics across all SANnav Management
Portals using a simple, intelligent
dashboard

© Copyright IBM Corporation 2021 29

IBM b-type GEN6 and GEN7 supported by WEBTOOLS 9.x

Naming conventions for IBM b-Type SAN

IBM Name IBM Model Brocade Name

Gen 7 (64 & 256 Gbps) Family

IBM System Networking SAN512B-7 8961-F78 Brocade X7-8

IBM System Networking SAN256B-7 8961-F74 Brocade X7-4

IBM System Networking SAN64B-7 8960-R64/P64 Brocade G720

Gen 6 (32 & 128 Gbps) Family

IBM System Networking SAN512B-6 8961-F08 Brocade X6-8

IBM System Networking SAN256B-6 8961-F04 Brocade X6-4

8960-F97/N97 Brocade G630 v2

IBM System Networking SAN128B-6
8960-F96/N96 Brocade G630 v1

8960-F65/N65 Brocade G620 v2

IBM System Networking SAN64B-6
8960-F64/N64 Brocade G620 v1

IBM System Networking SAN24B-6 8960-F24 Brocade G610

IBM System Networking SAN18B-6 8960-R18 Brocade 7810

© Copyright IBM Corporation 2021

FOS v9.0 New WebTools….

• Modern HTML-based WebTools

• No Java!! – avoids many compatibility and security vulnerability issue
• Operating systems: Windows 10 Pro, Window 2019, RedHat 8.0 & 8.1
• Browsers: Chrome, FireFox

• Modern JavaScript/HTML UI technologies same as SANnav provide

clean and consistent look and feel
• Dashboard view provide health and performance summary

• Supports launching from SANnav UI

• FOS v9.0 supports most existing WebTools features:

• Hardware and logical switch view
• Switch overview and settings
• Port settings and zone admin
• Access Gateway settings
• Zone Configurations
• EZSwitchSetup will only support Initial IP address, Gateway, N/W Mask setting
© Copyright IBM Corporation 2021
Launching FOS v9.0 New WebTools
• If the switch is configured with logical fabrics, you can log in to any of the
logical fabrics for which you have permission.
• Launch Web Tools directly from a browser or from SANnav Management
Portal.
• To launch directly from a Web browser, open your browser, enter the IP
address of the switch, and press Enter.
• You can use HTTP or HTTPS, for example: http://10.77.77.77 or
https://10.77.77.77

• Enter the user name, password, and logical switch name or fabric ID
(FID). For the first switch login, the default user name is admin and the
default password is password. Web Tools prompts you to change the
default password
• Web Tools is launched directly from a Web browser or from SANnav
Management Portal. You can launch Web Tools on any workstation with a
compatible operating system and Web browser installed.

© Copyright IBM Corporation 2021

Overview of the Web Tools User Interface
• Once familiar with the basic components of Web Tools, you can quickly start monitoring and
managing your switch. The following screenshot shows the basic layout of the Web Tools user
interface
• 1. Navigation bar. Contains links to feature pages.
• 2. Profile menu. Displays the link for logging out.
• 3. Subnavigation bar. Provides the page title and includes buttons and menus to take actions
within the page.
• For more information about the SANnav Web Tools product, see Web Tools section of the
Brocade® Fabric OS® Web Tools User Guide, 9.0.x.
. https://docs.broadcom.com/doc/FOS-90x-WebTools-UG

© Copyright IBM Corporation 2021

 Dashboard
• Provides high level switch information at-a-glance to enable quick monitoring and troubleshooting.
• Dashboard auto refreshes after every 60 seconds.

© Copyright IBM Corporation 2021

Dashboard Switch Events

• Clicking on a specific bar on Switch events widget displays a list of the events
with the selected severity
• Memory Utilization
• Power Supply and FANs
• Temperature

© Copyright IBM Corporation 2021

 Switch Overview

• Switch overview displays the near real time pictorial hardware view of the switch – View auto
refreshes every 60 seconds
• LEDs indicate the status of various Switch components
• Ports tooltips indicate port type and other key information
• Port configuration can be launched by clicking on a switch port
• Ports not present in the current logical fabric, and vacant slots are greyed out
• Switch properties/details are displayed along with hardware view
• Switch configuration and others actions can be launched via Edit and Actions button/menu

© Copyright IBM Corporation 2021

© Copyright IBM Corporation 2021
WEBTOOLS Director View

© Copyright IBM Corporation 2021

Switch Overview - Actions
• Generate Report – Includes switch details, list of switches in the fabric, list of ISLs,
Nameserver details, Zoning details, SFP details
• Backup Configuration
• Restore Configuration
• Enable/Disable Switch Beacon – Flash LEDs for ports of the current logical switch
• Enable/Disable Chassis Beacon – Flash all LEDs of chassis in a specific pattern
• Mount/Unmount USB Driv
• Enable/Disable Switch
• Fast Boot • Reboot

© Copyright IBM Corporation 2021

Switch Ports - Overview
• Tabular view all Switch Ports and their detail
• Switch Ports Actions
• Quick access for individual port actions
• - Invoked by clicking on the down arrow icon (v) on each port row
• View/Edit
• Investigate
• Reserve/Release License
• Enable/Disable
• Actions Menu: Select multiple ports and then select the required action from Actions drop down
menu
• Rename
• Enable/Disable
• Error Statistic
• Advanced Action
40
© Copyright IBM Corporation 2021
Zoning Overview

• Zoning feature provides following zone management functionality:

• Create/View/Update/Delete of:
• Zone Configurations
• Also, Activate
• Zones
• Standard
• Peer
• LSAN
• Zone Aliases
• Preferences
• Zoning Policy

41
© Copyright IBM Corporation 2021
 FOS Version Management - Overview
• View FOS Versions on Primary and Secondary CPs
• Click “update” to perform Firmware Download to the switch
• Supports Firmware Download through Network and USB drive
• Network Protocols supported
• FTP
• SCP
• SFTP

© Copyright IBM Corporation 2021

 Configuration Management
• Web Tools Support Data Collection

© Copyright IBM Corporation 2021

When You’re Implementing NVMe
Over Fabrics, the Fabric Really Matters!
ATG Storage Template 2021

© Copyright IBM Corporation 2021

IBM b-type GEN7 Purpose-Built for NVMe Storage

Performance
Enable significant performance gains for flash
with low latency, NVMe-ready Fibre Channel

Seamless Integration
Deploy NVMe over Fibre Channel with no
rip-and-replace

Concurrent Traffic
Run NVMe and SCSI concurrently on same
network for gradual technology migration

© Copyright IBM Corporation 2021

Exploring NVMe over Fibre Channel

NVMe over Fibre Channel is relatively new, even thought its

component parts are not!
Gen 6 and Gen 7 support FC-NVMe
• Fibre Channel has been the leading enterprise storage technology since the
mid-1990’s
• Speeds of 16Gbps (GEN 5) and 32Gbps (GEN 6) are widespread.
• Gen 6 (2016), delivering twice the speed of GEN 5… Again, IBM and
Brocade announced GEN5 SAN EOL.
• GEN 7, which will double the speeds yet again, is already here!.
6
• You can deploy NVMe over Fibre Channel on an existing Brocade FC
infrastructure, providing it is relatively up to date!

7
© Copyright IBM Corporation 2021
FC-NVMe Basics
What is required for a full END-END IBM
NVMe-oF Exploitation!

ATG Storage Template 2021

© Copyright IBM Corporation 2021

NVMe-oF Basic Components

Acronym Definition
NVMe Drivers FC-NVMe Drivers will be provided by Fibre Channel
Vendors
NVMe Subsystem Non-volatile memory storage device
NVMe Controller A type of controller which supports minimal
functionality for discovery of NVMe media
controllers
NVMe Namespaces Similar to SCSI’s LUN (Logical Unit Number)
and Media identifier. The NSID is a set of logical block
addresses (LBA) on the NVM media
Queue Pairs A queue used to submit I/O commands to a
controller. Submission & Completion Queue.
© Copyright IBM Corporation 2021
NVMe over Fabrics Base Architecture Elements

NVMe Enabled Host NVMe-oF Network NVMe Storage

• Windows • Fibre Channel Subsystem
• Linux • RoCE V2 • NVMe Controller
• Unix • iWARP • NSID (Namespace ID)
• Vmware • TCP • NVMe Interface
• AIX
© Copyright IBM Corporation 2021
What you need – NVMe Enabled Host

NVMe Host Drivers

NVMe drivers in all major operating

systems.
Driver provides streamlined interface
between the O/S storage stack and
NVMe SSDs

© Copyright IBM Corporation 2021

NVMe/FC Host Operating Status…. IBM Qualified

The performance is much better with NVMe native multipath

© Copyright IBM Corporation 2021

What you need – NVMe Fabric Ports

Fabric Elements
• Subsystem Ports are associated with Physical Fabric Ports
• Multiple NVMe Controllers may be accessed through a single port
• Fabric Types; Fibe Channel, PCIe, RDMA (Ethernet RoCE/iWARP,
InfiniBand™),TCP

© Copyright IBM Corporation 2021

What you need – NVMe Subsystem

Architectural Elements
• NVMe Controller
• NVMe Namespace
• Fabric Ports

© Copyright IBM Corporation 2021

Why NVMe? What makes NVMe So Fast?
Instead of SCSI’s single command queue
(with a depth of 32 commands),
NVMe supports 65K queues with 65K
SCSI NVMe commands per queue, which means that a
far greater number of commands can be
executed simultaneously.

CPU CPU CPU CPU

Point-to-Point Up to 64K queues

Server to HBA
One Channel
w/ 64K commands
One Queue
per queue
32 Commands

© Copyright IBM Corporation 2021

Adopting and Deploying NVMe
Over Fiber Channel Fabrics
ATG Storage Template 2021

© Copyright IBM Corporation 2021

Investment Protection: No Rip Out and Replace
NVMe & SCSI concurrently is HUGE! IBM SANnav
Gen 5 HBAs
SCSI
Dual Emulex
• Run NVMe & SCSI concurrently on same Emulex HBAs by
SAN & HBAs with familiar fabric services Gen 6 HBA
Dual
• The Fibre Channel fabric must be Gen 6 (16 Emulex
Gbps) or better yet Gen 7 (32 Gbps)
NVMe NVMe Gen 7
• Servers using NVMe over Fibre Channel HBA
need Emulex Gen 6 (HBAs); Gen 6 HBAs
also work with Gen 7 fabrics 7
• You need a storage device that supports FC-
NVMe frame types. This can be an FC-
NVMe-enabled array 7 7 55% lower
latency
• Existing Gen 6 and 7 SANs can run NVMe
without code changes or disruptions
SCSI over FC, Translated
NVMe over FC, Natively
• Clone & Remap your existing SCSI luns to
NVMe NSIDs (Namespace ID)
NVMe Traffic
• Full fabric awareness, visibility and SCSI Traffic
manageability with existing SANnav &
Webtools technology

© Copyright IBM Corporation 2021

Exploiting Dual Protocol FCP and FC-NVMe
Best Practices…Dual-protocol infrastructure (Recommended)
Servers
• Deploy NVMe based arrays SCSI-on-FC
– Leverage existing SAN infrastructure FC-NVMe GEN6 HBAs
– Easily supported dual infrastructure
– How long will the transition take?
– Avoid risks
• Incremental Migration
– Applications dictate how individual volumes can
be migrated
– Changes can be rolled back easily without
DUAL Protocol SAN arrays Existing 32Gb FC fabric
disruption to hardware or cabling
Existing enterprise SCSI-on-FC
• FCP & NVMe over FC can both leverage FC Storage infrastructure FC-NVME
zoning
– Only SAN Zoning improves security
– Zoning restricts devices from accessing
network areas that should not be visiting!
• Discovery
– Emulex create drivers that leverage FCP for SCSI on FC Arrays FC NVMe Arrays
device discovery, then check those devices for
FC-NVMe traffic support
© Copyright IBM Corporation 2021
Where to start? The Lab!

• Steps to establish a FC-NVMe test lab. Servers

SCSI-on-FC
FC-NVMe GEN6 HBAs
– Set up a single server with an internal
drive, single switch, and a single
FC-NVMe enabled array

– Connect the server to an IP network

for access to other servers.

– Explore the HBA config and storage array

options you’ve read so much about! DUAL Protocol SAN arrays Existing 32Gb FC fabric
Existing enterprise SCSI-on-FC
– Configure a volume to use the NVMe array
Storage infrastructure FC-NVME
– Copy a file from the internal drive to the NVMe
volume and back again.

– Run your performance testing application (Iometer)

to benchmark your NVMe volume. Compare it to
your internal volume.
SCSI on FC Arrays FC NVMe Arrays
• Lather, rinse, and repeat until your paranoid nature is
satisfied!
© Copyright IBM Corporation 2021
Ten NVMe over Fibre Channel Takeaways

1. Reduce risk to business operations by extending the existing SCSI/FC network to

include NVMe/FC
2. Improve server CPU utilization and ROI with NVMe/FC.
3. Drive more application workload because NVMe’s input/ output operations per
second (IOPS) will have as much impact as its latency.
4. Install NVMe-ready devices today to allow applications to exploit the next generation
of low latency flash — Storage Class Memory
5. Leverage the multiple NVMe flash media form factors to cost optimize your IT
infrastructure
6. Allow the NVMe future to come at its own pace.
7. Include vendor support in your NVMe-FC adoption plan:
8. Fibre Channel (FC) and Ethernet/IP are optimized different
9. Dedicate a separate storage fabric for best results.
10. Maximize your Fibre Channel fabrics and minimize your migration by running both
SCSI/FC (FCP) and FC-NVMe.
© Copyright IBM Corporation 2021
Maximize Database Efficiency and Performance in a
VMware Environment using 32G NVMe over Fibre
Channel

IBM FlashSystem with Brocade Gen 7 Storage Networking and Emulex Fibre
Channel Technology

Tested at IBM ATG Labs in Herndon, VA

Brian Larsen Director, Partner Business Development, Broadcom

© Copyright IBM Corporation 2021

ESG Validation Paper Overview

• Objectives:
• Simulate a client use case by using a
“virtualized” environment with a “database”
application

• Identify performance advantage of using

NVMe over FC (NVMe/FC) versus a SCSI over
FC (SCSI/FC) in a “real-world” environment

• Validate the “seamless” migration of moving

from an existing SCSI/FC infrastructure to a
NVMe/FC infrastructure

• Validate “self-healing” capabilities using

advanced features for data traffic congestion

© Copyright IBM Corporation 2021 61

IBM Advanced Technology Group (ATG formerly WSC) Lab Environment

• Emulating a “Real World” Environment

• The h/w and s/w used in this lab setup is all available for
purchase and is qualified for this deployment
• The test was designed with a goal of quantifying
protocol performance benefits, not the maximum
performance capabilities of any specific platform
• Lab configuration, setup & methodology
• A typical database workload was tested with
HammerDB, an industry-standard, open source
database load testing and benchmarking tool
• Oracle 19c applications were tested in a VMware
vSphere 7 environment running on a 56-core Intel Xeon
server with an Emulex LPe35002 FC host bus adapter
connected through an IBM SAN64B-7 storage network
switch to an IBM FlashSystem 9200
• The storage network was configured with two 32G Fibre
Channel connections between the FlashSystem 9200
and the storage network switch.
• This configuration was zoned to make it easy to compare the
performance of applications that were deployed on different
storage networks (e.g., the latest 32G end-to-end NVMe over
Fabrics versus SCSI over 32G Fibre Channel).

© Copyright IBM Corporation 2021 62

NVMe over FC –Oracle/Vmware Performance Testing Result

• OLTP Testing and Results

Increasing Oracle OLTP Performance with End-to-End NVMe
• OLTP workload was tested with an Oracle 19c
database and HammerDB
• The HammerDB workload generator emulates the
activity of users in a typical online brokerage firm as Oracle
they generate trades, perform account inquiries, and
execute market research TPM
• The Oracle 19c database was configured with an 8KB +83%
database block size
• OLTP Results
• Oracle was able to sustain 83% more transactions
per minute when running NVME/FC as compared to
traditional SCSI/FC
• Throughput Performance Results Throughput
• NVMe/FC was 79% higher than traditional SCSI/FC MB/s
+79%

© Copyright IBM Corporation 2021 63

NVMe over FC –Oracle/Vmware Performance Testing Result

• CPU Impact Analysis

• Test Objective
• ESG looked at how NVMe/FC protocol improves the Maximizing CPU Efficiency for Transactions Per Minute (TPM)/ Core
efficiency of CPU and I/O overhead of application
servers
• CPU Efficiency Findings
• The CPU core utilization statistics quantify the
increased CPU utilization when testing NVMe/FC Oracle
• 32G SCSI/FC - 26% per core utilization TPM/Core
• 32G NVMe/FC – 35% per core utilization (+9% increase) +31%
Utilization
• End User Impact
• The small increase (+9%) in system CPU utilization
is more than offset by the increased transaction
processing (+31%)
• Correlating the application-level transactions per
minute with CPU core utilization provides a useful
measure of the I/O CPU efficiency benefits of end-
to-end NVMe for Oracle applications, in this case up
to 31% more TPM per core

© Copyright IBM Corporation 2021 64

ESG Vmware & vMotion Migration to NVMe/FC from SCSI/FC

• Validation Objective
• Confirm the ease of transition from an existing
SCSI/FC data path to an NVMe/FC data path
• Methodology
• VMware vCenter 7.0 was deployed on an ESXi host
containing multiple virtual machines (VMs) to
datastores loaded onto the IBM FlashSystem 9200
via an NVMe fabric
• The FlashSystem 9200 contained two volumes
• 10GB SCSI/FC disk (containing the “FCP200” datastore)
• 10GB NVMe/FC disk (containing the “NVMe200”
datastore).
• Simulated OLTP-like workload was loaded on
“VM21” with a 70/30 read/write ratio, 32 threads,
queue depth of two, and 4KB I/O size
• Simulated workload was used to characterize
performance before and after application migration Initial Workload & Performance Monitoring
• VM21 was initially connected to the datastore on SCSI/FC datastore – Migrated to NVMe/FC
“FCP200” and migrated to the datastore on datastore
“NMMe200 65
© Copyright IBM Corporation 2021
ESG Validation of Congestion Management & “Self-Healing” Capabilities

• Problem Statement:
Congestion “Self-Healing” Test Configuration:
• Broadcom customers consistently reported that finding A virtualized x86 server with virtualized applications are driving
and fixing SAN congestion problems is a top SAN workload to the IBM Flashsystems platform through two different
management challenge FC Adapters and a single FC SAN switch
• Validation Objective
• Confirm that Fabric Performance Impact Notification
(FPIN) implementation can provide “Self Healing” of
fabric congestion
• Methodology
• A simple configuration was used with two paths to the
storage controller
• IBM SANnav and Emulex SAN Manager (ESM) mgmt
tools monitored for congestion notification and reaction
• Simulated congestion was introduced into the fabric to Simulated Congestion Introduced:
understand the impact of FPIN and the infrastructure A deliberate speed mismatch between the flash array and
responses server HBAs were used to simulate a congestion slowdown
• Industry-standard FIO benchmark utility was used to for a higher priority data warehouse application after a
simulate the large block (1MB) sequential read traffic of the throughput-intensive workload was started
bully and victim workloads

© Copyright IBM Corporation 2021 66

ESG Validation of Congestion Management & “Self-Healing” Capabilities

• ESG Observations
• SAN congestion was flagged as a problem through the IBM
IBM SANnav Identifying Path Congestion
SANnav as an event that needed investigation.
• IBM SANnav made it easy to visually investigate the error
with a single click to drill down to explore the cause of the
alerts
• IBM b-type switches detected and identified the congestion impact,
the FPIN was sent to the end device, in this case, the Emulex FC HBA.
• Emulex SAN Manager (ESM) visualized the FPIN notifications
sent by b-type switches
• A moderate level of port congestion management was set for
the lower priority application
• This setting tells the Emulex FC HBAs to slow down the lower priority
“bully” traffic and eliminates the port congestion issue through an
automated process
• Client Benefit
• Finding, fixing, and automatically eliminating SAN
congestion problems saves time and money for IT
professionals and improves customer satisfaction and
ESM Receiving FPIN and Remediating Event
productivity for application users
© Copyright IBM Corporation 2021 67
 ESG Technical Validation Summary

• The Bigger Truth

• ESG validated that in a virtualized VMware vSphere 7
environment, an IBM FlashSystem 9200 end-to-end 32G
NVMe solution with IBM Gen 7 b-type storage networking and
Gen 7 Emulex FC HBAs:

• Delivered 83% more Oracle OLTP transactions per minute and 79%
higher throughput in a VMware ESXi 7.0 environment compared with
32G SCSI/FC technology

• Provided up to 31% more performance per CPU core for Oracle

databases

• Can help organizations to perform application migrations from

SCSI/FC to NVMe/FC-based storage easily and non-disruptively

• Have leveraged the FPIN specification to automatically find and fix

SAN congestion problems. Detecting and troubleshooting SAN
congestion with IBM SANnav Management Portal and displaying FPIN
notifications with Emulex SAN Manager (ESM) was intuitive and easy
Download the ESG Paper on IBM Community: ESG Technical Validation - Vmware/Oracle/NVMe over FC
© Copyright IBM Corporation 2021 68
Want to get up to speed on
NVMe Fast?

Get the Book!!!

Please take a copy
or scan for PDF
version.

© Copyright IBM Corporation 2021

IBM Redbook for b-type Gen 7

• What is Happening?
– The IBM Redbook team and Brocade have been working on Leveraging IBM b-type Gen 7
creating an updated Redbook for the IBM b-type Gen 7 Storage Networking to
offerings Create an Autonomous SAN
for the Future
• What can be expected?
– Gen 7 Product Detail – H/W & S/W
– Design considerations
– Management Tools
– & Much More

• When will it be available?

– The teams are targeting a draft being published sometime
early 2021
– The IBM Redbook team will let everyone know when this will
be ready for review

© Copyright IBM Corporation 2021

IBM Redbook for b-type Gen 7
• Authors:
– Brocade Business Development
– Brocade Principle Architects Leveraging IBM b-type Gen 7
– Brocade Solutioneers Storage Networking to
– Brocade Systems Engineers Create an Autonomous SAN
for the Future
• Table of Content:
– Introduction to an Autonomous SAN
– Gen 7 Hardware
– Fabric Operating System
– FC Virtual Fabrics
– FC SAN Design, Implementation and Migration
– FICON SAN Design, Implementation and Migration
– Extension Design, Implementation and Migration
– NVMe Design, Implementation and Migration
– Fabric Administration
– Fabric Automation
– SANnav Implementation and Setup
– SAN Security
© Copyright IBM Corporation 2021
Accelerate with IBM Storage Technical Webinar Series

Advanced Technology Group – Storage experts cover a variety of technical topics.

Audience: Clients who have or are considering acquiring IBM Storage solutions. Business Partners and IBMers are also welcome.

To automatically receive announcements of upcoming Accelerate with IBM Storage webinars, Clients, Business Partners and IBMers are
welcome to send an email request to [email protected].

Accelerate with IBM Storage Support Site: ATG – Storage Media Center Channel:
https://www.ibm.com/support/pages/node/1125513 https://ibm.biz/BdfEgQ

2021 Upcoming Webinars:

March 2 - Building small and medium SANs with IBM FlashSystem for Flash and NVMe Solutions
Register Here: https://ibm.biz/BdfH9U

March 9 - Say "WOW" to the FlashSystem 5200

Register Here: https://ibm.biz/Bdf4TC

March 18 - IBM Block Storage, OpenShift and IBM Cloud Paks, the rest of the story.
Register Here: https://ibm.biz/Bdf4VE

April 1 - Spectrum Virtualize 8.4 Technical Update

Register Here: https://ibm.biz/Bdf4Vq

© Copyright IBM Corporation 2021

IBM ATG – Customer Workshops, Demos & Test Drives and Studies

IBM ATG DS8000 Offerings

Customer Workshops
• DS8000 Advanced Functions Virtual Workshop
• IBM Storage Cyber Resiliency Virtual Workshop

Demo / Test Drives for IBM/Business Partner skill development and for remote customer demonstrations
• Business Resiliency with Copy Services Manager Demonstration and Test Drive
• DS8900F Demonstration GUI Demonstration and Test Drive
• DS8900F Safeguarded Copy Test Drive
• DS8900F Safeguarded Copy SME lead Demonstration

Studies
• DS8000 Global Mirror Bandwidth Sizing – estimate required network bandwidth for target Recovery Point Objective (“RPO”)
• DS8880 Transparent Cloud Tiering Savings Analysis – calculate CPU savings from TCT cloud offload of HSM host cycles
• DS8000 Safeguarded Copy Sizing – calculate the capacity needed to meet Logical Corruption Protection ("LCP") requirements

© Copyright IBM Corporation 2021 73

Accelerate with IBM Storage Technical Webinar Series - Survey

Please take a moment to share your feedback with our team!

You can access this 5-question survey via Menti.com with code 15 75 27 5 or

Direct link https://www.menti.com/mkg7a2x6q8

Or

QR Code

© Copyright IBM Corporation 2021