Fabric OS V6.4.0a Release Notes v3.0 (PDF, 609.79 KB)
Fabric OS V6.4.0a Release Notes v3.0 (PDF, 609.79 KB)
0a
Release Notes v3.0
Document History
Brocade Fabric OS v6.4.0a Release Notes v1.0 Initial Release June 4th, 2010
Brocade Fabric OS v6.4.0a Release Notes v2.0 Updated FICON Appendix, minor June 29th, 2010
corrections in Important Notes,
additional Bottleneck Detection
details.
Brocade Fabric OS v6.4.0a Release Notes v3.0 Updates throughout document September 9th,
including 7800 port upgrade license 2010
section, Important Notes for Zoning,
FCoE, AP7420, Integrated Routing,
Bottleneck Detection and other
areas, other miscellaneous updates.
No updates to the Defect Tables have
been made.
Copyright © 2001 - 2010 Brocade Communications Systems, Inc. All Rights Reserved.
Brocade, Fabric OS, File Lifecycle Manager, MyView, and StorageX are registered trademarks and the
Brocade B-wing symbol, DCX, and SAN Health are trademarks of Brocade Communications Systems, Inc.,
in the United States and/or in other countries. All other brands, products, or service names are or may be
trademarks or service marks of, and are used to identify, products or services of their respective owners.
Notice: The information in this document is provided ―AS IS,‖ without warranty of any kind, including,
without limitation, any implied warranty of merchantability, noninfringement or fitness for a particular
purpose. Disclosure of information in this material in no way grants a recipient any rights under Brocade's
patents, copyrights, trade secrets or other intellectual property rights. Brocade reserves the right to make
changes to this document at any time, without notice, and assumes no responsibility for its use.
The authors and Brocade Communications Systems, Inc. shall have no liability or responsibility to any
person or entity with respect to any loss, cost, liability, or damages arising from the information contained
in this book or the computer programs that accompany it.
Notice: The product described by this document may contain ―open source‖ software covered by the GNU
General Public License or other open source license agreements. To find-out which open source software
is included in Brocade products, view the licensing terms applicable to the open source software, and
obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.
Export of technical data contained in this document may require an export license from the United States
Government.
Overview
New Hardware Support
Brocade Fabric OS (FOS) v6.4.0 supports the following new hardware platforms:
Brocade FC8-64: 64-port 8Gb Fibre Channel blade for the DCX and DCX-4S
o Supports E, EX, F, FL and M-port types
o Supports auto sensing 2/4/8 Gbps port speed
o Needs special type of SFPs called mSFPs
Encryption Enhancements
The following software enhancements are made to the Encryption platforms (BES and FS8-18) in FOS v6.4.0:
Concurrent Disk and Tape Encryption support on the same Encryption Engine
This feature allows users to configure disk and tape containers on the same Encryption Engine (EE). In
pre-FOS v6.4.0, users needed to dedicate an Encryption Engine either for disk encryption or tape
encryption. This restriction is lifted in FOS v6.4.0. However, users must note that the encryption
performance could be lower when both disk and tape encryptions are concurrently performed on the
same Encryption Engine, when compared to a scenario where an Encryption Engine would be
performing either disk or tape encryption exclusively.
Disk device decommissioning support
This feature provides the ability to render all encrypted data on a disk to be made
irrecoverable before a disk is decommissioned. This is achieved by ensuring that
the encrypted data on a disk cannot be decrypted by any means.
Redundant Key ID metadata option in data replication environments
FOS v6.4.0 has been enhanced to perform key recovery operation in data
replication environments.
FOS v6.3.1b introduced several changes and enhancements that are also included in FOS v6.4. These
changes are not fully documented in existing Admin Guides or other materials but will be captured in future
documentation and existing documentation updates and revisions. A brief summary of these enhancements
follows:
General improvements to Bottleneck Detection on both 4G and 8G platforms including improved
accuracy on reporting latency and reporting of latency values in Bottleneck Detection events.
New option to configure the switch ―edge hold time,‖ allowing the switch to timeout frames for F_Ports
sooner than for E_Ports. Proper use of this capability (disabled by default) reduces the likelihood of
devices with high latencies causing frame drops in the core of the fabric and impacting other
unrelated flows. Details on usage and recommended settings will be available in separate
documentation.
Support for Class 3 frame transmit (Tx) timeout discard counters on 4G platforms (previously 4G
platforms only supported receive (Rx) timeout counters, reference Fabric Watch Administrator’s Guide
for details on use and configuration).
The Bottleneck Detection feature does not need any additional license and is available on all 4G and 8G
capable platforms.
Security Enhancements
FOS v6.4.0 adds support for FCAP authentication using third-party self signed certificates. Starting with FOS
v6.4.0 both Brocade issued certificates and/or third-party self signed certificates can be used for FCAP
authentication. Prior to FOS v6.4.0 only Brocade issued certificates were supported.
CEE Enhancements
FOS v6.4.0 adds support for IGMP snooping functionality on Brocade 8000 and FCOE10-24 blade. This
implementation supports IPv4 version of IGMP v1/v2. IGMP Snooping allows multicast data to be forwarded to
only interested member ports of a VLAN.
Deprecated Features
FAL API
FOS v6.4.0 no longer includes FAL API support. Equivalent functionality is available through the
following SMI-S agents:
o SMI-S 120.11.0
o DCFM 10.4 Professional Plus and DCFM 10.4 Enterprise editions.
Option to install only SMI-S without DCFM client capabilities is also supported.
Advanced FICON Acceleration – This licensed feature uses specialized data management techniques
and automated intelligence to accelerate FICON tape read and write and IBM Global Mirror data
replication operations over distance, while maintaining the integrity of command and
acknowledgement sequences. This license is available on the 7800 and the DCX/DCX-4S for the FX8-
24 on an individual slot basis.
Some models offer bundles that include 2 or more optionally licensed features. These bundles are defined
Access Gateway mode is also supported by Fabric OS v6.4.0, and is supported on the following switches: the
Brocade 300, 5100, VA-40FC, 8000, 5450, 5460, 5470, 5480, NC-5480 and M5424.
Standards Compliance
This software conforms to the Fibre Channel Standards in a manner consistent with accepted
engineering practices and procedures. In certain cases, Brocade might add proprietary supplemental
functions to those specified in the standards. For a list of FC standards conformance, visit the
following Brocade Web site: http://www.brocade.com/sanstandards
The Brocade 8000 and FCOE10-24 blade conform to the following Ethernet standards:
IEEE 802.1D Spanning Tree Protocol
IEEE 802.1s Multiple Spanning Tree
IEEE 802.1w Rapid reconfiguration of Spanning Tree Protocol
IEEE 802.3ad Link Aggregation with LACP
IEEE 802.3ae 10G Ethernet
IEEE 802.1Q VLAN Tagging
IEEE 802.1p Class of Service Prioritization and Tagging
IEEE 802.1v VLAN Classification by Protocol and Port
IEEE 802.1AB Link Layer Discovery Protocol (LLDP)
IEEE 802.3x Flow Control (Pause Frames)
The following draft versions of the Converged Enhanced Ethernet (CEE) and Fibre Channel over Ethernet (FCoE)
Standards are also supported on the Brocade 8000 and FCOE10-24 blade:
IEEE 802.1Qbb Priority-based Flow Control
IEEE 802.1Qaz Enhanced Transmission Selection
IEEE 802.1 DCB Capability Exchange Protocol (Proposed under the DCB Task Group of
IEEE 802.1 Working Group)
FC-BB-5 FCoE (Rev 2.0)
FT00X0054E9
TSB Summary
None There are no current TSBs outstanding for this FOS release.
TSB Summary
None There are no TSBs currently issued for FOS v6.4 releases.
Upgrading to Fabric OS v6.4.0 is only allowed from Fabric OS v6.3. This policy to support only one-level
migration, which began with FOS v6.0.0, provides more reliable and robust migrations for customers. By
having fewer major changes in internal databases, configurations, and subsystems, the system is able to
perform the upgrade more efficiently, taking less time and ensuring a truly seamless and non-disruptive
process for the fabric. The one-release migration policy also reduces the large number of upgrade/downgrade
permutations that must be tested, allowing Brocade to spend more effort ensuring the supported migration
paths are thoroughly and completely verified.
FOS does not support concurrent FC Routing (EX_Ports) and TopTalkers features.
If there are multiple node EGs (encryption groups) in a fabric, please complete firmwaredownload on one node
at a time before downloading on another node.
The Brocade 8000 does not support non-disruptive hot code loads (HCL). Upgrading the Brocade 8000 to FOS
v6.4 will be disruptive to the IO through the switch.
A code load of DCX or DCX-4s with one or more FCOE10-24 blades will disrupt the traffic going through those
FCOE10-24 blades.
Disable the ports in DCX Logical Switches that use 10 bit addressing mode that have 8 bit areas in the range
0x70-0x8F before upgrading to FOS v6.4.0. Otherwise firmware upgrade will fail with an error message. This
step is necessary even if users do not plan to use FC8-64 blades after performing firmware upgrade to FOS
v6.4.0. However, if areas 0x70-0x8F are not in use this step is not necessary. Please use portAddress CLI to
find out the areas in use within a Logical Switch.
If Bottleneck detection feature is currently enabled on the switch running FOS v6.3.x, you must disable it
before upgrading to FOS v6.4.0; otherwise, frame drops may occur due to increased Hot Code Load (HCL)
time.
7800 platform and FX8-24 blade must be power cycled after upgrading from FOS v6.3 to FOS v6.4. This is
necessary to load the new FPGA image that enables IPv6 capability for FCIP links to 7800 and FX8-24 blade.
This step is mandatory even if IPv6 will not be used on the FCIP ports. Not performing this step will result in
unpredictable behaviors on the FCIP links. Please note that in the case of FX8-24, only the FX8-24 blade needs
to be power cycled and not the entire DCX/DCX-4S chassis.
DCFM Compatibility
FOS v6.4 is compatible with Brocade’s Data Center Fabric Manager (DCFM) v10.4 management software.
DCFM is a comprehensive SAN management application that enables end-to-end management of Brocade
Data Center Fabrics. It is the next-generation successor product to legacy Brocade management products
(Brocade Fabric Manager (FM) and Brocade Enterprise Fabric Connectivity Manager (EFCM)).
DCFM 10.4 now includes introductory support for FOS switches or fabrics using Administrative Domains (ADs).
These details and more about DCFM’s new enhancements can be found in the DCFM 10.4 Release Notes,
DCFM 10.4 User Guide, and DCFM 10.4 Installation, Migration, & Transition Guide.
SMI Compatibility
FOS v6.4 is supported with SMI-S agent 120.11.0.
FOS v6.4 is supported with SMI-S Agent integrated with DCFM 10.4
Fabric OS Compatibility
The following table lists the earliest versions of Brocade software supported in this release, that is, the earliest
supported software versions that interoperate. Brocade recommends using the latest software versions to get
the greatest benefit from the SAN.
When using the Virtual Fabrics feature, it is highly recommended that all switches participating in a fabric with
a logical switch use the latest firmware available for those switches. All switches must be operating at
minimum firmware levels noted in the FOS Interoperability table below.
For a list of the effective end-of-life dates for all versions of Fabric OS, visit the following Brocade Web
site:
http://www.brocade.com/support/end_of_life.jsp
Silkworm 3016, 3250, 3850 and Brocade 3900, 4100, 4900, v5.3.2 (2G and 4G platforms) and
24000, 7500, 7500E, 5000, 200E, 48000 v6.1.0e and later 5 (4G platforms
only)
Brocade 4012, 4016, 4018, 4020, 4024, 4424 v5.3.1b, v6.1.0e and later 5
Brocade 7800, DCX and DCX-4S with FCOE10-24 or FX8-24 blades v6.3 and later
Brocade DCX with FS8-18 blade(s), Brocade Encryption Switch v6.1.1_enc and later
Brocade DCX/DCX-4S/48000 with FA4-18 blade(s), Brocade 7600 v5.2.x or later (DCX requires v6.0.x
or later, DCX-4S requires 6.2.x or
later)
Mi10k, M6140, ED-6064, ES-3232, ES-4300, ES-4400, ES-4500, ES- M-EOS v9.9.5 or later 3
4700 (McDATA Fabric Mode and Open Fabric Mode) 2 4
Blade Support
Fabric OS v6.4.0 software is fully qualified and supports the blades for the 48000 platform noted in the
following table:
Fabric OS v6.4.0 software is fully qualified and supports the blades for the DCX/DCX-4S noted in the following
table:
FC-IP/FC Router blade (FR4-18i) Up to a maximum of 4 blades of this type. This can be
extended under special circumstances, but must be
approved by Brocade’s Product Team. Up to 8 FR4-18i
blades can be installed in a DCX if they are used only for
FC FastWrite or FCIP without routing.
Note: the iSCSI FC4-16IP blade is not qualified for the DCX/DCX-4S.
Note: FICON Cascaded CUP with M-EOS and FOS qualified only on select platforms.
Scalability
All scalability limits are subject to change. Limits may be increased once further testing has been completed,
even after the release of Fabric OS. For current scalability limits for Fabric OS, refer to the Brocade Scalability
Guidelines document, available under the Technology and Architecture Resources section at
http://www.brocade.com/compatibility
FOS v6.4.0 no longer automatically enables the Management Server (MS) Platform capability when a switch
attempts to join a fabric that has these services enabled. This prevents a FOS v6.4.0 switch from joining such a
fabric, and ISL will be disabled with a RAS log message. To allow a FOS v6.4.0 switch to join such fabrics
msPlMgmtActivate command should be used to enable the Management Server platform services explicitly .
FCIP, FCIP Trunking and High Bandwidth (Brocade 7800 and FX8-24)
IPsec is not supported on XGE0 of FX8-24 blade in FOS v6.4.0. IPsec is supported on XGE1 and
GE0 through GE9.
IPsec is supported on FCIP tunnels that use only IPV4 connections.
FICON networks with FCIP tunnels do not support DPS (aptpolicy 3) configurations. This applies to
both emulating and non-emulating FCIP tunnels.
The maximum supported MTU size for the Brocade 7800/FX8-24 is 1500 with FOS v6.4.0.
FCIP connections are supported only between the Brocade 7800/FX8-24 and another 7800/FX8-
24. FCIP tunnels are not supported between the 7800/FX8-24 and the previous generation
Brocade 7500/FR4-18i platforms.
When multiple FCIP tunnels are present on a switch and additional circuits (and the network
bandwidth provided by those circuits) are added to an already active tunnel, there may be a short
period of time where some frame loss can occur due to the process to re-fresh the internal FC
frame routing tables in the switch. Therefore, additional circuits should only be added during low
I/O periods utilizing the FCIP Tunnel being modified. In addition, if the circuit operation
(addition/deletion) to the tunnel increases/decreases the total tunnel bandwidth, an FCIP Tunnel
(VE port) disable/enable sequence should be performed after the addition/deletion of the circuit.
This will allow the switch to adjust the internal routes to utilize the new bandwidth fully.
Switching modes between 10G and 1G is disruptive for FCIP traffic.
Keep alive timeout (milliseconds) - Valid range is 500ms to 7,200,000ms (inclusive).
FOS default value is 10000ms (10 seconds). If FICON is configured the recommended
value is 1000 ms (1 second), otherwise the recommended value is the default of 10
seconds. For impairment networks with 100ms latency and 0.5% packet loss, keep-alive
time out should be configured as 30seconds. If the local and remote circuit
configurations’ Keep Alive Timeout values do not match, the tunnel will use the lower of
the two configured values.
Software compression (available on the 7800 and FX8-24) modes 2 and 3 are only
supported in Open Systems environments.
In order to perform the following operations it is necessary to delete the FCIP
configuration on the affected ports first:
o Switching modes between 1G/10G/Dual.
o Moving VE/GE port between logical switches.
Under Traffic Isolation Zone, configurations with fail over enabled, Non-TI zone traffic will use the
dedicated path if no other E or VE paths through the fabric exist, or if the non-dedicated paths are
not the shortest paths. (A higher bandwidth tunnel with multiple circuits will become shortest
path compared to a single tunnel).
A VE/VEX Tunnel and E/EX FC port cannot connect to the same domain at the same time.
The recommended Keep Alive Timeout must be the same on tunnel/circuits on the switches on
both sides of a link.
Latency measurements supported on FCIP Tunnels (Tested limit under FOS v6.3.1):
o 1GbE - 200ms round trip time and 1% Loss
o 10GbE – 100ms round trip and 0.1% Loss
Brocade 7800 supports Optical and Copper Media types on GE0 and GE1 interfaces. Copper
Media type is default on GE0/GE1 ports and does not support auto-sense functions.
After inserting a 4G SFP in GE ports of an FX8-24 blade or 7800 switch, sometimes ―sfpshow‖
output might display ―Can not read serial data!‖ . Removing and re-inserting the SFP should
resolve this issue. It is recommended that users perform sfpshow immediately after inserting the
SFP and ensure SFP is seated properly before connecting the cables.
When running FOS v6.4.0, if any of the following features are enabled in the FCIP configuration, a
downgrade operation will be blocked until the features are removed from the FCIP config:
o IPv6
o IPSec on the FX8-24
o DSCP Markings
o Advanced Compression options 2 and 3 on the FX8-24
o VEX ports on the FX8-24
Hot plugging a CP with firmware level less than FOS v6.3.0 into a DCX or DCX-4S with an active
FCOE10-24 blade will result in the new standby CP not coming up.
Brocade recommends that Converged Mode be enabled on all interfaces connected to CNAs.
When operating in Converged Mode, tagged traffic on the native VLAN of the switch interface is
processed normally. The host should be configured not to send VLAN tagged traffic on the
switch’s native VLAN.
When operating in Converged Mode, tagged frames coming with a VLAN tag equal to the
configured native VLAN are dropped.
The Converged Network Adapter (CNA) may lose connectivity to the Brocade 8000/FCOE10-24 if
the CNA interface is toggled repeatedly over time. This issue is related to the CNA and rebooting
the CNA restores connectivity.
Although the Brocade 8000 and FCOE10-24 support the configuration of multiple CEE maps, it is
recommended to use only one CEE map on all interfaces connected to CNAs. Additionally, CEE
maps are not recommended for use with non-FCoE traffic. QoS commands are recommended for
interfaces carrying non-FCoE traffic.
It is recommended that Spanning Tree Protocol and its variants be disabled on CEE interfaces that
are connected to a server.
The Fabric Provided MAC Address (FPMA) and the Fibre Channel Identifier (FCID) assigned to a
VN_Port cannot be associated with any single front-end CEE port on which the FLOGI was
received.
LLDP neighbor information may be released before the timer expires when DCBX is enabled on a
CEE interface. This occurs only when the CEE interface state changes from active to any other
state. When the DCBX is not enabled, the neighbor information is not released until the timer
expires, irrespective of the interface state.
The FCoE Login Group Name should be unique in a fabric wide FCoE Login Management
Configuration. The merge logic is designed to modify the Login Group Name during merge when
Login group names in participating configurations conflict with each other. The current OUI of
00051E is being used by Brocade, while assigning the WWNs to 8000s, DCXs and DCX4Ss, which
would make only the last 3 bytes as different for any two 8000s, DCXs or DCX4Ss. Considering
this assignment method, the merge logic would rename the login group by including the last 3
bytes of WWN in the login group name, so that they are unique in the merged configuration.
For switches having different OUI indices from the 8 assigned to Brocade (for ex: 00051E and
006069), WWNs can differ in more than 3 bytes. In this case, after normal merge and a rename
Ethernet switch services must be explicitly enabled using the command ―fosconfig –enable ethsw”
before powering on an FCOE10-24 blade. Failure to do so will cause the blade to be faulted (fault
9). Users can enable ethsw after upgrading firmware without FC traffic interruption.
The Brocade 8000 does not support non-disruptive hot code loads (HCL). Upgrading the Brocade
8000 to FOS 6.4 or downgrading from v6.4 is disruptive to the IO through the switch.
A code load on a DCX or DCX-4s with one or more FCOE10-24 blades will disrupt the traffic going
through those FCOE10-24 blades.
HA Failover of CP blades in DCX or DCS-4s will also result in disruption of traffic through the
FCOE10-24 blades.
Connecting a Brocade 8000 to an FCR-capable switch with fcrbcast config enabled will cause a
storm of broadcast traffic resulting in termination of iswitchd.
When rebooting a DCX or DCX-4S with an FCOE10-24 blade, Qlogic CNA and LSAN zoning, the
switch will become very unresponsive for a period of time. This is due to the CNA sending
excessive MS queries to the switch.
An FCOE10-24 blade installed in the highest numbered slot of a DCX or DCX-4S chassis does not
send out FIP unsolicited advertisements. Therefore, it does not support FCoE functionality when
installed in this slot.
The Brocade 8000 and FCOE10-24 can handle 169 small FCoE frames in bursts. If you are using
the Brocade 8000 or FCOE10-24, and you delete a large number of v-ports with HCM, some of the
v-ports may not appear to be deleted. To correct this, disable and re-enable FCoE with the
following CLI commands:
switch:admin>fcoe --disable
switch:admin>fcoe --enable
Virtual Fabrics
On Virtual Fabrics capable platforms, the Virtual Fabrics feature must be enabled in order to
utilize the related capabilities including Logical Switches and Logical Fabrics. On units that ship
with FOS v6.3 installed, the Virtual Fabrics feature is enabled by default on capable platforms.
When creating Logical Fabrics that include switches that are not Virtual Fabrics capable, it is
possible to have two Logical Switches with different FIDs in the same fabric connected via a VF
incapable switch. Extra caution should be used to verify the FIDs match for all switches in the
same Logical Fabric.
A switch with Virtual Fabrics enabled may not participate in a fabric that is using Password
Database distribution or Administrative Domains. The Virtual Fabrics feature must be disabled
prior to deploying in a fabric using these features.
Virtual Fabrics is not supported on Brocade 7800.
VF dedicated ISLs are supported on FX8-24 blade. XISLs are not supported.
Encryption Behavior for the Brocade Encryption Switch (BES) and FS8-18
The ―cryptocfg –manual_rekey –all‖ command should not be used in environments with multiple
encryption engines (FS8-18 blades) installed in a director-class chassis when more than one
encryption engine has access to the same LUN. In such situations, use the ―cryptocfg –
manual_rekey <CTC> <LUN Num> <Initiator PWWN>‖ command to manually rekey these LUNs.
When adding Nodes to an Encryption Group, ensure all Node Encryption Engines are in an
Enabled state.
When host clusters are deployed in an Encryption environment, please note the following
recommendations:
o If two EEs (encryption engines) are part of a HAC, configure the host/target pair such that they
form a multipath from both EEs. Avoid connecting both the host/target pairs to the same EE.
This connectivity does not give full redundancy in case of EE failure resulting in HAC failover.
o Since quorum disk plays a vital role in keeping the cluster in sync, please configure the
quorum disk to be outside of the encryption environment.
The ―–key_lifespan‖ option has no effect for ―cryptocfg –add –LUN‖, and only has an effect for
―cryptocfg --create –tapepool‖ for tape pools declared ―-encryption_format native‖. For all other
encryption cases, a new key is generated each time a medium is rewound and block zero is
written or overwritten. For the same reason, the ―Key Life‖ field in the output of ―cryptocfg --show -
container -all –stat‖ should always be ignored, and the ―Key life‖ field in ―cryptocfg --show –
tapepool –cfg‖ is only significant for native-encrypted pools.
The Quorum Authentication feature requires a compatible DCFM release (DCFM 10.3 or later) that
supports this feature. Note, all nodes in the EG must be running FOS v6.3.0 or later for quorum
authentication to be properly supported.
The System Card feature requires a compatible DCFM release that supports this feature. Note,
all nodes in the EG must be running FOS v6.3.0 or later for system verification to be properly
supported.
The Brocade Encryption switch and FS8-18 blade do not support QoS. When using encryption or
Frame Redirection, participating flows should not be included in QoS Zones.
When using Brocade Native Mode, in LKM installations, manual rekey is highly recommended. If
auto rekey is desired, the key expiry date should be configured only when the LUN is created.
Never modify the expiry date after configuring a LUN. If you modify the expiry time, after
configuring the LUN the expiration date will not update properly.
SKM is supported with Multiple Nodes and Dual SKM Key Vaults. Two-way certificate exchange is
supported. Please refer to the Encryption Admin Guide for configuration information. If using dual
SKMs on BES/FS8-18 Encryption Group, then these SKM Appliances must be clustered. Failure
The RKM Appliance A1.6, SW v2.7 is supported. The procedure for setting up the RKM Appliance
with BES or a DCX/DCX-4S with FS8-18 blades is located in the Encryption Admin Guide.
Support for registering a 2nd RKM Appliance on BES/FS8-18 is blocked. If the RKM Appliances
are clustered, then the virtual IP address hosted by a 3rd party IP load balancer for the RKM
Cluster must be registered on BES/FS8-18 in the primary slot for Key Vault IP.
With Windows and Veritas Volume Manager/Veritas Dynamic Multipathing, when LUN sizes less
than 400MB are presented to BES for encryption, a host panic may occur and this configuration is
not supported in the FOS v6.3.1 or later release.
HCL from FOS v6.3.x to v6.4 is supported. Cryptographic operations and I/O will be disrupted but
other layer 2 traffic will not.
Relative to the BES and a DCX with FS8-18, all nodes in the Encryption Group must be at the
same firmware level of FOS v6.2 or later before starting a rekey or First Time Encryption operation.
Make sure that existing rekey or First Time Encryption operations complete before upgrading any
of the encryption products in the Encryption Group. Also, make sure that the upgrade of all nodes
in the Encryption Group completes before starting a rekey or First Time Encryption operation.
To clean up the stale rekey information for the LUN, follow one of the following two methods:
Method 1:
1. First, modify the LUN policy from ―encrypt‖ to ―cleartext‖ and commit. The LUN will
become disabled.
2. Enable the LUN using ―cryptocfg --enable –LUN‖. Modify the LUN policy from ―clear-
text‖ to ―encrypt‖ with ―enable_encexistingdata‖ to enable the first time encryption
and do commit. This will clear the stale rekey metadata on the LUN and the LUN can
be used again for encryption.
Method 2:
1. Remove the LUN from Crypto Target Container and commit.
2. Add the LUN back to the Crypto Target Container with LUN State=‖clear-text‖,
policy=‖encrypt‖ and ―enable_encexistingdata‖ set for enabling the First Time
Encryption and commit. This will clear the stale rekey metadata on the LUN and the
LUN can be used again for encryption.
TEMS key vault support troubleshooting tips:
o Regarding TEMS key vault (KV) communication with a Brocade encryption group, the
default communication port setting for the TEMS KV is 37208, however, the Brocade
encryption members and leader use 9000 so this needs to be reset on NCKA.
Additionally, the following is a checklist of things to review if the initial attempt to connect
to the KV fails:
Check physical and logical connection via a ping on port 9000, this should be the
first check.
For the group leader node, the kac client cert and the kv cert files are to be
identical.
When disk and tape CTCs are hosted on the same encryption engine, re-keying cannot be done
while tape backup or restore operations are running. Re-keying operations must be scheduled at a
time that does not conflict with normal tape I/O operations. The LUNs should not be configured
with auto rekey option when single EE has disk and tape CTCs.
Gatekeeper LUNs used by SYMAPI on the host for configuring SRDF/TF using in-band
management must be added to their containers with LUN state as ―cleartext‖, encryption policy as
―cleartext‖ and without ―-newLUN‖ option.
For new features added to encryption in FOS v6.4.0, such as, disk device decommissioning,
combined disk-tape encryption support on the same encryption engine, and redundant key ID
metadata option for replication environments, all the nodes in the encryption group must be
running FOS v6.4.0 or higher versions of FOS. Firmware downgrade will be prevented from FOS
v6.4.0 to a lower version if one or more of these features are in use.
Special Notes for HP Data Protector backup/restore application
Tape Pool encryption policy specification:
o On Windows Systems, HP Data Protector can be used with tape pool encryption
specification only if the following pool label options are used:
Pick from Barcode
User Supplied – Only 9 characters or less
For other options, behavior defaults to Tape LUN encryption policy.
o On HP-UX systems, HP Data Protector cannot be used with tape pool encryption
specification for any of the pool options. The behavior defaults to Tape LUN
Encryption Policy.
Tape LUN encryption policy specification:
o No restrictions, tape LUN encryption policy specification can be used with HP Data
Protector on HP-UX and Windows systems.
The disk device decommission operation works properly only with Containers (CTC) without
hyphen (-) in the container (CTC) name.
Adaptive Networking/Flow-Based QoS Prioritization
When using QoS in a fabric with 4G ports or switches, FOS v6.0 or later must be installed on all
products in order to pass QoS info. E_Ports from the DCX to other switches must come up AFTER
6.0 is running on those switches.
Flow based QoS is NOT supported on FC8 blades in the Brocade 48000.
Any products that are not capable of operating with FOS 6.0 may NOT exist in a fabric with Flow
based QoS. Major problems will occur if previous generation 2G products exist in the fabric.
Access Gateway
When running Adaptive Networking in AG mode note the following:
o QoS takes precedence over ingress rate limiting
o Ingress Rate Limiting is not enforced on trunked ports
Bottleneck Detection
Due to memory constraints, when using Bottleneck Detection on the Brocade 48000, a maximum
of 100 ports should be configured and enabled for monitoring at any time.
FCR
IPFC over FCR is now disabled by default. Switches that are upgraded to FOS v6.3 will retain their
configuration settings for IPFC over FCR. The change to the default configuration only applies to
new units shipping with FOS v6.3 or units running v6.3 that are reset to a default configuration.
Use fcrbcast - - enable to explicitly enable IPFC over FCR.
Broadcast frame forwarding is not supported in an FCR fabric with a Brocade 8000. By default,
broadcast frame forwarding is disabled on the FC router. If your edge fabric includes a Brocade
8000, do not enable broadcast frame forwarding on the FC router because this can degrade FCR
performance when there is excessive broadcast traffic.
With FC8 blades, the switch must be disabled to change the backbone fabric ID.
With routing and dual backbone fabrics, the backbone fabric ID must be changed to keep the IDs
unique.
When using FC Routing in a backbone to edge configuration with an Mi10K in the edge fabric,
users may experience slow throughput for hosts attached to the Mi10K. Users may encounter
this following a bounced IFL connection between the backbone and edge fabric. This slowdown
can be resolved by disabling/enabling the Mi10K ports for the hosts that are impacted.
Mi10K Directors operating with firmware prior to M-EOSn v9.9.5 may experience repeated system
faults when attached as an FCR edge switch to a Brocade 7800 EX Port. To avoid this, ensure
that the Mi10K is operating with M-EOSn v9.9.5 or later when in an edge fabric that will be
attached to a Brocade 7800 FCR Backbone.
VEX edge to VEX edge device sharing will not be supported.
FC FastWrite
When an FC FastWrite Initiator is moved to a port that doesn't have FC FastWrite enabled, I/O will
recover and revert to the slow path route (non FC FastWrite). This is a behavioral change from
FOS v6.2.x.
Traffic Isolation over FCR
All switches and Fibre Channel Routers both in edge and backbone fabrics must be running FOS
v6.1.0 or later in order to support this feature.
In order for Traffic Isolation over FCR to function properly, the associated TI zones in each fabric
(both edge fabrics and backbone fabric) need to have failover ENABLED.
TI over FCR is only supported in edge-to-edge configurations. There is no support for TI in
backbone to edge routing configurations.
Native Connectivity
FOS-based platforms operating in interopmodes 2 or 3 should never be deployed in a fabric
without at least one M-series switch. FOS switches in interopmode 3 (McDATA Open Fabric Mode)
do not support configuration of zoning without an M-series switch in the fabric. When migrating
from M-series to B-series switches, all B-series switches should be configured to interopmode 0
(Brocade Native mode) once the last M-series switch has been removed from the fabric.
M-EOSc switches may exhibit a behavior where they block all attached devices with a reason
indication of ―Blocked Temporarily, Internal‖. Users that experience this may have power cycled
the M-series switch while it was participating in a fabric with Frame Redirection zoning, a
capability used for FOS-based application or encryption services . If the switch is still participating
in the fabric with Frame Redirection, issue the ―cfgsave‖ command from a Brocade FOS-based
switch with the Frame Redirection zone in its defined zone database. If the M-EOS switch is no
longer attached to the fabric with Frame Redirection zoning, issue the
―Config.Zoning.deleteSplZoneSet‖ command via CLI to the M-EOS switch.
FCAP
If VF is enabled on a switch, HTTPS and FCAP certificates should always be imported in the Default
Switch. Certificates imported in a non-Default Switch will not be available after hafailover
operation.
The pkicert (1.06) utility may cause evm errors, so each new switch should be isolated from the
fabric and placed in non-vf mode to install new certificates.
FICON
Refer to Appendix: Additional Considerations for FICON Environments for details and notes for
deployment in FICON environments.
This behavioral change in saved zone alias WWN members will not impact most environments.
However, in a scenario where a switch with a zone alias WWN member with upper case characters
(saved on the switch with pre-FOS v6.2.0 code) is merged with a switch with the same alias
member WWN in lower case characters, the merge will fail, since the switches do not recognize
these zoning configurations as being the same.
For additional details and workaround solutions, please refer to the latest FOS Admin Guide
updates or contact Brocade Customer Support.
ICLs
If a DCX with an 8-link ICL license is connected to a DCX with a 16-link license, the DCX with the
16-link license will report enc_out errors. The errors are harmless, but will continue to increment.
These errors will not be reported if a DCX with a 16-link license is connected to a DCX-4S with only
8-link ICL ports.
If ICL ports are disabled on only one side of an ICL link, the enabled side may see enc_out errors .
A switch running FOS v6.4.0 in InteropMode 0 (Brocade Native Mode) cannot connect to an EX_Port on an
AP7420. A switch running FOS v6.4.0 in InteropMode 2 or InteropMode 3 can be connected to EX ports on an
AP7420.
The fabric parameter ―fabric.ops.mode.longdistance‖ is now deprecated and should not be used.
Although this setting only affects devices logged in at 8G, changing the mode is disruptive
regardless of the speed the port is operating at. The setting is retained and applied any
time an 8G device logs in. Upgrades to FOS v6.3.1 or v6.4 from prior releases supporting
only modes 0 and 1 will not change the existing setting, but switches or ports reset to
factory defaults with FOS v6.3.1 or v6.4 will be configured to Mode 0 by default. The
default setting on new units may vary by vendor. Please use portcfgshow CLI to view the
current portcfgfillword status for that port.
Modes 2 and 3 are compliant with FC-FS-3 specifications (standards specify the
IDLE/ARBF behavior of Mode 2 which is used by Mode 3 if ARBF/ARBF fails after 3
attempts). For most environments, Brocade recommends using Mode 3, as it provides
more flexibility and compatibility with a wide range of devices. In the event that the
default setting or Mode 3 does not work with a particular device, contact your switch
vendor for further assistance.
For the configure command, in FOS v6.4.0, the default value that displays for Maximum Logins per
switch is different than the value that displays in FOS v6.3.x. The default value has not changed; it
was displayed incorrectly in FOS v6.3.x, and is now corrected.
Area Comments
FCIP VEX ports are not supported on the 7800 and FX8-24 blade in a FICON environment
FCIP When performing multiple cabling changes to the SAN fabric in a FICON Emulating FCIP
Tunnel configuration with the Brocade 7800 or FX8-24 blade, either disable all of the
FCIP Tunnels or issue the switch disable command on all FCIP interconnected switches
to avoid IFCCs in a mainframe environment. Issuing either a switch disable or an FCIP
Tunnel disable command will allow the FCIP FICON Emulation processing state-machine
to execute an orderly cleanup process and allow normal activation of the new
configuration. When all cabling and Traffic Isolation Zone manipulations have been
completed, enable the switches or the FCIP Tunnels.
Firmware Non-disruptive Hot Code Load is only supported on director class switches (48000,
Downloads DCX, and DCX-4S). Comprehensive non-disruptive Hot Code Load is not supported on
the 7500 or 7800 or a DCX, DCX-4S or 48000 with an FR4-18i or FX8-24 blades since
the FCIP tunnels will go down for 10-15 seconds and all traffic in the tunnels will be
disrupted.. IFCCs may result if traffic is not stopped while downloading firmware.
Firmware Replacement of a CP card in the Brocade 48000 may cause disruption of I/O traffic.
Downloads Brocade recommends that the CP be replaced during a scheduled downtime to prevent
disruption in FICON environments.
Firmware The CUP device must be varied offline to all MVS partitions before starting a code load.
Downloads The CUP device can be varied back online after the code load completes. Failure to vary
off the CUP devices may result in missing interrupt.
Interoperability When connecting an 8G capable port in a Brocade switch to an IBM Virtualization
Engine TS7700, the port must be configured to a minimum of 16 buffers to avoid
IFCCs at the channel and loss of FICON paths to the control unit. This requires the
Extended Fabric license on the Brocade switch.
Manageability In a mixed fabric environment, an M-EOS switch must be principal switch if the fabric is
in Interopmode 2 (McDATA Fabric Mode).
Criteria Value
ITW (Invalid Transmission Words) 25
CRC (Cyclical Redundancy Check) 3
Protocol Errors 2
State Change 7
Note: In a FICON environment, the time base polling interval MUST be set to one minute
for granular control and response. By default, Port Fencing time base is set to one
hour.
Manageability Firmware download is executed sequentially if ECFM is used for downloading code to
FOS switches.
Manageability As a "Best Practice" for deploying FOS switches/directors into a FICON environment,
verify the FOS version shipped with the most current FOS recommendation. It is
recommended to update all FOS switch/directors to the same FOS levels for
production.
Manageability The remote CUP may not work when the channel is connected to an 8G blade on a
48000 cascaded to a remote switch.
Manageability FMS must be enabled on the local switch for the remote CUP to work.
Optics Brocade recommends using 50 micron multimode fiber optic cabling rated at 2000
MHz-km (OM3 fiber) for connecting to 8 Gb/sec short wavelength (SX) small form factor
pluggable optics (SFPs). Other 50 micron and 62.5 micron multimode fiber may be
used as an alternative, but distance limitations may exist.
Serviceability Performance of optical links depends upon the cleanliness of the cables and
connectors, especially at 8 Gb/sec or higher speeds. Consult with your switch and cable
vendors for proper cable maintenance.
Serviceability The 48 port blade (FC8-48) is supported as follows:
• The switch, or logical switch, must be configured for Brocade Native mode
(interopmode 0).
• It is only supported on VF enabled chassis on the DCX.
• It is not supported in the default switch on the DCX.
Serviceability When the mainframe goes through a resetting event (ConfIg POR, IPL, POR), in rare
instances, some of the ports may come up in an "Invalid Attach" state. To recover these
ports, vary the CHPIDs offline and back online. This is most likely caused by other ports
on the system experiencing link level errors and should be debugged accordingly.
Traffic Isolation Enable Lossless DLS or Lossless DPS when activating Traffic Isolation (TI) Zones to
Zones avoid any traffic disruption.
Traffic Isolation Traffic Isolation (TI) Zoning with FICON supports enabling or disabling of the failover
Zones option. Assistance from service support should be sought before attempting to enable
this feature.
Traffic Isolation Deactivating TI Zone with failover disabled may caused IFCC's. Enable failover prior to
Zones deactivating TI Zone to avoid IFCCs.
Interoperability
Within a fabric, current major releases will work with previous major releases on the same platform. When
cascading switches, it is recommended to keep all switches in the fabric at the same code level. Although not
expressly prohibited, having two switches in the same fabric that differ by more than one major FOS release
level is not recommended. For example, a switch at FOS v6.4.0a connected to another switch at v6.3.0d is OK.
Connecting a switch running FOS v6.4.0a to a switch running FOS v6.1.0a is not recommended.
The following table indicates supported intra-fabric interoperability between hardware platforms, supported
management software levels, and recommended firmware versions.
DCX/
10.4.1 NS S S S 1,4 S S S S
DCX-4S
48000 10.4.1 NS S NS S S S S
M6140/
10.4.1 9.7.4 S S1 NS S1 S1
Mi10K
5100/ 10.4.1 NS S S S S
5300
4100/
4900/ 10.4.1 9.7.4 S NS NS
50002
7500/
10.4.1 NS S NS
7500E3
7800 10.4.1 NS S
Table Notes:
S=Supported
NS = Not Supported
FR4-18i Extension blade is only interoperable with 7500 or 7500E extension switches.
FX8-24 Extension blade is only interoperable with 7800 extension switch.
FC8-64, FS8-18, FCOE10-24 and FA4-18i are not supported in a FICON environment.