FabricPath Operation and

Troubleshooting
Carlo Schmidt, Customer Support Engineer
BRKDCT-3313
Acronyms / Definitions
Acronyms Definitions Acronyms Definitions
ACL Access Control List FP FabricPath
ASIC Application Specific Integrated Circuit FTAG Forwarding Tag
ASID Anycast Switch Identifier LID Local Identifier
BD Bridge Domain LTL Local Target Logic
CE Classical Ethernet MIM MAC-in-MAC (common reference to FP
header)
DBUS / RBUS Data Bus / Result Bus PACL Port-based ACL
DRAP Dynamic Resource Allocation Protocol RACL Router-based ACL
DSID Destination Switch Identifier RPF Reverse Path Forwarding
ELAM Embedded Logic Analyzer Module SoC Switch-On-Chip
ES Emulated Switch SSID Source Switch Identifier
FE Forwarding Engine VACL Vlan-based ACL
FF Flood to Fabric VDC Virtual Device Context

 Reference Slide
Agenda
• FabricPath Overview
Benefits, Restrictions, and Configuration
• Key Concepts
Encapsulation, Trees, Topologies, STP
• Data Plane
Forwarding, Load-Balancing, MAC Learning
• vPC+
Challenges and Operation
• Troubleshooting
Verification steps, tools, and examples
 FabricPath Benefits
Single path between 2 points in L2 network
Existing • Stability/Resilience at scale
• Disruptive convergence
Layer2

Shortest path between switches + equal-cost load-balancing

• Core does not need to learn end host MAC addresses
FabricPath • More resilient to loops
• No topology constraints, L3 anywhere
• Easy scaling / Non-disruptive merge
Fabricpath Overview
Unicast: Known Destination MAC
Ingress SSID comes DSID comes from Intermediate TTL Egress
FabricPath from S10’s MAC address switches forward decremented at FabricPath
(Edge) Switch own switchID table for MAC B based on DSID every FP switch (Edge) Switch

DSID 20
SSID 10 DMAC B
DMAC B SMAC A
SMAC A Payload
DMAC B
Payload
SMAC A
Payload

MAC A MAC B

CE FabricPath CE
FabricPath Overview
Multidestination (broadcast, multicast, unicast flood)
Ingress FP SSID comes MAC B is
Root switch Root switch
Switch selects from S10’s unknown DSID
for Tree 1 for Tree 2
Tree (FTAG) own switchID = FloodSID → FabricPath interface

DMAC B → CE interface
DMAC B SSID
SSID FTAG 1 → Tree 1
FTAG 1 DMAC B
DMAC B SMAC A DMAC B → Tree 2
SMAC A Payload SMAC A
DMAC B
Payload Payload
SMAC A
Payload

MAC A MAC B

CE FabricPath CE
FabricPath support & configuration
• N7K with N7K-F1 linecard as of 5.1.1
N7K(config)# install feature-set fabricpath
• N7K with N7K-F2 linecard as of 6.0.1
N7K(config)# feature-set fabricpath
• N7K + FEX as of 6.1.1 (with N7K-F2) for CE N7K(config)# interface Ethernet4/1
ports N7K(config-if)# switchport mode fabricpath
• F2E as of 6.1.2 ...

• N7K with N7K-F3 linecard as of 6.2.6 N7K(config)# vlan 3002

N7K(config-vlan)# mode fabricpath
• N5500 as of 5.1.3
• no L3 module required
• N5500 + FEX as of 5.1.3 for CE ports
• N6K as of 6.0.2
• Enhanced L2 license required FabricPath
• Packaged as feature-set (plugin)
FabricPath & CE Vlans
FabricPath
• Two types of vlans
 CE (Classic Ethernet, default) Classic Ethernet
 FabricPath (FP)

• FP vlans cannot go on M1, M2 modules

• Only FP vlans will be carried over FP interfaces
Core = switchport mode fabricpath
• FP vlans can be mixed with CE vlans on edge interfaces Edge = switchport mode access || trunk

Port Type VLANs allowed VLANs allowed to

N7K(config)# vlan 3002 to be configured be brought up
N7K(config-vlan)# mode ? N7K-M1, N7K-M2 FP, CE CE
ce Classical Ethernet VLAN mode N7K-F1, N7K-F2, N7K-F3 Edge FP, CE FP, CE

fabricpath Fabricpath VLAN mode N7K-F1, N7K-F2, N7K-F3 Core FP, CE FP

N5500, N6000 Edge FP, CE FP, CE
N5500, N6000 Core FP, CE FP
Agenda
• FabricPath Overview
Benefits, Restrictions, and Configuration
• Key Concepts
Encapsulation, Trees, Topologies, STP
• Data Plane
Forwarding, Load-Balancing, MAC Learning
• vPC+
Challenges and Operation
• Troubleshooting
Verification steps, tools, and examples
 Outer Outer FP
CRC
Encapsulation DA
(48)
SA
(48)
Tag
(32)
DMAC SMAC 802.1Q Etype Payload
(new)

47 0
Endnode Endnode R O SubSwitch
U/L I/G S O Switch ID
ID ID
V O ID Local ID
[ 5:0] [ 7:6] D

6 1 1 2 1 1 12 bits 8 bits 16 bits

Outer SA:
 SwitchID ingress FP switch system ID
N7K# show fabricpath switch-id | include SYS|\*
 SubswitchID is used in some cases of VPC+
Legend: '*' - this system
 LID is specific to the implementation
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED
• N7K the LID is generally the port index of the ingress interface
*2028 b414.89e3.a041 Primary Confirmed No No
• N5K/N6K LID most of the time will be 0
• EndnodeID is not currently used
N7K# sh mac address-table address 0000.1234.5678
Outer DA:
VLAN MAC Address Type age Ports/SWID.SSID.LID
 For known SA/DA is taken from MAC table for DMAC
---------+-----------------+--------+---------+------------------
 For broadcast and multicast is the same as DMAC 3000 0000.1234.5678 dynamic 0 2.0.1054
 For unknown unicast DA is 010f.ffc1.01c0 (flood to vlan)
 For known unicast DA, but unknown SA is 010f.ffc1.02c0 (flood to fabric) Switch_ID SubSwitch_ID LID
Example 100 1 65535
Ethernet II, Src: 02:00:64:01:FF:FF, Dst: 01:00:5e:00:00:02, Type: 0x8903
FabricPath Switch IDs, System IDs … and DRAP
• Each FP switch is identified by unique number (ID), dynamically assigned or static

• Dynamic Resource Allocation Protocol (DRAP) is responsible for allocating switch IDs and resolving
duplicate-ID conflicts. Conflicts are resolved by renumbering switches with higher systemID
(DRAP can only auto resolve non-static switch ID)
N7K# show fabricpath switch-id
1 2 3
FABRICPATH SWITCH-ID TABLE
=========================================================================
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED
+
----------+----------------+------------+-----------+--------------------
*3 c062.6bac.e343 Primary Confirmed Yes No
4 5
30 547f.ee02.ce3c Primary Confirmed Yes No
3
40 547f.ee04.5cfc Primary Confirmed Yes No

• When partitioned FP network is merged (or new switch joins the fabric) connecting interface is not =
enabled for data before all conflicts are resolved
N7K# show fabricpath conflict all

N7K(config-if-range)# no shut
Port State
---------------+------------------------ 1 2 3
Ethernet3/31 Suspended due to conflicts
%FABRICPATH-2-
============================================== 4 5
FABRICPATH_LINK_BRINGUP_STALLED_STATIC: Link
Fabricpath Conflicts
6
bringup stalled due to conflicts
SYSTEM-ID SWITCH-ID STATIC
---------------+--------------+---------------
c062.6bac.e343 3 Yes
c062.6bac.e342 3 Yes
 Network Merges / Conflict resolution
• Goal is to connect two networks with conflicting switch IDs
without incurring packet loss N7k# show fabricpath switch-id
1) Allocate new switch-id as secondary – tentative Legend: '*' - this system
• Wait allocate delay time SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED
2) Make new switch-id as secondary - confirmed ----------+----------------+------------+-----------+--------------------
• Wait transition delay time *332 b414.89e3.a042 Primary Confirmed Yes No
3) Swap primary and secondary switch-ids
N7k# show fabricpath isis switch-id
• Wait transition delay time
Legend: C - Confirmed, T - tentative, W - swap
4) Delete old switch-id (now a secondary switch-id) S - sticky, E - Emulated Switch
'*' - this system
More About Graceful Merge System-ID Primary Secondary Reachable Bcast-Priority
MT-0
b414.89e3.a042* 332 [C] 0[C] Yes 222 [S]
Graceful merge changes the switch-id of a switch to
resolve switch-id collisions N7k# show fabricpath timers
Allocate Delay Timer : 10
The switch-id to change is based on the system-id Transition Delay Timer : 10
being higher value, or being dynamic Link-up Delay Timer : 10

For a time period the switch is identified by two switch-

ids, packets for both are accepted but outgoing packets
only carry the primary switch-id
FabricPath Trees → FabricPath interface
• Known unicast traffic is load-balanced across equal-cost → Tree 1
routes → Tree 2

• FabricPath uses two loop-free trees for unknown unicast,

broadcast and multicast traffic
• Two trees are for load-balancing S1
• For each packet, tree is selected by ingress FP switch and choice is carried SysID 50

in the packet header S2

R
S3
• Root of tree1 is the switch with highest Priority (highest sysID for tie) SysID 10 SysID 20
Lower SysID wins
• Root of tree2 is the switch with 2nd highest Priority (highest sysID for tie)
• Tree is a least-cost-to-the-root graph, with lower sysID used as tie-breaker
• In case of Tree1 root failure both roots are reelected
• Up to 16 trees starting in 7.0 on Nexus 5000 and 6000 S4
SysID 30
Root Election / Tree construction
S101# show fabricpath isis database detail
• Every switch advertises its system ID Fabricpath IS-IS domain: default LSP database
LSPID Seq Number Checksum Lifetime A/P/O/T
& Priority S1.00-00 0x000000E2 0x0FBB 1054 0/0/0/1
• Once all nodes have spoken Instance : 0x000000DD
Area Address : 00
Broadcast Root is elected (Highest NLPID : 0xC0
priority then Highest Mac address Hostname : S1 Length : 2
Extended IS : S202.00 Metric : 40
wins) Extended IS : S101.00 Metric : 40
• Broadcast root system will Elect & Extended IS : S102.00 Metric : 40
Extended IS : S2.00 Metric : 40
Advertise Roots for additional Extended IS : S201.00 Metric : 40
multicast Trees (currently only 2 trees) Capability : Device Id: 1 Base Topology
Base Topo Ftag :
• Each node will independently run SPF Graph 1: Root: S1 Primary: 1, Secondary: 0 Nickname 1
with Tree Root and create 2 Trees Graph 2: Root: S2 Primary: 2, Secondary: 0 Nickname 2
Base Topo Trees :
• Since Multicast roots are advertised Trees desired: 2 Trees computed: 2 Trees usable: 2
by Broadcast Root system (Tree 1), in Base Topo Roots :
case of failure of the latter both Tree 1 Graph 1: Root Nickname: 1
Graph 2: Root Nickname: 2
and Tree 2 will re-converge Version :
Version: 1 Flags: 0
Nickname :
Priority: 0 Nickname: 1 BcastPriority: 255
Nickname Migration :
Swid: 1 Sec. Swid: 0
 Outer Outer FP
CRC
Encapsulation DA
(48)
SA
(48)
Tag
(32)
DMAC SMAC 802.1Q Etype Payload
(new)

Ethertype 0x8903 FTAG TTL

16 bits 10 bits 6 bits

• Ethertype for FabricPath packets is 0x8903
• TTL set to 32 and is decremented at every hop. Packet is discarded when TTL reaches 0.
• FTAG: (Forwarding TAG) Used for multidestination traffic; carries the ID of the tree chosen at the
FabricPath ingress switch. DRAP is responsible to keep FTAGs unique/consistent. For known unicast,
FTAG carries topology ID
Nexus# show fabricpath isis topology summary
Fabricpath IS-IS domain: default FabricPath IS-IS Topology Summary
MT-0
Configured interfaces: Ethernet4/4
Number of trees: 2
Root for Tree 1, FTAG 1
Tree id: 1, ftag: 1, root system: 001b.54c2.4244, 4
Tree id: 2, ftag: 2, root system: 001b.54c2.4243, 3 Root for Tree 2, FTAG 2

Wireshark decodes FP encapsulation (tested on 1.8.3) : EditPreferencesProtocolsCFPEnable Dissector

 1 2 Accept packets from 1,4
root
Reverse Path Forwarding Check Accept packets from 3

• RPF: check where the source switch of the packet is Accept packets from 4,1,2
and only accept packets from the interface we would
have used if we were to send packet to that source 4 3
• At each FP hop RPF check is performed for multidestination traffic against
source switchID + FTAG May also use
N7K# show l2 multicast trees show fabricpath isis trees

(ftag/2, topo/0, Switch-id 40), uptime: 1w0d, isis

Outgoing interface list: (count: 1, '*' is the preferred interface)
* Interface Ethernet3/39, [admin distance/115] uptime: 1d23h, isis
Packets with FTAG==2 from
switch 30 will be accepted from (ftag/2, topo/0, Switch-id 30), uptime: 1w0d, isis
interface e3/35 Outgoing interface list: (count: 1, '*' is the preferred interface)
* Interface Ethernet3/35, [admin distance/115] uptime: 02:56:04, isis

(ftag/2, topo/0, Switch-id 100), uptime: 1w0d, isis

Outgoing interface list: (count: 1, '*' is the preferred interface)
* Interface Ethernet3/39, [admin distance/115] uptime: 1d23h, isis
Packets with FTAG==1 from
switch 30 will be accepted from (ftag/1, topo/0, Switch-id 30), uptime: 02:56:06, isis
interface e3/35 Outgoing interface list: (count: 1, '*' is the preferred interface)
* Interface Ethernet3/35, [admin distance/115] uptime: 02:56:06, isis
 Topologies Default Topology allowed
• Routing table & Trees (FTAGs) are per topology on all FP links

• Switch ID is shared across all topologies

• FP interface may belong to several topologies FP links in Topology 0

and Topology 1
• N7K: up to 8 topologies support starting in 6.2
Pod 1 Pod 2
• N5K/N6K: As of 5.2.1 default + 1 extra topology is supported;
Vlan 100-199 Vlan 200-299
main use is to permit separate L2 pods to use same local vlan
Vlan 1000-1099 Vlan 1000-1099
set
N7K# show fabricpath topology vlan
Topo-Description Topo-ID Configured VLAN List fabricpath topology 1
-------------------------------- --------- ------------------------------------- member vlan 100-199
0 0 1-99, 200-4095 !
1 1 100-199 interface Port-channel1
switchport mode fabricpath
N7K# show fabricpath topology interface fabricpath topology 1
Interface Topo-Description Topo-ID Topo-IF-State
------------------- -------------------------------- ---------- -------------
port-channel1 0 0 Up
Ethernet6/4 0 0 Up
Ethernet6/5 0 0 Up
port-channel1 1 1 Up
Topologies + Vlans
• Flood/Multicast/Broadcast trees are per-vlan, made by pruning Topology Tree
• If vlan is not present on the switch, that switch will not be part of per-vlan tree
• This may lead to connectivity issues when not all transit switches in topology have all vlans
• similar to connectivity issues caused by liberal pruning vlans off trunks with MST

• Make sure each vlan exists in every transit switch in a topology

VL10 VLAN 10 VLAN 20 VLAN 30

Topology Tree VL30

VL10
VL10
VL20
VL20
VL30
FabricPath Software Architecture & Hardware tables
Supervisor
DRAP
on the Supervisor Engine: Engine

FabricPath IS-IS
• FabricPath IS-IS routing protocol process that forms the core of the FabricPath
control plane U2RIB L2FM

• DRAP Dynamic Resource Allocation Protocol, ensures network-wide unique and

consistent Switch IDs and FTAGs
U2FIB MTM
• Resolves switch id conflicts

Hardware Drivers
• U2RIB Unicast Layer 2 RIB, containing the “best” unicast Layer 2 routing
information
Switch Table Other HW MAC Table
• L2FM Layer 2 forwarding manager, controls MAC address table I/O Module

on the Linecards:
• U2FIB – Unicast Layer 2 FIB, managing the hardware unicast routing table

• MTM – MAC Table Manager, managing the hardware MAC address table
Fabric Path Control Plane initialization flow
S101# show processes cpu | egrep "2rib|drap|fab|l2fm|PID"
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process Processes start
9169 750 16723 0 0.00% 0.00% 0.00% - l2fm (isis, u2rib, m2rib, drap)
9215 1050 7843 0 0.00% 0.00% 0.00% - m2rib
9555 1050 36161 0 0.00% 0.00% 0.00% - u2rib System ID obtained
9556 14740 163944 0 0.00% 0.00% 0.00% - isis_fabricpath from backplane MAC
9557 820 31339 0 0.00% 0.00% 0.00% - drap

---------------------------------------------------------------------------- Switch ID is obtained from DRAP

S101# show fabricpath isis
Fabricpath IS-IS domain : default
System ID : 8478.ac0e.4743 IS-Type : L1 Fabric-Control SVI: Unknown As FP interfaces links come up,
hellos sent and adjacencies formed
...
Process is up and running
... Switch ID conflicts (if any) resolved
Interfaces supported by Fabricpath IS-IS :
port-channel1
Ethernet6/27 FP Interfaces allowed to forward
Ethernet6/28 data
----------------------------------------------------------------------------
S101# show fabricpath switch-id
Legend: '*' - this system Unicast SPF is calculated
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED/
ANYCAST
--------------+----------------+------------+-----------+-------------------- Routes installed to U2RIB
* 101 8478.ac0e.4743 Primary Confirmed Yes No
...
Fabric Path Control Plane initialization flow
S101# show fabricpath isis interface
Fabricpath IS-IS domain: default
Interface: port-channel1 Processes start
Status: protocol-up/link-up/admin-up (isis, u2rib, m2rib, drap)
…
LSP interval: 33 ms, MTU: 1500 System ID obtained
P2P Adjs: 1, AdjsUp: 1, Priority 64 from backplane MAC
Hello Interval: 10, Multi: 3, Next IIH: 00:00:03
Level Adjs AdjsUp Metric CSNP Next CSNP Last LSP ID
1 1 1 40 60 Inactive ffff.ffff.ffff.ff-ff Switch ID is obtained from DRAP
Topologies enabled:
Level Topology Metric MetricConfig Forwarding
0 0 4000 no UP As FP interfaces links come up,
1 0 40 no UP ------------------------------------------- hellos sent and adjacencies formed
---------------------------------
S101# show fabricpath isis adjacency Switch ID conflicts (if any) resolved
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID SNPA Level State Hold Time Interface
S102 N/A 1 UP 00:00:25 port-channel1 FP Interfaces allowed to forward
S1 N/A 1 UP 00:00:28 Ethernet6/27 data
S2 N/A 1 UP 00:00:27 Ethernet6/28
----------------------------------------------------------------------------
S101# show fabricpath isis spf-log Unicast SPF is calculated
Fabricpath IS-IS domain: default SPF information
SPF log for Topology 0
Total number of SPF calculations: 55 Routes installed to U2RIB
Log entry (current/max): 20/20
Ago Level Reason Count Total
1d09h 1 New LSP S201.00-00 3 0.001141
1d09h 1 Updated LSP S2.00-00 2 0.000965
…
 Fabric Path Control Plane initialization flow
S101# show fabricpath isis route
Processes start
Fabricpath IS-IS domain: default MT-0 (isis, u2rib, m2rib, drap)
Topology 0, Tree 0, Swid routing table
1, L1
via Ethernet6/27, metric 40 System ID obtained
2, L1 from backplane MAC
via Ethernet6/28, metric 40
200, L1 Switch ID is obtained from DRAP
via Ethernet6/27, metric 80
via Ethernet6/28, metric 80 How to read
... To reach switch 200 in topology 1 As FP interfaces links come up,
send packets to either Eth6/27 or hellos sent and adjacencies formed
----------------------------------------------------------------------------
Eth6/28
S101# show fabricpath route
FabricPath Unicast Route Table Switch ID conflicts (if any) resolved
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric] FP Interfaces allowed to forward
ftag 0 is local ftag data
FabricPath Unicast Route Table for Topology-Default
...
1/102/0, number of next-hops: 1 Unicast SPF is calculated
via Po1, [115/40], 1 day/s 10:01:12, isis_fabricpath-default
1/200/0, number of next-hops: 2
via Eth6/27, [115/80], 1 day/s 10:02:32, isis_fabricpath-default Routes installed to U2RIB
via Eth6/28, [115/80], 0 day/s 10:20:17, isis_fabricpath-default
FabricPath IP Multicast
• Control plane:
• IGMP snooping operates as usual in FabricPath edge switches
• FabricPath IS-IS learns multicast group membership from IGMP snooping on edge
switch
• FabricPath edge switch announces group interest by using GM-LSPs, creating “pruned
trees” for each group on each multidestination tree
• Data plane:
• Hardware selects which multidestination tree to use for each flow based on hash
function
• Once tree is selected, traffic constrained to pruned tree (FTAG) for that IP multicast
group, based on MAC table lookup
Key FabricPath Multicast Processes
on the Supervisor Engine: DRAP
Supervisor
Engine
• FabricPath IS-IS routing protocol that forms the core of the FabricPath
control plane FabricPath IS-IS IGMP

• DRAP Dynamic Resource Allocation Protocol, extension to FabricPath IS- M2RIB L2FM
IS that ensures network-wide unique and consistent Switch IDs and
FTAGs MFDM
• IGMP Provides IGMP snooping support for building multicast forwarding
database
• M2RIB Multicast Layer 2 RIB, contains the multicast Layer 2 routing M2FIB MTM
information
Hardware Drivers
• L2FM Layer 2 forwarding manager, controls the MAC address table
• MFDM Multicast forwarding distribution manager, connects platform- Switch Table Other HW MAC Table
independent control-plane processes and platform-specific processes on I/O Module
I/O modules
on the Linecards:
• M2FIB – Multicast Layer 2 FIB, manages the hardware multicast routing
table
• MTM – MAC table manager, manages the hardware MAC address table
 S1 S2

FabricPath
•
Multicast Control Plane
IGMP/IGMP snooping tracks connected hosts/routers interest in
S10 S30

receiving multicast
S20
Receiver Receiver
• ISIS distributes information from igmp snooping to other FP nodes 239.1.2.3 239.1.2.3
using GM-LSPs. Intermediate nodes flood GM-LSPs
Source
• A pruned subtree is created for each group (+flood, OMF) per vlan per FTAG 239.1.2.3

Vlan FTAG MAC Switches Interfaces Vlan FTAG MAC Switches Interfaces
1 1 0100.5e01.0203 S10,S30 E1/10,E1/30 1 1 0100.5e01.0203 S10,S30 E1/1
1 2 0100.5e01.0203 S10,S30 E1/2 1 2 0100.5e01.0203 S10,S30 E1/10,E1/30

Root S1 S2 S1 S2 Root
Tree1 E1/2 E1/1 Tree2
E1/10 E1/30 E1/10 E1/30
S10 S30 S30
S10

S20 S20
MAC A E1/1 MAC B MAC A E1/2 MAC B

Vlan FTAG MAC Switches Interfaces Vlan FTAG MAC Switches Interfaces
1 1 0100.5e01.0203 S10,S30 E1/1 1 2 0100.5e01.0203 S10,S30 E1/2
STP & FabricPath
• No STP inside FP network
• BPDUs do not traverse FP network
(dropped at FP edge, with the exception of TCNs, see next slide)
• FP network pretends to be 1 switch from STP point of view: all FP edge FabricPath
switches send BPDUs with the same Bridge ID c84c.75fa.60xx (xx is domain ID
in hex, default 00)
• Before FP ports are up, switch will use its own Bridge ID
(like STP without FP would do)
• Ports inside FP cannot be blocked, FP edge switches will always want to have
STP designated role, if superior BPDU is received such port will be blocked as
L2GW inconsistent
N7K# show spanning-tree interface e3/1 detail
Port 385 (Ethernet3/1) of VLAN2000 is broken (L2 Gateway Backbone Port Inconsistent)
Designated root has priority 34768, address c84c.75fa.6000
…
N7K(config)# spanning-tree vlan 2000 priority 8192
22:27:28 %STP-2-L2GW_BACKBONE_UNBLOCK:
L2 Gateway Backbone port inconsistency cleared unblocking port Ethernet3/1 on VLAN2000.
STP, FabricPath & TCNs
Flush MACs learned from
• When CE STP domains are connected to multiple FP switches STP S4,S5
TCN handling might be needed to maintain accuracy of MAC address
tables inside CE
• Example if link CE1-CE2 goes down, link CE2-CE3 will become forwarding.
Now to reach MAC B, switches inside FP need to send traffic to S5 instead
of S4… S1 STP Domain 1 S3

• To achieve this, FP switches when receiving a TCN from CE will propagate S3 T

C
T
C

it to all FP switches in the network (via ISIS) N

N
FabricPath
N

C
N
T
C

• Each FP switch will flush all remote MAC addresses learned from switches T

MAC A
N
in the same STP domain as domain originating the TCN C T
S4 T CS5
• In addition, if FP switch is also part of the same STP domain, it will STP Domain 2 N
propagate TCN to the CE domain X
• TCNs are not propagated to CE in domain 0 (default domain) CE1 CE2 CE3
MAC B

N7K# conf t
N7K(config)# spanning-tree domain ?
<1-1023> Domain Identifier
N7K# sh spanning-tree summary Flush MACs learned on CE
Switch is in rapid-pvst mode
L2 Gateway Domain ID: 100
...
Control Plane Protection
• Both N7K, N6K, and N5K recognize and protect FP ISIS traffic at COPP level

• COPP needs to be updated when deploying FabricPath; standard profiles are FP-aware as of 5.2(1)
N7K# show policy-map interface control-plane N5K# show policy-map interface control-plane class
Control Plane 7K copp-system-class-isis
service-policy input: copp-policy-strict
class-map copp-class-critical (match-any) Control Plane
… service-policy input: copp-system-policy-default
match access-group name class-map copp-system-class-isis (match-any)
copp-acl-mac-fabricpath-isis match protocol isis_dce
police cir 1024 kbps , bc 4800000 bytes 5K
…
set cos 7 conformed 751957 bytes; action: transmit
police cir 39600 kbps , bc 250 ms violated 0 bytes; 6K
module 1 :
conformed 5136527710 bytes; action: transmit
violated 0 bytes; action: drop

• In case of complex CE-side STP topologies (with blocking ports), usual STP safeguards are recommended
(Bridge Assurance & Dispute / UDLD)

• On N7K-F1 cards: rate-limiters allow up to 4500 PPS worth of control plane FabricPath packets

Note: These 4500 PPS include also transit packets

Agenda
• FabricPath Overview
Benefits, Restrictions, and Configuration
• Key Concepts
Encapsulation, Trees, Topologies, STP
• Data Plane
Forwarding, Load-Balancing, MAC Learning
• vPC+
Challenges and Operation
• Troubleshooting
Verification steps, tools, and examples
FabricPath: Forwarding Tables
Supervisor
• FabricPath uses 3 tables to forward frames DRAP
Engine

FabricPath IS-IS
• MAC address table
U2RIB L2FM
VLAN, MAC Address, Port (local or remote),
FTAG (for non-unicast)
U2FIB MTM
• Switch-ID table Hardware Drivers
remote switch-ID, local next-hop interfaces (up to 16)
Switch Table Other HW MAC Table

• Multidestination tree table I/O Module

Per Tree: remote switch-ID, local next-hop/RPF interface

Tree#1 (broadcast, unknown unicast, IP multicast)
Tree#2 (IP multicast)
Forwarding: unicast CEFP This is meant to illustrate key decisions
in forwarding, some details are
abstracted away
unicast

Unknown unicast
DA N ODA = MC1 (Flood2BD)
Known

DA = Destination Address
Y
SA = Source Address
Unknown source  Flood to update MACs
SA ODA = Outer Destination Address
ODA = MC2 (FF)
Known
N OSA = Outer Source Address
MC1 = 010F.FFC1.01C0
Y
ODA = L2_lookup (DA) MC2 = 010F.FFC1.02C0

Ftag == Vlan2Ftag(Vlan)
Choose FTAG
FTAG for unicast Ftag = F(Vlan,SA/DA,…)
is topology ID
TTL = 32

OSA.SW/SubSW = local
OSA.LID=LID(ingress_port)

Forward
Forwarding: broadcast/multicast CEFP
BC || MC

ODA = DA

Ftag = Hash(Vlan,SA/DA,…) Broadcasts are flooded along FTAG1

* Exception in vPC+
TTL = 32

OSA.SW/SubSW = local
OSA.LID=LID(ingress_port)

Forward Frame is flooded on CE side as well (based on DA)

Each egress port decides whether to encapsulate the frame in MIM
depending on port type (FP,CE)
Forwarding: FP->FP or FP->CE
MIM packet
RPF is checked against
Decrement(TTL)
OSA.SwID + FTAG

Y TTL<1 N ODA is N Fail

unicast RPF
Multicast lookups are done using VLAN,
Y Pass FTAG, and ODA
(each multicast mac appears twice)
Destination = N ODA.SwID Destination =
Sw_Table(FTAG, ODA.SwID) is local L2_Table(Vlan, FTAG, ODA) SubSwitchID lookups are omitted here

Y Remember about special LIDs

Dest = LID or (Sup, Flood, …)
Dest = L2_table(DA,VLAN)
FF frames are forwarded out of CE ports
only when DA is locally learned
Forward

Drop
Load-balancing
• N7K: Unicast and Multicast load-balancing are separate
• N5K/N6K: Unified load-balancing mechanism for unicast and multicast
N7K# show fabricpath load-balance • Symmetric: idea is to make ab and ba
ECMP load-balancing configuration: flows take same path by sorting addresses,
L3/L4 Preference: Mixed before feeding them to hash
Hash Control: Symmetric
Rotate amount: 6 bytes • Rotate: polarization avoidance; hash result is
Use VLAN: TRUE
rotated by specified number of bytes.
Ftag load-balancing configuration: Number is derived from unique system MAC
Hash Control: Symmetric
Rotate amount: 6 bytes
Use VLAN: TRUE

N7K# show fabricpath load-balance unicast forwarding ftag 1 switchid 30 flow l2 src-mac 001c.57ad.ecc3
dst-mac 547f.ee02.ce3c ether-type 0x800 vlan 2000 module 3
128b Hash Key generated : 1ffb80b38f02000019000715eb7b30d5
This flow selects interface Eth3/25
Reducing impact of forwarding loops

• Transient loops might occur during convergence (as with

L3 routing)
• To contain impact of these loops FabricPath uses TTL.
Starting in 6.2(2), can set the initial TTL via
fabricpath [multicast | unicast] ttl
• For Multidestination Trees Reverse Path Forwarding
check performed on source switch ID

Nexus5k# show platform fwm info asic-errors 0

DROP_TTL_EXPIRED: res0 = 23 res1 = 0 [10]

Nexus7K-F2# show hardware internal errors module 4 | inc ign ttl

47 Ingress redirect due to dtag_ttl check 0000000000000002 41-44 -
MAC Address Learning
 Conversational learning is
disabled on L3 edge
• Learning MAC addresses is not required in FabricPath Core as switches (when SVI is up
switching is based on Switch ID on FP VLAN)
• FP Edge switches learn local MAC addresses (behind edge ports)  This does not apply to a
conventionally case where F-series is
connected to M-series in
• FP Edge devices learn remote addresses (behind Core-facing ports)
different VDC by external
using conversational learning
cable
• For packets arriving from FP, source MAC (not outer SA!) is learned when
destination MAC of the frame is already known on any Edge port of this
 When M and F are in the
switch
same VDC, special
• No learning from broadcasts (though existing entries will be updated) handling is needed to
forward packets from
• Normal Learning from multicasts (example: HSRP address) MFP core – this is
orchestrated by MCM
(mixed chassis manager)
Conversational MAC Address Learning
S1 S2 S3
A B
MAC Port MAC Port MAC Port

• A sends an ARP for B (broadcast)

S1 S2 S3
A B
MAC Port MAC Port MAC Port
A 1

• B sends ARP reply (unicast) to A

S1 S2 S3
A B
MAC Port MAC Port MAC Port
A 1 B 1
B S3.0.1

• A sends unicast packet to B

S1 S2 S3
A B
MAC Port MAC Port MAC Port
A 1 B 1
B S3.0.1 A S1.0.1
FabricPath Scale Nexus5500 Nexus6000 N7K-F1 N7K-F2 N7K-F3 N7K-M
series

32K MACs 128K MACs* 16K MACs 16K MACs 64K MACs 128K MACs
per SoC per SoC per SoC

Potential bottleneck if
F1/F2 used in L3 Spine

L3

Spine
L2
Spine
L3 Spine
Leaf Leaf

VLAN 100 VLAN 200 VLAN 100 VLAN 200 VLAN 100 VLAN 200 VLAN 100 VLAN 200

Leaf Layer Optimized conversational learning Leaf Layer Optimized conversational learning

Spine No MAC learning (forwarding based on SWID) Spine Learns all MAC addresses in order to
route between VLANs
 FabricPath Proxy L2 Learn
• Goal: Increase MAC table size in FabricPath for F1/F2E modules
• Solution: Offload MAC learning to M-series module at L2/L3 boundary
• Prerequisites: 6.2(2) on N7K (Spine and Leaf) , M1/M2 + F2E or M1/M2 + F1

Configuration
Learn All ! From default VDC (Prevents F2E/F1 from learning on multicast frames)
M1/M2
Remote MACs no hardware fabricpath mac-learning module <x> [port-group <y>]
L3
L2 ! From fabricpath VDC (prevents F2E/F1 from learning remote MACs)
Spine no mac address-table fabricpath remote-learning
No MAC
SoC Learning

Leaf

! If you are using F2 for Leaf core ports to prevent learning from
broadcast/multicast
no hardware fabricpath mac-learning module <x> [port-group <y>]
VLAN 100 VLAN 200 VLAN 100 VLAN 200
 FabricPath MAC Learning Changes: Why?
VLAN MAC Index
200 A gpc1 • M-Series MAC tables contain VLAN, MAC, and port index
(no concept of SWID, SSWID, LID in M-Series MAC table)
M sends
S1 M1/M2 frame to gpc1
• For FP MACs, the destination SWID is mapped to an
L3
GPC SWID internal gateway port-channel (GPC) index which is
L2 programmed in the M-series MAC table
gpc1 S201
FP F translates
SoC
• FP SoC will translate GPC to SWID before sending out FP
frame to
SWID 201, LID port.
FP FP FFFFMAC miss,
SoC SoC causes flood to • Challenge: No way for FP SoC to determine LID for packet
local CE ports
S101 S201
from M-Series module if MAC is not present in local MAC
table. Therefore, packet from M-Series sent out FP with
CE CE flood LID.
SoC SoC If FP SoC on destination switch has not learned MAC, then
packet will be flooded out local CE ports

VLAN 100 VLAN 200 • Solution: Sync MACs on CE SoC to FP SoC.

X, Y, Z A, B, C
FabricPath MAC Learning Changes
• To support L2 proxy learning, MACs learned on CE ports will be synced to all SoCs

A, B, C
A, B, C
CE
CE
SoC
SoC

Learns MAC
A,B,C
FP Learn all MACs on CE
FP Learn all MACs on CE SoC
SoC Sync local CE ports. Learn remote
No MACs ports. Learn remote
MACs to FP SoC MACs via
Learned MACs via FP
FP conversational learning
conversational learning SoC
SoC Learns MAC
X,Y,Z

CE 6.1(2) for F2/F2E CE

SoC
SoC 6.2(2) in F1
X, Y, Z X, Y, Z
Agenda
• FabricPath Overview
Benefits, Restrictions, and Configuration
• Key Concepts
Encapsulation, Trees, Topologies, STP
• Data Plane
Forwarding, Load-Balancing, MAC Learning
• vPC+
Challenges and Operation
• Troubleshooting
Verification steps, tools, and examples
VPC+: Why, What and How (1) MAC B

• Goal: provide redundant, active-active L2 links to separate FP switches

with active-active HSRP
• Challenge 1: depending on the path the packet AB takes, switch S3 will
learn MAC A behind S1 or S2 (or MAC will be moving) S3

• Solution: introduce Emulated Switch S100 to represent devices behind

VPCs: MAC A will appear behind S100 in S3 MAC address table. HSRP Fabric Path
MAC is advertised with emulated switch as a source – taking advantage of
VPC+ multipathing
S3# show mac address-table address 0000.0000.000a S1 S2
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+-------------------
3000 0000.0000.000a dynamic 30 F F 100.0.0
S3# show fabricpath route switchid 100
1/100/0, number of next-hops: 2
via e1/1, [115/20], 1 day/s 05:56:40, isis_fabricpath-default
via e1/2, [115/20], 1 day/s 05:56:38, isis_fabricpath-default S100
S3# show fabricpath switch-id
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED
----------+----------------+------------+-----------+--------------------
1 0000.0000.1001 Primary Confirmed Yes No
2 0000.0000.3002 Primary Confirmed Yes No MAC A
*3 0000.0000.3003 Primary Confirmed Yes No
100 0000.0000.1010 Primary Confirmed No Yes
44
VPC  VPC+ Fabric Path

• To enable VPC+ an Emulated Switch ID must be configured in VPC domain

on both peers (must be the same on both peers and globally unique). ES
represents ALL VPC+ channels of the domain S1 S2
• Peer-link and VPC+ ports must be fabric-path capable
• Peer-link is FP interface
(no STP, only FP vlans are carried, VPC check is no more ).
VPC+ channels are CE
S100
• VPC+ domain must be the root for CE STP, otherwise VPC+ channels will be
blocked as L2GW inconsistent
• FP switches use same STP bridge ID but peer-switch is still recommended
S1# show vpc
vPC domain id : 2 vpc domain 2
vPC+ switch id : 100 fabricpath switch-id 100
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
vPC fabricpath status : peer is reachable through fabricpath
...
vPC role : primary
Number of vPCs configured : 0
...
Fabricpath load balancing : Disabled
Port Channel Limit : limit to 244
 HSRP (and VRRP) in VPC+
• HSRP when enabled on VPC+ peers uses Emulated Switch ID as a source switch
and thus benefits from VPC+ multipathing
• Control-plane-wise one peer will be active and other will be standby, but data-
plane-wise both peers will be forwarding traffic (same as in VPC) S3

• FabricPath devices will have ECMP route to Emulated Switch

Fabric Path
S3# show mac address-table vlan 100 address 0000.0c9f.f064
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
100 0000.0c9f.f064 dynamic 0 F F 100.0.65535
s3# show fabricpath route switchid 100 S1 S2
1/100/0, number of next-hops: 2
via e1/1, [115/20], 1 day/s 05:56:40, isis_fabricpath-default
via e1/2, [115/20], 1 day/s 05:56:38, isis_fabricpath-default

• CE devices will have HSRP VMAC pointing to a port-channel S100

CE1# show mac address-table vlan 100 address 0000.0c9f.f064
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------ CE1
* 100 0000.0c9f.f064 dynamic 0 F F Po1

• If only HSRP active-active is required VPC+ channels are optional

VPC+: Why, What and How (2) MAC B

• Challenge 2: flooded packets from A (with OSA of S100) might come to

S3 from S1 or from S2, but RPF can only be 1 interface S3

RPF RPF
• Solution: S1 and S2 advertise to S3 (via ISIS TLV) an affinity to single FTAG2,S100
FTAG1,S100
FTAG each, S3 will program RPF according to affinity. Multidestination
1/1 1/2
traffic coming from VPC+ will be set to use FTAG 1 for VPC leg on S1
and FTAG 2 for VPC leg on S2
S3# show fabricpath route switchid 100 Affinity Affinity
FabricPath Unicast Route Table FTAG1 FTAG2
1/100/0, number of next-hops: 2
via Eth1/1, [115/40], 11 day/s 00:59:35, isis_fabricpath-default
via Eth1/2, [115/40], 11 day/s 01:03:27, isis_fabricpath-default S1 S2
S3# show fabricpath isis database detail | i Affinity|Host|Numg
Hostname : S1 Length : 2
Affinity : Use FTAG1 Use FTAG2
Nickname: 100 Numgraphs: 1 Graph-id: 1
Hostname : S2 Length : 2
Affinity :
Nickname: 100 Numgraphs: 1 Graph-id: 2 S100
S3# show l2 multicast trees
(ftag/2, topo/0, Switch-id 100), uptime: 1d01h, isis
Outgoing interface list: (count: 1, '*' is the preferred interface)
* Interface Ethernet1/2, [admin distance/115] uptime: 1d01h, isis MAC A
(ftag/1, topo/0, Switch-id 100), uptime: 6d00h, isis
Outgoing interface list: (count: 1, '*' is the preferred interface)
* Interface Ethernet1/1, [admin distance/115] uptime:
47 6d00h, isis
 VPC+: Why, What and How (3) MAC B

• Challenge 3: multidestination packets from FP to CE need to be load-

S3
balanced too
RPF RPF
• Solution: S1 and S2 will each be ‘designated forwarder’ for FTAG of FTAG1,S100 FTAG2,S100
their affinity: traffic for FTAG of affinity will be forwarded out of VPC and 1/1 1/2
other FTAG traffic will be forwarded by peer
S1# show vpc Affinity Affinity
vPC domain id : 100 FTAG1 FTAG2
vPC+ switch id : 100
...
vPC Peer-link status S1 S2
---------------------------------------------------------------------
1 Po1 up 2000-2001,3000-3001
vPC status DF: FTAG1 Po101 DF: FTAG2
-------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans vPC+ Attrib
-- ---------- ------ ----------- ------ ------------ -----------
101 Po101 up success success 10 DF: Yes S100

vPC status
-------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans vPC+ Attrib MAC A
-- ---------- ------ ----------- ------ ------------ ----------- vpc domain 100
101 Po101 up success success 10 DF: Partial
fabricpath multicast load-balance
VPC+: Prevention of Duplicate Packets
• How is packet received from VPC+ and flooded on S1 prevented from being
flooded on S2 to same VPC+ again?
• N7K-F1 linecards:
Each VPC+ will have its own sub-switch ID. Mac addresses will be learned behind
<es_id>.<subsw_id>.<lid>, for example 100.11.65535
Fabric Path
(emulated switch 100, sub-switch 11, LID 65535). S2 will recognize ES + SubSwitch tuple as
its own port and will not flood the frame back to VPC

• N7K-F2, N7K-F3 linecards & N5K, N6K:

By default same as above, as below with ‘fabricpath multicast load-balance’ S1 S2
Each VPC+ peer will be forwarding only for 1 FTAG and traffic coming from other peer will
have different FTAG. For example (previous slide) flooded packet coming from S1 will have

X
FTAG1, but S2 will only flood FTAG2 packets out of the VPC

Required for FEX FP with N7K-F2

 VPC Failover
• VPC+ member link goes down
• Traffic diverted over Peer-Link
• Peer-Link goes down (but Peer-Keepalive up) S3
• Primary: No action
• Secondary: Bring down VPC+ channels
Fabric Path
• Stop advertising reachability to Emulated Switch

S3# show fabricpath route switchid 100

1/100/0, number of next-hops: 1
via e1/1, [115/20], 1 day/s 07:14:24, isis_fabricpath-default S1 S2

• Dual active is much less likely than with normal VPC: if Peer-Link and
Peer-Keepalive go down, but peer is reachable via FP – secondary will
not become primary S100

S1# show vpc

vPC domain id : 2
vPC+ switch id : 100
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
vPC fabricpath status : peer is reachable through fabricpath
Anycast HSRP
• Goal: provide N-gateway solution to increase redundancy and bandwidth
• Alternatives:
1. vPC/vPC+ provides 2 active gateways. Failure of a single gateway reduces available inter-vlan traffic by
half
2. GLBP allows more than 2 active gateways. Drawbacks:
• No ECMP load-balancing since a single virtual MAC is assigned to a single SwitchID
• Non-deterministic distribution of virtual MAC addresses (hard to troubleshoot)
• Solution: Anycast HSRP

Active Standby Listen Listen

L3
L2 All 4 devices actively
routing traffic for the HSRP
Fabric Path virtual MAC
 Anycast HSRP
• The HSRP virtual MAC is bond to an Anycast SwitchID (ASID)
• ASID uses similar concept to vPC+ ES, where each Anycast gateway advertises the ASID via
new Anycast HSRP Sub-TLV
• Each Anycast gateway will actively route traffic for the HSRP virtual MAC
feature interface-vlan
Configure HSRP under the feature hsrp
interface - HSRP version2
Code Requirement
required interface Vlan100 N7K
ASID
ip address 10.1.100.1/24 • 6.2(6)
hsrp version 2 N5K/N6K
hsrp 100 • 6.0(2)N2(1) (SubTLV only)
ip 10.1.100.254
L3 S1 S2 S3 S4 • 7.0(0)N1(1)
interface Vlan101
L2
ip address 10.1.101.1/24
hsrp version 2
hsrp 101
ip 10.1.101.254

hsrp anycast 1 ipv4

switch-id 1000 Configured the ASID for this
vlan 100-101 anycast bundle and
4 Equal Cost no shutdown
Routes to ASID associate vlans
 Anycast HSRP
S202# show fabricpath isis database detail | i "LSPID|00-00|Nickname: 1000"
LSPID Seq Number Checksum Lifetime A/P/O/T
S1.00-00 0x00000100 0x815E 762 0/0/0/1
Nickname: 1000 Numgraphs: 2 Graph-id: 1, 2
S2.00-00 0x00000103 0xC618 776 0/0/0/1
Nickname: 1000 Numgraphs: 2 Graph-id: 1, 2  Each switch sends ISIS
... TLVs advertising ASID
S202# show fabricpath route switchid 1000
...  ECMP routes built toward
1/1000/0, number of next-hops: 4 ASID to increase
via Eth1/6, [115/40], 0 day/s 03:00:18, isis_fabricpath-default
via Eth1/7, [115/40], 0 day/s 03:02:55, isis_fabricpath-default redundancy and
via Eth1/8, [115/40], 0 day/s 03:01:08, isis_fabricpath-default bandwidth
via Eth1/9, [115/40], 0 day/s 03:03:45, isis_fabricpath-default
 HSRP Active Hellos are
S202# show mac address-table dynamic
Legend:
sent out with a OSA of the
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC ASID and SA of the virtual
age - seconds since last seen,+ - primary entry using vPC Peer-Link MAC
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 101 0000.0c9f.f065 dynamic 10 F F 1000.0.65535
* 100 0000.0c9f.f064 dynamic 10 F F 1000.0.65535
Agenda
• FabricPath Overview
Benefits, Restrictions, and Configuration
• Key Concepts
Encapsulation, Trees, Topologies, STP
• Data Plane
Forwarding, Load-Balancing, MAC Learning
• vPC+
Challenges and Operation
• Troubleshooting
Verification steps, tools, and examples
 S1 S2 FP Vlans 100-199

FabricPath: Configuration
install feature-set fabricpath
feature-set fabricpath S101 S102 S201 S202

vlan 100-199
mode fabricpath Best practice to manually
fabricpath switch-id 101 configure switch-id A ES S100 B C ES S200 D
vpc domain 100
fabricpath switch-id 100
fabricpath multicast load-balance

! Fabricpath core ports

interface Ethernet6/4 - 5
switchport mode fabricpath ! S1
fabricpath domain default
! Peer-link root-priority 255
interface port-channel1 ! S2
switchport mode fabricpath fabricpath domain default
root-priority 254
! vPCs are CE ports (mode access or mode trunk)
interface port-channel20 Configure roots for
switchport FTAG 1 and 2
switchport mode trunk
vpc 20
 S1 S2 FP Vlans 100-199

FabricPath: Health Check

S101# sh sys internal plugin info global | begin l2mp | head lines 5
Feature-set id: 2, name: l2mp S101 S102 S201 S202
vdc: 1 state: PLUGIN_ENABLED_STATE
vdc: 2 state: PLUGIN_ENABLED_STATE
vdc: 3 state: PLUGIN_ENABLED_STATE

A ES S100 B C ES S200 D
S101# show system internal sysmgr service all | i 2rib|drap|fabric|PID
Name UUID PID SAP state Start count Tag Plugin ID
isis_fabricpath 0x41000243 6475 436 s0009 1 N/A 1
 FabricPath plugin in good drap 0x0000024E 6476 448 s0009 1 N/A 1
state m2rib 0x00000250 6435 449 s0009 1 N/A 1
u2rib 0x00000254 6474 452 s0009 1 N/A 1
 Services running for URIB,
MRIB, DRAP, ISIS S101# show processes cpu | i 2rib|drap|fabric|PID
 CPU levels are reasonable PID
6435
Runtime(ms) Invoked
410 335
uSecs 5Sec
1 0.00%
1Min
0.00%
5Min
0.00%
TTY
-
Process
m2rib
 Memory below limits 6474 170 735 0 0.00% 0.00% 0.00% - u2rib
6475 690 3764 0 0.00% 0.00% 0.00% - isis_fabricpath
6476 200 725 0 0.00% 0.00% 0.00% - drap

S101# show processes memory | i 2rib|drap|fabric|PID

PID MemAlloc MemLimit MemUsed StackBase/Ptr Process
6435 11149312 923422860 273965056 ffd8cb40/ffffffff m2rib
6474 3657728 564849190 262389760 ffbc5b80/ffffffff u2rib
6475 30515200 814058995 479059968 ff8eed50/ffffffff isis_fabricpath
6476 3067904 619628416 262160384 ffa58950/ffffffff drap
 S1 S2 FP Vlans 100-199

FabricPath: Health Check

S101# show fabricpath isis
System ID : 8478.ac0e.4743 IS-Type : L1 Fabric-Control SVI: Unknown
Process is up and running
Interfaces supported by Fabricpath IS-IS : S101 S102 S201 S202
port-channel1
Ethernet6/27
Ethernet6/28

S101# show fabricpath topology vlan active A ES S100 B C ES S200 D

Topo-Description Topo-ID Active VLAN List
-------------------------------- --------- -------------------------
0 0 100-199

S101# show fabricpath isis interface brief

Fabricpath IS-IS domain: default
 ISIS is running Interface Type Idx State Circuit MTU Metric Priority Adjs/AdjsUp
 system ID is accurate --------------------------------------------------------------------------------
port-channel1 P2P 3 Up/Ready 0x01/L1 1500 40 64 1/1
 Interface list matches configuration Ethernet6/27 P2P 1 Up/Ready 0x01/L1 1500 40 64 1/1
 Active Vlans match configuration Ethernet6/28 P2P 2 Up/Ready 0x01/L1 1500 40 64 1/1
 Interfaces in Up/Ready state S101# show fabricpath isis adjacency detail
 Adjacencies established Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID SNPA Level State Hold Time Interface
 Adjacencies stable S102 N/A 1 UP 00:00:25 port-channel1
Up/Down transitions: 1, Last transition: 3w5d ago
Circuit Type: L1
Topo-id: 0, Forwarding-State: UP
 S1 S2 FP Vlans 100-199

FabricPath: Health Check

S101# show fabricpath isis traffic port-channel 1
Fabricpath IS-IS domain: default
Fabricpath IS-IS Traffic for port-channel1:
PDU Received Sent RcvAuthErr OtherRcvErr ReTransmit
P2P-IIH 734 733 0 0 n/a S101 S102 S201 S202
CSNP 2 1 0 0 n/a
PSNP 113 113 0 0 n/a
LSP 131 134 0 0 0

S101# show fabricpath switch-id A ES S100 B C ES S200 D

FABRICPATH SWITCH-ID TABLE
Legend: '*' - this system
'[E]' - local Emulated Switch-id
'[A]' - local Anycast Switch-id
Total Switch-ids: 10
=========================================================================
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED/
ANYCAST  No growing errors on interfaces
--------------+----------------+------------+-----------+----------------  All switches and ES are seen and in
1 8478.ac0e.4742 Primary Confirmed Yes No
2 8478.ac5b.2b42 Primary Confirmed Yes No confirmed state
[E] 100 8478.ac0e.4743 Primary Confirmed No Yes
100 8478.ac5b.2b43 Primary Confirmed No Yes
* 101 8478.ac0e.4743 Primary Confirmed Yes No
102 8478.ac5b.2b43 Primary Confirmed Yes No
200 547f.eed6.70fc Primary Confirmed No Yes
200 547f.eedb.7e7c Primary Confirmed No Yes
201 547f.eed6.70fc Primary Confirmed Yes No
202 547f.eedb.7e7c Primary Confirmed Yes No
 S1 S2 FP Vlans 100-199

FabricPath: Unicast Example (MAC)

S101# show mac-address-table address-table vlan 100
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False S101 S102 S201 S202
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------ vPC30 vPC40
* 100 0000.0000.000a dynamic 0 F F Po30
100 0000.0000.000d dynamic 0 F F 200.0.0
A ES S100 B C ES S200 D

S101# show hardware mac address-table 6 vlan 100

FE | Valid| PI| BD |
| | | |
MAC | Index| Stat| SW | ... | SWID| SSWID| LID
| | ic | | ... | | |
7K  MACs are present in software
---+------+---+------+---------------+-------+-----+-----+ ... |-----|------|------- MAC table
7 1 1 245 0000.0000.000a 0x00408 0 0x089 0x064 0x00b 0x00408  Use Platform Dependent
7 1 0 245 0000.0000.000d 0x00000 0 0x009 0x0c8 0x000 0x00000
commands to check hardware
MAC table
S101# show system internal pixm info ltl 0x408 7K  On S101, MAC D matches
software remote address (200.0.0)
PC_TYPE PORT LTL RES_ID LTL_FLAG CB_FLAG MEMB_CNT
------------------------------------------------------------------------------  MAC A has local SWID/SSWID
Normal Po30 0x0408 0x1600001d 0x00000000 0x00000002 1 100.11 with LID 0x408

LID 0x408 maps to local Po30 Hex SWID/SSWID

0xc8 0x00 = 200 0
0x64 0x0b = 64 11
 S1 S2 FP Vlans 100-199

FabricPath: Unicast Example (MAC)

S202# show mac address-table vlan 100
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link S101 S102 S201 S202
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 100 0000.0000.000a dynamic 0 F F 100.11.65535 vPC30 vPC40
* 100 0000.0000.000d dynamic 0 F F Po40
A ES S100 B C ES S200 D

S202# show platform fwm info hw-stm | i HW|VLAN|_|---|000a|000d 5K

HW STM Contents  MACs are present in software
dleft loc - bucket_type:line:bucket_number
misc - learn_type:ecc:valid:fcf MAC table
cdce format - ig:ul:switch_id:subswitch_id:end_node_id:pbp_idx:local_id  Use Platform Dependent
VLAN MAC Address Port loc misc cdce
------+----------------+--------------+--------+-------+--------------------
commands to check hardware
1.100 0000.0000.000d Po40 1:1111:0 1:0:1:0 2.0.c8.0.0.15 (e:0) MAC table
1.100 0000.0000.000a l2mp-nh 1:2918:0 1:0:1:0 2.0.64.b.ff.ff (e:0)  On S202, MAC A matches
software remote address
(100.11.65535)
S202# show platform fwm info lif port-channel 40 | i local_id 5K
Po40 pd: local_id 21 endnode_id 0 endnode_id_alloced 1 vif_id 0  MAC A has local SWID/SSWID
200.0 with LID 0x15 (0x15 = 21)

LID 21 maps to local Po40

FabricPath: What command comes from where
show fabricpath switch

show fabricpath conflict all | link | switch | transitions

show fabricpath isis switch

show fabricpath isis interface Supervisor

DRAP Engine
show fabricpath isis adjacency
FabricPath IS-IS
show fabricpath isis database
U2RIB L2FM
show fabricpath isis route

show fabricpath route

U2FIB MTM
show mac address-table
Hardware Drivers
slot <> show fabricpath unicast routes vdc
Switch Table Other HW MAC Table
slot <> show hardware internal forwarding inst <> table <> 7K
I/O Module
show platform fwm info l2mp route ftag <> switch <> hw 5K 6K
slot <> show hardware mac address-table 7K

show platform fwm info hw-stm 5K 6K

 S1 S2 FP Vlans 100-199

FabricPath: Unicast Example (SWID)

S101# show fabricpath isis database detail
Fabricpath IS-IS domain: default LSP database
LSPID Seq Number Checksum Lifetime A/P/O/T
S201.00-00 0x00000006 0xF8A7 957 0/0/0/1 S101 S102 S201 S202
Hostname : S201 Length : 4
Capability : Device Id: 201 Base Topology
Affinity : vPC30 vPC40
Nickname: 200 Numgraphs: 1 Graph-id: 1
Nickname : A B C D
ES S100 ES S200
Priority: 0 Nickname: 201 BcastPriority: 64
Priority: 0 Nickname: 200 BcastPriority: 0
S202.00-00 0x00000007 0x5F3B 884 0/0/0/1
Hostname : S202 Length : 4  Route for destination SWID present in
Capability : Device Id: 202 Base Topology ISIS table and U2RIB
Affinity :
Nickname: 200 Numgraphs: 1 Graph-id: 2
Nickname : S101# show fabricpath route switchid 200
Priority: 0 Nickname: 202 BcastPriority: 64 FabricPath Unicast Route Table
Priority: 0 Nickname: 200 BcastPriority: 0 'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
...
S101# show fabricpath isis route 1/200/0, number of next-hops: 2
Fabricpath IS-IS domain: default MT-0 via Eth6/27, [115/80], 0 day/s 00:21:58,
Topology 0, Tree 0, Swid routing table isis_fabricpath-default
... via Eth6/28, [115/80], 0 day/s 00:21:58,
200, L1 isis_fabricpath-default
via Ethernet6/27, metric 80
via Ethernet6/28, metric 80
 S1 S2 FP Vlans 100-199

FabricPath: Unicast Example (SWID)

module-6# show fabricpath unicast routes vdc 3 ftag 1 switchid 200
Route in VDC 3 7K
--------------------------------------------------------------------------------
FTAG | SwitchID | SubSwitchID | Loc/Rem | RPF | RPF Intf | Num Paths | Merge S101
V S102 S201 S202
--------------------------------------------------------------------------------
0001 | 0200 | 0000 | Remote | Yes | Eth6/27 | 2 | 1
-------------------------------------------------------------------------------- vPC30 vPC40
...
PD Information for ECMP: A ES S100 B C ES S200 D
Common Info Two equal costs routes via
-------------------------------- Eth6/27 and Eth6/28. RPF
AMM key : 0x6000024 interface Eth6/27
--------------------------------  Use Platform Dependent commands to
Next Hop | Interface | LID
-------------------------------- verify route for destination SWID is
0 | Eth6/27 | 0000006a present in hardware
1 | Eth6/28 | 0000006b
 On N7K, first attach to appropriate
module via “attach module x”
S202# show platform fwm info l2mp route ftag 1 swid 100 5K
-------------------------------------------------------------------
l2mp_route[0x99f23ac]
route_type: 10 (0xa) merge_version: 1 (0x1)
iic interface: Eth1/7 (0x1a006000) Two equal costs routes via
ftag: 1 (0x1) switchid: 100 (0x64)-> l2mp_nexthop[0x8944dc4] Eth1/7 and Eth1/8. RPF
num_paths: 2 interface Eth1/7
nh[1]: Eth1/7 (0x1a006000)
nh[2]: Eth1/8 (0x1a007000)
FabricPath: what comes from where

show fabricpath isis switch

Supervisor
DRAP
show fabricpath isis topology summary Engine

FabricPath IS-IS IGMP

show fabricpath isis tree
M2RIB L2FM
show fabricpath isis database mgroup detail

show fabricpath mroute MFDM

show l2 multicast trees

M2FIB MTM
show ip igmp snooping groups
Hardware Drivers
show forwarding distribution l2 multicast [vlan <>] 7K
Switch Table Other HW MAC Table

I/O Module
 S1 S2 FP Vlans 100-199

FabricPath: Multidestination (Flood)

S101# show fabricpath isis topology summary
FabricPath IS-IS Topology Summary
Fabricpath IS-IS domain: default
MT-0 S101 S102 S201 S202
Configured interfaces: port-channel1 Ethernet6/27 Ethernet6/28
Max number of trees: 2 Number of trees supported: 2
Tree id: 1, ftag: 1, root system: 8478.ac0e.4742, 1 vPC30 vPC40
Tree id: 2, ftag: 2 [transit-traffic-only], root system: 8478.ac5b.2b42, 2
Ftag Proxy Root: 8478.ac0e.4742 A B C D
ES S100 ES S200

S1# show fabricpath isis trees

S101# show fabricpath isis trees MT-0
MT-0 Topology 0, Tree 1, Swid routing table  Check the topology roots for
Topology 0, Tree 1, Swid routing table 2, L1 each FTAG
1, L1 via port-channel1, metric 20
via Ethernet6/27, metric 0 100, L1  Map out the active links
2, L1 via Ethernet6/19, metric 40  How to read: on which
Repeat on each
via Ethernet6/27, metric 20 101, L1 interface in given FTAG will this
102, L1 switch to map out via Ethernet6/19, metric 40
via Ethernet6/27, metric 40 complete 102, L1 switch accept multidestination
200, L1 forwarding tree via Ethernet6/20, metric 40 traffic from given switch
via Ethernet6/27, metric 40 (FTAG 1) 200, L1  Example: accept traffic from
201, L1 via Ethernet6/21, metric 40
via Ethernet6/27, metric 40 201, L1 switch 100 on E6/19 in FTAG1
202, L1 via Ethernet6/21, metric 40
via Ethernet6/27, metric 40 202, L1
via Ethernet6/22, metric 40
 S1 S2 FP Vlans 100-199

FabricPath: Multidestination (Flood)

S101# show fabricpath mroute vlan 100 flood

(vlan/100, *, *), Flood, uptime: 02:01:06, isis

Outgoing interface list: (count: 5) S101 S102 S201 S202
Switch-id 1, uptime: 02:01:06, isis
Switch-id 2, uptime: 02:01:06, isis
vPC30 vPC40
Switch-id 102, uptime: 01:59:40, isis
Switch-id 201, uptime: 02:01:06, isis
Switch-id 202, uptime: 02:01:06, isis A ES S100 B C ES S200 D

S101# show fabricpath mroute vlan 100 flood resolved

(ftag/2, vlan/100, *, *), Flood, uptime: 02:01:32, isis

Outgoing interface list: (count: 5)
 Flood entry – traffic that will be flooded to all active ports
Interface Ethernet6/28, Switch-id 1, uptime: 02:01:31, isis (minus receiving port) in a Vlan
Interface Ethernet6/28, Switch-id 2, uptime: 02:01:31, isis (remember about dynamic pruning)
Interface Ethernet6/28, Switch-id 102, uptime: 02:00:07, isis
Interface Ethernet6/28, Switch-id 201, uptime: 02:01:31, isis
 Ignore multiple appearances of the same interface
Interface Ethernet6/28, Switch-id 202, uptime: 02:01:31, isis (interface appears 1 per destination switch)
(ftag/1, vlan/100, *, *), Flood, uptime: 02:01:32, isis
Outgoing interface list: (count: 5)
Interface Ethernet6/27, Switch-id 1, uptime: 02:01:31, isis
Interface Ethernet6/27, Switch-id 2, uptime: 02:01:31, isis
Interface Ethernet6/27, Switch-id 102, uptime: 02:00:07, isis
Interface Ethernet6/27, Switch-id 201, uptime: 02:01:31, isis
Interface Ethernet6/27, Switch-id 202, uptime: 02:01:31, isis
 S1 S2 FP Vlans 100-199
Remember
FabricPath: IP Multicast RPF check

S202# show ip igmp snooping groups vlan 100

Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core port

Vlan Group Address Ver Type Port list S101 S102 S201 S202
100 */* - RF Eth1/7
RF Eth1/8 vPC30 vPC40
100 239.1.1.1 v2 D Po40

A ES S100 B C ES S200 D
S101# show fabricpath isis database mgroup detail | egrep "LSPID|Group|00-01" Multicast Multicast
LSPID Seq Number Checksum Lifetime A/P/O/T Sender Receiver
S201.00-01 0x00000093 0xEA2C 1092 0/0/0/1
Group-Address : IP Multicast : Vlan : 100 Groups : 1  *,G from local IGMP snooping
Group : 239.1.1.1 Sources : 0
S202.00-01 0x00000090 0xBD66 709 0/0/0/1  Local IGMP/snooping entries are
Group-Address : IP Multicast : Vlan : 100 Groups : 1 redistributed into FP
Group : 239.1.1.1 Sources : 0
 L2 multicast prune subtrees built
on each FP switch
S101# show fabricpath mroute vlan 100  S101 hashes multicast to FTAG 1
(vlan/100, 0.0.0.0, 239.1.1.1), uptime: 20:35:57, isis (remember vPC+ affinity)
Outgoing interface list: (count: 2)
Switch-id 201, uptime: 20:35:57, isis S101# show fabricpath mroute vlan 100 ftag 1
Switch-id 202, uptime: 20:35:57, isis
(ftag/1, vlan/100, 0.0.0.0, 239.1.1.1), uptime: 20:47:34, isis
Outgoing interface list: (count: 2)
Interface Ethernet6/27, Switch-id 201, uptime: 22:26:18, isis
Interface Ethernet6/27, Switch-id 202, uptime: 22:26:18, isis
 S1 S2 FP Vlans 100-199

FabricPath: IP Multicast
QUIZ
Both S201 and S202 receive multicast S101 S102 S201 S202

stream, who forwards out vPC 40? vPC30

x vPC40

A ES S100 B C ES S200 D
S202# show vpc 40
vPC status Multicast Multicast
--------------------------------------------------------------------------- Sender Receiver
id Port Status Consistency Reason Active vlans vPC+ Attrib
-- ---------- ------ ----------- ------ ------------ -----------
40 Po40 up success success 100-199 DF: Partial,
FP MAC:
200.0.0
 vPC+ in partial status which
means multidestination traffic is
S201# show fabricpath isis database detail S201.00-00 | sec Affinity
Affinity :
load-balanced between vPC peers
Nickname: 200 Numgraphs: 1 Graph-id: 1  S201 has affinity for FTAG 1
S201# show fabricpath isis database detail S202.00-00 | sec Affinity  S202 has affinity for FTAG 2
Affinity :
Nickname: 200 Numgraphs: 1 Graph-id: 2  S201 will forward this frame
FabricPath: Hardware Multicast MAC
• Multicast MACs are stored differently from usual 0100.5exx.xxxx
F1
module-4# show hardware mac address-table vlan <vlan> vdc <vdc> fe <fe>
FE | Valid| PI| BD | MAC | Index|...| PV | RD| NN| UC|PI_E8| SWID| SSWID| LID
| | | | | |...| | | | | | | |
---+------+---+------+---------------+-------|...|----|---|---|---|-----|-----|------|-------
4 1 0 52 0100.ef01.0203 0x07ffb ... 0x00 0 0 0 0 0x000 0x000 0x07ffb
4 1 0 52 0100.ef04.0506 0x07ffb ... 0x00 0 0 0 0 0x000 0x000 0x07ffb
4 1 0 52 0100.ef01.0203 0x07ffb ... 0x00 0 0 0 0 0x000 0x000 0x07ffb
4 1 0 52 0100.ef04.0506 0x07ffb ... 0x00 0 0 0 0 0x000 0x000 0x07ffb

• Each mac appears twice: once per FTAG, use ‘show hard internal forwarding … table mac’ to find which
is which
F2 module-6# show hardware mac address-table vlan <vlan> vdc <vdc> fe <fe>
FE | Valid| PI| BD | MAC | Index| Stat| SW | Modi| Age| ... | SWID| SSWID| LID
| | | | | | ic | | fied|Byte| ... | | |
---+------+---+------+---------------+-------+-----+-----+-----+----+ ... |-----|------|-------
7 1 1 245 0000.0000.000a 0x00408 0 0x009 1 199 ... 0x064 0x00b 0x00408
7 1 0 245 0000.0000.000d 0x00000 0 0x009 1 199 ... 0x0c8 0x000 0x00000
7 1 0 245 4180.0f01.0101 0x07fd8 1 0x000 0 0 ... 0x000 0x000 0x07fd8
7 1 0 245 4180.0f01.0101 0x07fda 1 0x000 0 0 ... 0x000 0x000 0x07fda
Looking back in time

• show fabricpath isis internal event-history adjacency

events related to adjacencies (up/down/etc)
• show fabricpath isis internal event-history urib
FP events related to URIB updates
(for example to see whole history for given switch ID)
• show fabricpath isis internal event-history events
Overall FP event history: DRAP interactions, switch additions, removals, SPF-
related events
• show fabricpath isis internal event-history drap
switch ID, FTAG related events
Tools
Troubleshooting Tools: Pong
• Pong can be equated to L2Ping + L2TraceRoute
• Depends on IEEE 1588v2 HW support F-series, N5500, and N6000 all support PTP, but
N5K/N6K at present doesn’t support pong
• Works by sending 2 types of packets: 1 packet to store timestamps at each hop and 2nd
to collect stored timestamps
S101# pong destination-swid 2 destination-mac 8478.ac5b.2b42 vlan 100 details

Legend (*) - software delay(not hardware latency) Send frame to SWID 2

(#) - reverse path (SysID of SWID 2 = 8478.ac5b.2b42)
(NA) - not available
--- ------------------------- --------------------------
* By default, Frame sent on VLAN 1. Be
Hop System-mac (switch-id) Switching time sure to specify appropriate VLAN
(sec, nsec)
--- ------------------------- --------------------------
1
2
84-78-ac-0e-47-43 ( 101)
84-78-ac-0e-47-42 ( 1)
5588
5588
353692400
353692896
 MACs that can be reached:
3 84-78-ac-0e-47-42 ( 1) 5588 353698488
Egress from SWID 101 - SysID or static
4 84-78-ac-5b-2b-42 ( 2) 5588 415486312 Ingress SWID 1  Not supported over ECMP on F2
5
6
84-78-ac-5b-2b-42 (
84-78-ac-0e-47-42 (
2)
1)
5588
5588
930158536
868372664
Egress SWID 1
7 84-78-ac-0e-47-42 ( 1) 5588 868378248 Etc..
8 84-78-ac-0e-47-43 ( 101) 5588 868378768
Round trip time: 0sec 14144 nsec
Troubleshooting Tools: FPOAM
• FPOAM (Fabricpath Operations Administration and Management) is an effective tool set to
monitor and diagnose data plane failures in FP networks.
• ping fabricpath
• traceroute fabricpath
• mtrace fabricpath

202# mtrace fabricpath ftag 2 repeat 1

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,

'D' - Destination Unreachable, 'X' - unknown return code,
'V' - VLAN nonexistent, 'v' - VLAN in suspended state,
'm' - malformed request, 'C' - Cross Connect Error,
'U' - Unknown RBridge nickname, 'n' - Not AF,
'*' - Success, Optional Tlv incomplete,
'I' - Interface not in forwarding state, S101 S102 S201 S202
'S' - Service Tag nonexistent, 's' - Service Tag in suspended state,
'c' - Corrupted Data/Test

Fabricpath mtrace for multicast ftag 2, vlan 1

ES S100 ES S200
Code SwitchId Interface State TotalTime
==================================================
! 201 Rcvd on Eth1/2 fwd 3ms
! 101 Rcvd on Eth1/2 fwd 4ms
! 102 Rcvd on Eth1/2 fwd 4ms
 S1 S2 FP Vlans 100-199
Troubleshooting Tools: FPOAM
• OAM Profiles can be used to replicate
dataplane packet and follow the forwarding S202
S101 S102 S201
path

202# traceroute fabricpath switch-id 1034 profile 2 A ES S100 B C ES S200 D

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,

'D' - Destination Unreachable, 'X' - unknown return code,
'V' - VLAN nonexistent, 'v' - VLAN in suspended state,
'm' - malformed request, 'C' - Cross Connect Error,
'U' - Unknown RBridge nickname, 'n' - Not AF,
202# show run fabricpath | section "oam profile 2"
'*' - Success, Optional Tlv incomplete,
fabricpath oam profile 2
'I' - Interface not in forwarding state,
vlan 100
'S' - Service Tag nonexistent, 's' - Service Tag in suspended state,
flow forward
'c' - Corrupted Data/Test
ether-type 0x800
ip source 100.1.1.20
Sender handle: 14
ip destination 10.1.1.30
Hop Code SwitchId Interface State TotalTime PathId
mac-address source 0000.1010.1010
============================================================
mac-address destination 0000.3333.3333
1! 2 Rcvd on Eth6/2 fwd 3ms
protocol 1
2 ! 100 Rcvd on Eth1/1 fwd 4ms
 S1 S2 FP Vlans 100-199
Troubleshooting Tools: Counters
S202(config)# ip access-list test-stats
S202(config-acl)# statistics per-entry
S202(config-acl)# permit ip host 10.1.100.101 host 10.1.100.201 S101 S102 S201 S202
S202(config-acl)# permit ip any any
S202(config-acl)# interface ethernet 1/7
S202(config-if)# ip port access-group test-stats in vPC30 vPC40
S202(config-if)# end
A ES S100 B C ES S200 D

S202# show ip access-lists test-stats

IPV4 ACL test-stats

 Find the likely interface to receive packets
statistics per-entry (note multidestination traffic might follow
10 permit ip 10.1.100.101/32 10.1.100.201/32 [match=0]
20 permit ip any any [match=0]
different path  sh fab isis trees)
 Configure ACL with ‘statistics per-entry’
! Sent 5000 frames which explicitly matches traffic in question
S202# show ip access-lists test-stats
 Attach ACL to ingress FP port as a PACL
IPV4 ACL test-stats  Check the counters
statistics per-entry
10 permit ip 10.1.100.101/32 10.1.100.201/32 [match=5000]  Run test traffic
20 permit ip any any [match=0]  Check the counters again
 Compare
Troubleshooting Tools: Counters  Find ingress interface & attach to
respective linecard
S1# attach module 6
 Find Ingress FE instance
module-6# show hardware internal dev-port-map  Configure statistics (use FE+1)
--------------------------------------------------------------  Print statistics
CARD_TYPE: 48 port 10G
FP port | PHYS | MAC_0 | L2LKP | L3LKP | QUEUE |SWICHF  Run traffic
...  Print statistics again – note statistics
19 4 4 4 4 4 0
20 4 4 4 4 4 0 are in HEX
21 5 5 5 5 5 0  Compare
22 5 5 5 5 5 0
...
module-6# test fabricpath unicast configure route-stats vdc 2 ftag 1 switchid 200 fe 5 table [mp | sw] commit

module-6# show fabricpath unicast route-stats vdc 2 ftag 1 switchid 200 fe 5

------------------------------------------  Use MP table to get per next-hop stat
| VDC | FTAG | SwitchID | SubSwitchID | if there is >1 next-hop, else use SW
-------------------------------------------
| 002 | 0001 | 0200 | 000 |
table
| FE | Adjacency | Statistics |
| 4 | Eth6/21| 0000000000 |
| 4 | Eth6/22| 0000000000 |
module-6# show fabricpath unicast route-stats vdc 2 ftag 1 switchid 200 fe 5
------------------------------------------
| VDC | FTAG | SwitchID | SubSwitchID |
-------------------------------------------
| 002 | 0001 | 0200 | 000 |
| FE | Adjacency | Statistics |
| 4 | Eth6/21| 0000000000 |
| 4 | Eth6/22| 0000000064 |
 Troubleshooting Tools: Error/Drop Counters
• Usual datapath troubleshooting apply on N7K  show hardware internal errors
7k# show hardware internal errors module 6 | diff
often produces lengthy outputs, use
... send 2000 transit packets using ping with timeout 0 ... diff to just see what has changed
7k# show hardware internal errors module 6 | diff
< 1008 Self-forwarding check OSA drop 0000000287061579 3 –
between 2 timed samples
> 1008 Self-forwarding check OSA drop 0000000287063630 3 - (with some test traffic in the middle)
< 2514 Ingress packets marked with drop_oth sent to IB 0000000002127119 4 –
> 2514 Ingress packets marked with drop_oth sent to IB 0000000002127173 4 -

< 50 smallcnt DSWID/DSSWID miss and DCE frame, def-gw disabled 0000000000000563 5-6 –
> 50 smallcnt DSWID/DSSWID miss and DCE frame, def-gw disabled 0000000000002563 5-6 -

• And on N5K/N6K
N5K# sh platform fwm info pif e1/5 | i stats|cdce  PIF (physical interface) maintains RX/TX
Eth1/5 pd: tx stats: bytes 304069130 frames 913992 discard 0 drop 0 and drop counters
Eth1/5 pd: rx stats: bytes 9647836468 frames 8319249 discard 0 drop 1650
Eth1/5 pd cdce_addr: switchid 30 sub-switchid 0, endnodeid 0  Check if drops are non-zero & growing
Eth1/5 pd cdce_addr: Mcast 0, locally-adm 1, OutOfOrder/don't learn 0 (also check the ASIC number)
Eth1/5 pd cdce_addr: localid 5, pbp_idx 0
 Use ASIC-errors command to get a
N5K# sh platform fwm info asic-errors 0 breakdown of drop reasons (and see if any
Printing non zero Carmel error registers:
DROP_SRC_VLAN_MBR: res0 = 495188 res1 = 0 [12]
are growing in with test/ping traffic)
DROP_CDCE_SW_TBL_RPF_MISS: res0 = 4 res1 = 0 [30]
DROP_SRC_FTAG_BITMAP_MBR: res0 = 5 res1 = 0 [31]
DROP_SRC_MASK_TO_NULL: res0 = 332912 res1 = 0 [44]
Troubleshooting Tools: ELAM
• When the going gets tough… 
• Embedded Logic Analyzer Module (ELAM) is an engineering tool that is used to look
inside Cisco ASICs.
• ELAM is architecture specific and therefore will have different capabilities and different
CLI syntax across different forwarding engines (FE).
• It is possible to use ELAM as a capturing tool to validate:
1. Was the packet received
ELAM is NOT a supported feature.
2. On which interface/VLAN did the packet arrive It is a diagnostic tool designed for
3. What did the packet look like internal use. Anything and
everything about it may change from
4. How was the packet altered and where was it sent
version to version without any notice
• It is not intrusive
• It can be used at a very granular level to troubleshoot a single traffic flow which can be
an invaluable tool to network administrators.
 Troubleshooting Tool: ELAM Workflow

Identify the Configure an After ELAM

expected ingress ELAM trigger to triggers, display
Start the ELAM
Forwarding capture specific and analyze the
Engine (FE) frame data

Once triggered data can be displayed and analyzed

Typical ELAM challenges
 Identifying the correct capture point and trigger

 Understanding the captured data (for complex cases)

Troubleshooting Tools: ELAM
• Basics to know before performing an ELAM
• Data Bus (DBUS) and Result Bus (RBUS)
The DBUS contains several platform specific internal fields along with the header
information from a frame required to make the forwarding decision. We use the DBUS
information to validate where the frame was received and basic data about the frame.
The RBUS will contain information about the forwarding decision to help determine if the
frame was altered and where it was sent.
• Local Target Logic (LTL)
The LTL is an index used to represent a port or group of ports. The source LTL index and
the destination LTL index tell us which port the frame was received and where it was sent.
 S1 S2 FP Vlans 100-199
Troubleshooting Tools: ELAM Example
• Packet from host 10.1.100.101 <-> 10.1.100.201, expected ingress
interface Eth6/19 on N7K-F2 linecard of S1
S101 S102 S201 S202
S1# attach module 6
Attaching to module 6 ... vPC30 vPC40
module-6# show hardware internal dev-port-map
+-----------------------------------------------------------------------+
A ES S100 B C ES S200 D
+----------------+++FRONT PANEL PORT TO ASIC INSTANCE MAP+++------------+
+-----------------------------------------------------------------------+
FP port | PHYS | MAC_0 | L2LKP | L3LKP | QUEUE |SWICHF
...
19 4 4 4 4 4 0 Linecard L2/L3 ASIC name
...
Eth6/19 is on FE instance 4
module-6# elam asic clipper instance 4 (code name clipper) M-series Eureka/Lamira
module-6(clipper-elam)# layer2
module-6(clipper-l2-elam)# trigger dbus ipv4 ingress if source-ipv4- F1 Orion
address 10.1.100.101 destination-ipv4-address 10.1.100.201 F2 Clipper
module-6(clipper-l2-elam)# trigger rbus ingress if trig
module-6(clipper-l2-elam)# start Configure a trigger specific to
F3 Flanker
module-6(clipper-l2-elam)# status this source/destination IP
L2 DBUS Triggered
L2 RBUS Triggered
Start the ELAM, send the
traffic and wait for it to trigger
 S1 S2 FP Vlans 100-199
Troubleshooting Tools: ELAM Example
module-6(clipper-l2-elam)# show dbus
<snip>
port-id : 0x2 last-ethertype : 0x800 S101 S102 S201 S202
vlan : 0x64 destination-index : 0x0
source-index : 0x62 bundle-port : 0x0
status-is-1q : 0x1 trill-encap : 0x0 vPC30 vPC40
mac-in-mac-valid : 0x1 dtag-ttl : 0x20
recirc-acos : 0x0 dtag-ftag : 0x1 A B C D
ES S100 ES S200
source-ipv4-address: 10.1.100.101
destination-ipv4-address: 10.1.100.201
mim-destination-mac-address: 0200.c800.0000
mim-source-mac-address: 0200.640b.ffff
destination-mac-address 0000.0000.000d
 Frame received on VLAN 100 (0x64) from a
source-mac-address: 0000.0000.000a
 ODA (0c8.00.0000) = 200.0.0 source-index of 0x62 (next slide)
 OSA (064.0b.ffff) = 100.11.65535  mac-in-mac valid (this is a FP frame)
module-6(clipper-l2-elam)# show rbus
<snip>  dtag-TTL: fabricpath TTL of 32 (0x20)
di-ltl-index : 0x65 l3-multicast-di : 0x0
source-index : 0x62 vlan-id : 0x64
dtag=ftag : 0x1 dtag-ttl : 0x1f
mim-destination-mac-address: 0200.c800.0000  Frame transmitted on vlan 100 (0x64) to a destination
mim-source-mac-address: 0200.640b.ffff
index of 0x65 (next slide)
 dtag-TTL: fabricpath TTL decremented to 31 (0xf1)
 S1 S2 FP Vlans 100-199
Troubleshooting Tools: ELAM Example

S1# show system internal pixm info ltl 0x62 Get mapping of
source index to S101 S102 S201 S202
Member info
------------------ physical port
Type LTL vPC30 vPC40
---------------------------------
PHY_PORT Eth6/19 A ES S100 B C D
ES S200

S1# show system internal pixm info ltl 0x65 Get mapping of
Member info
destination index to
------------------ physical port
Type LTL
---------------------------------
PHY_PORT Eth6/22

• ELAM confirms that frame was received on Eth6/19, VLAN 100 with an OSA of
100.11.65535 and ODA of 200.0.0.
• ELAM also confirms that frame was forwarded out Eth6/22 on VLAN 100 with a
decremented FP TTL
Troubleshooting Tools: show tech

• show tech fabricpath isis

• show tech fabricpath switch-id
• show tech fabricpath topology
• Neither of these include FP routes, macs or comprehensive forwarding related
data. Collect these separately:
• show tech l2fm detail
• show tech l2fm l2dbg
• show tech forwarding l2 unicast
• show tech forwarding l2 multicast

84
Troubleshooting Example: Broken HSRP
S1 S2
• Problem statement: HSRP active & standby do not ‘see’ each other in
certain vlans. For example in vlan 1317 standby (S2) ‘sees’ the active (S1),
but on active standby is unknown. A number of vlans are affected. This is
new deployment. S3 S4

• Initial assessment: possible reason for HSRP router not ‘seeing’ other
router is HSRP hello packets not being received. In our case it is likely
active router, not receiving hello packets from standby
• Quick debug on S1 confirms it only sends hellos in vlan 1317
S1# debug hsrp engine packet hello interface vlan 1317
10:03:30 hsrp: Vlan1317[17/V4]: Hello out Active pri 100 ip 10.13.17.254
10:03:31 hsrp: Vlan1317[17/V4]: Hello out Active pri 100 ip 10.13.17.254
10:03:32 hsrp: Vlan1317[17/V4]: Hello out Active pri 100 ip 10.13.17.254

• …and on S2 we see hellos being sent and received…

S2# debug hsrp engine packet hello interface vlan 1317
10:03:30 hsrp: Vlan1317[17/V4]: Hello in from 10.13.17.1 State Active pri 100 ip 10.13.17.254
10:03:30 hsrp: Vlan1317[17/V4]: Hello out Standby pri 50 ip 10.13.17.254
10:03:31 hsrp: Vlan1317[17/V4]: Hello in from 10.13.17.1 State Active pri 100 ip 10.13.17.254
10:03:31 hsrp: Vlan1317[17/V4]: Hello out Standby pri 50 ip 10.13.17.254
 Troubleshooting Example: Broken HSRP
S1 S2
• Are the HSRP frames from S2 to S1 getting lost?
E1/1
S1# sh fabricpath load-balance multicast ftag-selected flow-type l2 dst-mac E1/1
0100.5e00.0002 src-mac 0000.0c07.ac11 ether-type 800 vlan 1317 module 1
... S3 S4
FTAG SELECTED IS : 1

S2# sh fabricpath load-balance multicast ftag-selected flow-type l2 dst-mac

0100.5e00.0002 src-mac 0000.0c00.0123 ether-type 800 vlan 1317 module 1
...
FTAG SELECTED IS : 2  S1S2 FTAG 1 traffic uses Po1
(peer-link)
• Findings so far:  S1S2 FTAG 2 traffic uses E1/1
(goes through S4)
• Working and Non-working packets may follow different paths
• Time to look at the Trees S2# show fabricpath isis trees
MT-0
S2# sh fabricpath isis topology summary Topology 0, Tree 1, Swid routing table
MT-0 1, L1
Configured interfaces: port- via port-channel1, metric 20
channel1 Ethernet1/1 Ethernet1/2 ...
Number of trees: 2
Tree id: 1, ftag: 1, root system: 0000.0000.0002, 2 Topology 0, Tree 2, Swid routing table
Tree id: 2, ftag: 2, root system: 0000.0000.0004, 4 1, L1
via Ethernet1/1, metric 40
...
 Troubleshooting Example: Broken HSRP
S1 S2
• S4 is transit switch for HSRP S2  S1 traffic, hence we will not see packets
E1/1
in debug. We need to look at the data plane level if hello packet
E1/1
arrives/leaves.
• Options: SPAN, Counters, ELAM S3 S4
• Let’s try hardware counters…

S4# show hardware internal errors module 1  CBL drops grow at about the rate of  Root cause: Vlan missing from transit switch
... HSRP hellos. CBL stands for Color  All FP vlans must be defined on all FP
Blocking logic (or Vlan Blocking
|------------------------------------------------------------------------| switches, otherwise there might be issues
| Device:Orion Fwding Driver Role:L2 Mod: 1 |
Logic). Essentially, hardware logic similar to this for flooded traffic. ISIS will
| Last cleared @ Thu Apr 11 11:11:11 2011
| Device Statistics Category :: ERROR defining whether given port/vlan is prune off unnecessary flood traffic towards
blocking or forwarding packets.
|------------------------------------------------------------------------| tree branches that do not have ports behind
Instance:0 them.
ID Name Value Ports
-- ---- ----- -----
29 smallcnt Pkt dropped due to CBL 0000000000001227 1-2 - S4# show fabricpath mroute vlan 1317
2014 Ingress packets marked with drop_oth sent to IB 0000000000001227 1 – ERROR: Vlan 1317 does not exist

S4# show hardware internal errors module 1 | diff S4# show vlan id 1317
...wait some seconds... VLAN 1317 not found in current VLAN database
S4# show hardware internal errors module 1 | diff
< 29 smallcnt Pkt dropped due to CBL 0000000000001229 1-2 –
> 29 smallcnt Pkt dropped due to CBL 0000000000001235 1-2 -
Troubleshooting: Common Pitfalls

• All FP Vlans must be present on all FP switches

• else multicast trees might not be correct
• TCNs not propagated to required FP or CE switches. Configure STP domain
where TCNs need to be propagated. Else, connectivity might be broken after
re-convergence until MACs age out or are relearned
• At power up or reload, CE-side comes up faster than FP-side
• L2GW Inconsistency, ensure that FP switches have been configured with
superior priority before connecting to CE switches.
CLI cheatsheet
• Interfaces in FP mode
show fabricpath isis interface [brief]

• ISIS adjacencies
show fabricpath isis adjacency [detail]

• Root information for the trees

show fabricpath isis topology summary

• RPF information for the trees

show fabricpath isis trees

• OIFs for the trees

show fabricpath mroute

• Affinity to Ftags
show fabricpath isis database detail
show system internal m2rib ftag

• Pong
pong destination-swid <sw#> destination-mac <mac-address> vlan <vlan> count <#> … [detail]
Summary
• Core Concepts
Known Unicast  Best path with ECMP, Rest  Tree-balanced
• Control Plane
ISIS in the core, STP / IGMP snooping at CE
• Data Plane
MAC address table, SwitchID table, Tree table (RPF)
• Troubleshooting
Understand what should be happening, verify what is happening,
find a deviation, zoom in and repeat

90
Thank you