Configuration Guide

Netvisor ONE

July 2018
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL
ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND
RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE
PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE
FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET
FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE
INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE
SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR PLURIBUS NETWORKS
REPRESENTATIVE FOR A COPY.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND
SOFTWARE ARE PROVIDED “AS IS” WITH ALL FAULTS. PLURIBUS NETWORKS DISCLAIMS
ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION, THOSE OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR
ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL PLURIBUS NETWORKS BE LIABLE FOR ANY INDIRECT, SPECIAL,
CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST
PROFITS OR LOSS OR DAMAGE TO DATA, ARISING OUT OF THE USE OR INABILITY TO USE
THIS MANUAL, EVEN IF PLURIBUS NETWORKS HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
Any Internet Protocol (IP) addresses used in this document are not intended to be actual
addresses. Any examples, command display output, and figures included in the document
are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content
is unintentional and coincidental.
COPYRIGHT © 2018 PLURIBUS NETWORKS, INC. ALL RIGHTS RESERVED. NETVISOR, NVOS, VFLOW,
VLAG AND VNM, ARE REGISTERED TRADEMARKS, AND THE PLURIBUS NETWORKS LOGO, PLURIBUS
NETWORKS, ONVL, PLURIBUSCARE, FREEDOMCARE, ADAPTIVE CLOUD FABRIC, FREEDOM, UNUM AND
INSIGHT ANALYTICS ARE TRADEMARKS OF PLURIBUS NETWORKS, INC. ALL OTHER BRANDS AND PRODUCT
NAMES ARE REGISTERED AND UNREGISTERED TRADEMARKS OF THEIR RESPECTIVE OWNERS.

Version 3.0.0 March 2018

Version 2.6.2 October 2017
Version 2.6.0 August 2017

Pluribus Networks
www.pluribusnetworks.com
2
 3
Pluribus Networks
www.pluribusnetworks.com
Table of Contents
Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Documentation Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Obtaining Documentation and Submitting a Service Request . . . . . . . . . . . 3
About the Netvisor ONE CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Important Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Entering Commands and Getting Help . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Finding Command Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Alternate Command Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Specifying IP Address Netmasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Specifying Measurement Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Customizing Show Output Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Specifying a Switch or Fabric for Command Scope . . . . . . . . . . . . . . . . . . 9
Installing Netvisor ONE and Initial Configuration . . . . . . . . . . . . . .10
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Using the Serial Console Port for Initial Configuration . . . . . . . . . . . . . . 11
Autoconfiguration of IPv6 Addresses on the Management Interface Support . . . . 13
Changes to the End User License Agreement (EULA) . . . . . . . . . . . . . . . . . . . 14
Zero-Touch Provisioning Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Transport Layer Security Protocol 1.2 Support . . . . . . . . . . . . . . . . . . . . . . . 19
GREP Support for Netvisor OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Running Commands on a Local Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Changing Other Switch Setup Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Setting the Date and Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Changing the Default Timezone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Viewing User Sessions on a Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Confirming Connectivity on the Network . . . . . . . . . . . . . . . . . . . . . . . 22
Adding License Keys to Netvisor OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Enabling Administrative Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Modifying and Upgrading Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Displaying and Managing Boot Environment Information . . . . . . . . . . . . . . . . . 31
Rolling Back to Previous Versions of Netvisor . . . . . . . . . . . . . . . . . . . . . . . . 31
Creating Switch Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Managing RMAs for Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
RMA Use Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Configuring Port Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . .36
Displaying Port Numbering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Configuring Ports for Different Throughput . . . . . . . . . . . . . . . . . . . . . 37
Displaying Port Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Displaying Port Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Using Port Buffering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Auto-Recovery of a Disabled Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Loop-Free Layer 2 Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Managing Control Plane Traffic Protection (CPTP) . . . . . . . . . . . . . . . . . 44

pluribusnetworks.com i
 Enhancements for Control Plan Traffic Protection . . . . . . . . . . . . . . . . . . . . 46
Additional Control Plane Traffic Protection Enhancements . . . . . . . . . . . . . . . 48
Display Physical Port Layer 2 Information . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Configuring Minimum and Maximum Bandwidth on Ports . . . . . . . . . . . . . . . . 51
Changes to Class of Service (CoS) Behavior . . . . . . . . . . . . . . . . . . . . . . . . . 52
Configuring Port Storm Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Enabling Jumbo Frame Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
About Port Isolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Example Use-Case Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Configuring Port Isolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
Support for Priority-based Flow Control . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Support for Priority-based Flow Control Port Statistics . . . . . . . . . . . . . . . . . 59
Support for Fabric Guard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Introducing Netvisor ONE Foundational Objects . . . . . . . . . . . . . .61
About the Netvisor ONE Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Creating an Initial Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Adding Switches to an Existing Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Fabric Over Management Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Displaying Fabric Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Displaying Information about Nodes in the Fabric . . . . . . . . . . . . . . . . . . . . . 65
About Fabric Transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Rolling Back and Rolling Forward Transactions . . . . . . . . . . . . . . . . . . . . . . . 67
Configuring Ports for Fabric Communication . . . . . . . . . . . . . . . . . . . . . . . . 69
Configuring Link Aggregation Control Protocol (LACP) . . . . . . . . . . . . . . 70
Active-Standby Link Aggregation on Management Interfaces . . . . . . . . . . . . . . 71
Configuring Trunking for Link Aggregation (LAG) . . . . . . . . . . . . . . . . . . . . . 71
High Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Configuring a Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Modifying a Trunk or VLAG Configuration by Changing the LACP Mode . . . . . . . 76
Safely Restoring Ports for Cluster Configurations . . . . . . . . . . . . . . . . . . . . . 77
Configuring Layer 2 Multipathing for Virtual Chassis Link Aggregation (VLAG) 79
VLAG Topology Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Configuring Active-Active VLAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Routing over VLAGs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Configuring Virtual Wire Features . . . . . . . . . . . . . . . . . . . . . . . .87
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Enabling Virtual Wire Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Configuring Ports for Virtual Wire Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Implementing Unidirectional and Bidirectional Virtual Wire Links . . . . . . . . . . 92
Support for CRC Checks for Virtual Wire Mode . . . . . . . . . . . . . . . . . . . . . . . 94
Support for Many to One Port Associations . . . . . . . . . . . . . . . . . . . . . . . . . 95
Packet Load Balancing over One to Many Links . . . . . . . . . . . . . . . . . . . . . . . 95
Building a Virtual Wire Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Example: Configuring a Fabric for Virtual Wire Switches . . . . . . . . . . . . . . . . 98
Example: Configuring a Fabric for Unidirectional Virtual Wire . . . . . . . . . . . . . 99
Inline Services for Virtual Wire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .101
Configuring and Displaying Statistics . . . . . . . . . . . . . . . . . . . . . . . . . 106
Adding VCF-Insight Analytics for Network Visibility . . . . . . . . . . . . . . . . . . . .108
Configuring Layer 2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . 110
Configuring Tagged and Untagged VLANs . . . . . . . . . . . . . . . . . . . . . . 110

ii pluribusnetworks.com
 Reserved VLANs and VLAN 0 and 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .110
Displaying VLAN Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112
Configuring Rapid Spanning Tree Protocol (RSTP) . . . . . . . . . . . . . . . . 114
Fast Failover for STP and Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .116
Active-Active VLAG Forwarding with Loopback Recirculation . . . . . . . . . . . . .117
Multiple Spanning Tree Protocol (MSTP) . . . . . . . . . . . . . . . . . . . . . . . 119
About Port Hairpinning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
Command Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .120
Configuring VXLANs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
About VXLANs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Configuring VXLANs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .124
Configuring VXLANs and Tunnels . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Configuring a VXLAN with Netvisor ONE . . . . . . . . . . . . . . . . . . . . . . . . . . .126
Configuration Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .126
Creating Tunnels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .128
Egress ECMP Load Distribution for VXLAN Traffic from the VTEP Switch . . . . . . 129
VXLAN Routing In and Out of Tunnels . . . . . . . . . . . . . . . . . . . . . . . . 131
VXLAN Port Termination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Virtual Link Extension with Cluster Configurations . . . . . . . . . . . . . . . 134
Virtual Link Static Bidirectional Association . . . . . . . . . . . . . . . . . . . . . . . . .137
Port Replication for Virtual Link Extensions . . . . . . . . . . . . . . . . . . . . 137
Support for Configuring Keep-Alive Time for Virtual Link Extension (VLE) . . . . . 138
Support for Virtual Link Extension (VLE) Analytics . . . . . . . . . . . . . . . . . . . .139
Configuring Layer 3 Features . . . . . . . . . . . . . . . . . . . . . . . . . . 140
Configuring vRouter Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
IPv6 Hardware Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
IPv6 Neighbor Discovery Process Support and Optimization . . . . . . . . . . . . . .142
Displaying Hardware Routes History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .145
Configuring MTU Parameters for vRouter Interfaces . . . . . . . . . . . . . . . . . . .145
Support for IPv4 and IPv6 on a vRouter Interface . . . . . . . . . . . . . . . . . . . . .145
IPv6 Support for vRouter Loopback Addresses . . . . . . . . . . . . . . . . . . . . . . .147
Configuring Prefix Lists for BGP and OSPF . . . . . . . . . . . . . . . . . . . . . . . . . .148
Configuring Packet Relay for DHCP Servers . . . . . . . . . . . . . . . . . . . . . . . . .149
Configuring Hardware Routing for a vRouter . . . . . . . . . . . . . . . . . . . . . . . .149
Support for Displaying Quagga Routing and Debug Information for vRouters . . . 150
Viewing Quagga Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .151
Support for Hardware vRouter Migration . . . . . . . . . . . . . . . . . . . . . . . . . .151
Configuring BGP on a vRouter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
Additional BGP Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .155
Support for BGP SNMP MIBs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .156
Support for AS and AS Prepending and BGP . . . . . . . . . . . . . . . . . . . . . . . . .156
Bidirectional Forwarding Detection Support for IPv6 BGP Neighbor and IPv6 Static Routes 157
Support for Border Gateway Protocol (BGP) Communities . . . . . . . . . . . . . . .157
Configuring Open Shortest Path First (OSPF) . . . . . . . . . . . . . . . . . . . . 160
Display Default Timers for OSPF Configurations . . . . . . . . . . . . . . . . . . . . . .162
Adding Areas and Prefix Lists to OSPF . . . . . . . . . . . . . . . . . . . . . . . . . . . . .164
BFD Support for OSPF Fault Direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . .164
Support for Route Maps for OSPF Routes . . . . . . . . . . . . . . . . . . . . . . . . . . .166
Support for OSPF SNMP MIBs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .167
Adding Default Route Information Settings for OSPF Routing . . . . . . . . . . . . .167
Adding Metric and Metric Type for Route Maps . . . . . . . . . . . . . . . . . . . . . .169

pluribusnetworks.com iii
 Configuring Routing Information Protocol (RIP) . . . . . . . . . . . . . . . . . . 170
Configuring Static Routes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .171
Support for Bidirectional Forwarding Detection on Static Routes . . . . . . . . . . 172
Adding IPv6 Link-Local Addresses for Static Routing . . . . . . . . . . . . . . . . . . .173
Configuring Multicast Listener Discovery (MLD) . . . . . . . . . . . . . . . . . . 174
MLD Snooping of IPv6 Neighbors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .176
Multicast Listener Discovery (MLD) Snooping per VLAN . . . . . . . . . . . . . . . . .176
Creating MLD Static Sources and Static Groups . . . . . . . . . . . . . . . . . . . . . . .177
Displaying MLD Statistics for a VLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .178
Configuring Virtual Router Redundancy Protocol . . . . . . . . . . . . . . . . . 178
Configuring the VRRP ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .179
Example Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .179
Layer 3 Table Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
Active-Active VLAG with Link-State Layer 3 Unicast Protocols . . . . . . . . 183
Using an L3 Network to Establish the Netvisor Fabric . . . . . . . . . . . . . . 183
Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .183
Multiple Neighbors over a Layer 3 Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . .186
Support for Bidirectional Forwarding Detection (BFD) and Static Routes . . . . . . 187
Support for Policy-based Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .187
Cluster Active-Active Routing Support for IPv6 Addresses . . . . . . . . . . . . . . .188
Support for PIM Source Specific Multicast (PIM-SSM) Forwarding . . . . . . . . . . .189
Virtual Routing and Forwarding (VRF) Support . . . . . . . . . . . . . . . . . . . . . . .193
Configuring Virtual Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Implementing Virtual Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Specifying the Type of VNET Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . .198
Creating a Virtual Network (VNET) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .199
VNET High Availability (HA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .199
Configuring Network Security . . . . . . . . . . . . . . . . . . . . . . . . . 202
Creating and Implementing Access Control Lists (ACLs) . . . . . . . . . . . . 202
MAC ACLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
Using MAC ACLs to Deny Network Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . .202
Configuring a MAC ACL to Allow Network Traffic . . . . . . . . . . . . . . . . . . . . .204
Configuring a MAC ACL to Deny Network Traffic . . . . . . . . . . . . . . . . . . . . . .205
IP ACLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Using a Deny IP ACL to Block Network Traffic . . . . . . . . . . . . . . . . . . . . . . .205
Using IP ACLs to Allow Network Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . .207
Configuring IP ACLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .207
Configuring an Internal Deny ACL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .207
Configuring an External Deny ACL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .208
Configuring an External Allow IP ACL . . . . . . . . . . . . . . . . . . . . . . . . . . . . .208
Support for DHCP Snooping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .209
Support for Router Advertisement (RA) Guard . . . . . . . . . . . . . . . . . . . . . . .211
Administering your Switches and Fabric . . . . . . . . . . . . . . . . . . 215
Fabric Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
Using the Fabric Transaction Commands . . . . . . . . . . . . . . . . . . . . . . . . . . .215
Displaying Fabric Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .217
Troubleshooting the Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .217
Displaying System Statistics on a Switch . . . . . . . . . . . . . . . . . . . . . . . . . . .219
Configuring Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Sending Log Messages to Syslog Servers . . . . . . . . . . . . . . . . . . . . . . . . . . .222
Forwarding Log Files to an External Linux Server . . . . . . . . . . . . . . . . . . . . .223

iv pluribusnetworks.com
 Saving Diagnostic Files and Exporting to an External Server . . . . . . . . . . . . . .224
Using Facility Codes with Log Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . .224
Displaying Log Counters Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . .225
Viewing Log Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .225
Displaying Log Counters Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . .227
Displaying System Statistics on a Switch . . . . . . . . . . . . . . . . . . . . . . . . . . .227
Exceptions for Audit Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .227
Configuring SNMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
SNMP Communities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .229
Users and SNMPv3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .229
Modifying the SNMP Engine ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .231
Additional Supported MIBs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .235
Supported Traps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .235
Using Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
Configuring vFlow for Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
Using vFlows to Disable Communication . . . . . . . . . . . . . . . . . . . . . . . 242
Use Case Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .243
Configuring Mirroring for vFlows and Ports . . . . . . . . . . . . . . . . . . . . . . . . .244
Managing Traffic Classes with vFlow . . . . . . . . . . . . . . . . . . . . . . . . . 246
Applying CoS Queue Mapping based on Re-Marked DSCP in vFlow . . . . . . . . . . . 247
Displaying Multiple Objects for Show Commands . . . . . . . . . . . . . . . . . . . . .248
Support for Policy-based Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .249
Using Application Flows and Statistics . . . . . . . . . . . . . . . . . . . . . . . . 250
Displaying Standard Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .250
Understanding vFlow Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .251
Creating vFlows with the Scope Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . .254
Example Use Cases for vFlows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .254
Creating Multiple vFlows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .254
Configuring Bandwidth Sharing for a Single VLAN . . . . . . . . . . . . . . . . . . . . .255
Configuring vFlows in Virtual Wire Mode . . . . . . . . . . . . . . . . . . . . . . . . . .256
Support for TCP Parameters using vFlows . . . . . . . . . . . . . . . . . . . . . . . . . .256
Configuring vFlows with User Defined Fields (UDFs) . . . . . . . . . . . . . . . . . . .257
Configuring DSCP to CoS Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .259
Configuring Priority-based Flow Control . . . . . . . . . . . . . . . . . . . . . . . . . . .261
About sFlow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
Configuring the sFlow Collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .264
Enabling sFlow on the Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .264
Adding Additional Ports to sFlow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .265
Counter Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .265
Packet Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .265
Agent to Collector Datagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .265
Analyzing Live Traffic Using Wireshark . . . . . . . . . . . . . . . . . . . . . . . . . . . .267
Using Wireshark to Analyze Packets in Real Time . . . . . . . . . . . . . . . . 268
Internet Protocol Flow Information Export (IPFIX) . . . . . . . . . . . 270
IPFIX Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
IPFIX Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .270
IPFIX Collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .270
Information Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .271
Abstract Data Types Supported by IPFIX . . . . . . . . . . . . . . . . . . . . . . . . . . .272
Data Type Semantics Supported by IPFIX . . . . . . . . . . . . . . . . . . . . . . . . . . .273

pluribusnetworks.com v
 Information Elements Supported by Netvisor OS and IPFIX . . . . . . . . . . . . . . .274
Configuring IPFIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .283
Configuring vCenter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
vCenter Connection Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
Configuring a vCenter Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .286
Auto Provisioning for vCenter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .286
Automatic Link Aggregation on EXSi-facing Ports for vCenter . . . . . . . . . . . . .287
Support for VLAN Alarms in vCenter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .288
Configuring Open Virtual Switch . . . . . . . . . . . . . . . . . . . . . . . . 290
Configuring OVSDB with Netvisor OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .290
Using OpenSSL TLS Certificates for OVSDB and other Services . . . . . . . . . . . . .291
Open Virtual Switch Database (OVSDB) Error Reporting . . . . . . . . . . . . . . . . .294
Appendix A - Acknowledgments for Open Source Software . . . . . . 295

vi pluribusnetworks.com
Preface
Audience
Organization
Conventions
Documentation Feedback
Obtaining Documentation and Submitting a Service Request

Audience
This publication is for experienced network administrators responsible for configuring and
maintaining network switches with some expertise in the following areas:
Network Administration
Storage Administration
Server Administration
Application Delivery Administration
Network Security Administration

Organization
This publication is organized as follows:
Installing Pluribus Netvisor® ONE and Initial Configuration
Configuring Virtual Wire features
Introducing Netvisor ONE foundational object
Layer 2 features
Layer 3 features
Administering your switches and fabric
VNET and other virtual services
Using Analytics
Internet Protocol Flow Information (IPFIX)
Acknowledgments for Open Source Software

Conventions
This document uses the following conventions:
Convention Indication
Bold font Keywords, user interface elements, and user-entered text
appear in bold font.
Italic font Document titles, new or emphasized terms, and variables
that you supply values are in italic font.

Pluribus Networks
www.pluribusnetworks.com
1
Convention Indication
[] Elements in square brackets are optional.
{x|y|z} Required elements are grouped in curly braces and are
separated by vertical bars.
[x|y|z] Optional parameters are grouped in brackets and
separated by vertical bars.
String A non-quoted set of characters. Do not use quotation
marks around the string or the string includes the
quotation marks.
courier font Command Line Interface (CLI) commands and samples
appear in courier font.
<> Nonprinting characters such as passwords are indicated by
angle brackets.
[] Default responses to system prompts are in angle
brackets.
CLI >Indicates that you enter the following text at the
network-admin@switch
command prompt.

Informational Note:
Indicates information of special interest.

Caution!
Indicates a situation that could cause equipment failure or loss of data.

TIP!
Indicates information that can help you solve a problem.

Warning! Indicates information that could impact you or your network.

2
Pluribus Networks
www.pluribusnetworks.com
 Timesaver:
Indicates information that can help you save time.

Documentation Feedback
To provide technical feedback on this document, or to report an error or omission, please
send your comments to [email protected]. We appreciate your feedback.

Obtaining Documentation and Submitting a Service Request

For information on obtaining documentation, submitting a service request, and gathering
additional information, please visit www.pluribusnetworks.com/support.

Pluribus Networks
www.pluribusnetworks.com
3
About the Netvisor ONE CLI
This chapter provides information for understanding and using the Pluribus Networks
Netvisor ONE command line interface (CLI) on a Netvisor ONE switch.
 Important Terms
 Entering Commands and Getting Help
 Finding Command Options
 Specifying IP Address Netmasks
 Specifying Measurement Units
 Customizing Show Output Formats
 Specifying a Switch or Fabric for Command Scope

Important Terms
The following list of important terms and concepts as well as definitions is important for
understanding Netvisor ONE features and determine the best configuration to meet your
needs.
Term Meaning
API Application Programming Interface to the Netvisor OS
OS switch. It has a similar scope as the CLI.
CLI Command Line Interface to the Netvisor OS OS
switch. Depending on the command, it can be
executed for an individual switch, a cluster, or a
fabric.
Cluster A pair of Netvisor OS OS switches configured as a
high availability group. You can configure a number of
clusters in the fabric, but a switch can be a member
of one cluster.
Fabric A set of Netvisor OS switches configured as a single
entity. Any switch can only be a member of one
fabric.
Flow NetFlow identifies packet flows for both ingress and
egress IP packets and provides statistics based on
these packet flows. NetFlow does not require any
change to either the packets themselves or to any
networking device.
In-band Management The IP address of the switch on a production or
Address management network for administration and
inter-switch communication.
LACP Link Aggregation Control Protocol allows a
non-Netvisor OS device to have multiple connections
to the same switch, for example, IEEE 802.3ad
trunks.

Pluribus Networks
www.pluribusnetworks.com
5
Term Meaning
vFlow A logical, manageable connection within or
throughout the fabric.
VLAG Virtual Link Aggregation Group is the Netvisor OS
method for multiple connecting hosts to multiple
switches, switches to each other, and switches to
other switches.

Entering Commands and Getting Help

Enter commands, options, and arguments at the CLI prompt. A command name must be
typed, but the included command-completion and help features contribute to the command
entry process.
To display a list of commands used within a command mode, enter a question mark (?), or
use the tab key, or type help at the command prompt. Display keywords and arguments for
each command with this context-sensitive help feature. Use complete commands and display
keywords and arguments for each command using the tab key to assist with
context-sensitive command help and completion.
Table 1 lists the command entered to get help specific to a command, keyword, or argument.

Table 1: Getting Help

abbreviated- command-entry? Displays a list of commands that begin with a

specific character string. Do not leave a space
between the string and question mark.
abbreviated- command-entry Completes a partial command name.
<tab>
? Lists all commands.
command ? Lists all keywords for the command. Leave a
space between the command and the
question mark.
command keyword ? Lists all arguments for the keyword. Leave a
space between the command and the
question mark.

Where a text string is used, such as name-string, the following characters are allowed as
part of the text string: a-z, A-Z, 0-9, _ (underscore), . (period), , (comma), : (colon), and -
(dash).

Informational Note: If you enter an invalid command, then using the ? and tab
key have no effect and do not return any changes to the CLI.

6
Pluribus Networks
www.pluribusnetworks.com
 Informational Note: The CLI contains an editing ability similar to UNIX
and Linux functionality using emacs keys. For example, p steps backward
through previous commands, n moves to the next command in the
history, a moves to the first character in the command and e moves to
the end of the line, u erases the current line, and w erases the previous
word.
Informational Note: Also, use the up and down arrows on your keyboard
to retrieve the last command entered at the CLI.

Finding Command Options

The syntax consists of optional or required keywords. Enter a question mark (?) at the
command prompt or after entering part of a command followed by a space to display
keywords for a command. Netvisor CLI displays a list of available keywords along with a brief
description of the keywords. For example, to display all of the keywords for the command
user, enter user ?.
Table 2, Finding Command Options, displays examples of using the question mark (?) to
assist you with entering commands.

Table 2: Finding Command Options

CLI network-admin@switch > ? Displays a list of commands that begin with

a specific character string. Do not leave a
All commands: space between the string and question mark.
acl-ip-create
acl-ip-delete
...
Switch> user auth Completes a partial command name.
User: <user>
Password: <password>
? Lists all commands.
command ? Lists all keywords for the command. Leave a
space between the command and the
question mark.
command option ? Lists all arguments for the option. Leave a
space between the command and the
question mark.

Informational Note: Other useful options, especially for displaying

statistics, include sort, interval, duration, and show diff interval.

Pluribus Networks
www.pluribusnetworks.com
7
Alternate Command Format
The CLI has an alternate command format in that the commands start with a verb instead of
a noun. This format omits the hyphen in the command names. For example,
connection-stats-show can also be entered as show connection-stats. The command
formats have the same features and can be used interchangeably.

Specifying IP Address Netmasks

Some commands require you to specify an IP address netmask. Pluribus Networks Netvisor
ONE supports both CIDR and subnet notations.
For example, you specify the range of IP addresses from 192.168.0.0 to 192.168.0.255 by
either entering 192.160.0.0 for the IP address input for a CLI command or either 24 or
255.255.255.0 for the netmask.

Specifying Measurement Units

Many commands include input and output of capacity and throughput. Netvisor ONE displays
network values in bits and storage values in bytes. Scale factors are allowed on input and
displayed in output as well as shown in Table 3, “Scale Numbers”.

Table 3: Scale Numbers

Scale Indicator Meaning (Networking) Meaning (Storage)
K or k Kilobits Kilobytes
M or m Megabits Megabytes
G or g Gigabits Gigabytes
T or t Terabits Terabytes

Customizing Show Output Formats

Customize the output generated by the show commands by using optional arguments
described in Table 4, “Show Output Formats”.

Table 4: Show Output Formats

format Displays only the columns matching the list of column header names.
<column_name1>, NOTE: The list of column names is comma-separated without spaces.
<column_name2>,
<column_nameX>

8
Pluribus Networks
www.pluribusnetworks.com
Table 4: Show Output Formats

format all Displays all available column headers. This output is also called
verbose mode.
By default, show commands output a terse set of the most commonly
useful column headers.
parsable-delim Displays the output of show command by separating columns by the
<separator> specified <separator> character(s).
For example, parsable-delim , produces a comma-separated output
(CSV).
NOTE: If the parsable-delim option is specified, the column header
names (titles) are suppressed from the output.

Specifying a Switch or Fabric for Command Scope

Netvisor uses the switch as the building block of a fabric, and the goal of the Netvisor ONE
design is an easy to manage fabric of switches managed as a single switch. The CLI runs
commands on the local switch, a cluster of switches, other switches in the fabric, or the
entire fabric. You do not log into each switch to run commands.
By default, commands run on the switch you logged into and for example, the command
port-config-modify port 5 disable disables port 5 on the switch you logged into on the
network.
To specify a different switch for a single command, use the switch prefix. For example,
switch pleiades23 port-config-modify port 28 enable enables port 28 on pleiades23,
even if the CLI connects to a different switch in the fabric.
To specify a different switch for a series of commands, use the switch prefix with no
command. For example, type switch pleiades24 <return>. The CLI prompt changes to
indicate that pleiades24 executes the commands. Run additional commands on pleiades24
rather than the physically connected switch.
For most CLI show commands, the command displays results from all switches in the fabric
by default. For example, when you enter the CLI command port-show on the switch,
Netvisor ONE displays all ports on all switches in the fabric.
To specify that a CLI show command applies to a specific switch, use the switch prefix to the
CLI command. For example, to display only ports on switch pleiades24, type the command
switch pleiades24 port-show.

Pluribus Networks
www.pluribusnetworks.com
9
Installing Netvisor ONE and Initial Configuration
 Using the Serial Console Port for Initial Configuration
 Autoconfiguration of IPv6 Addresses on the Management Interface Support
 Changes to the End User License Agreement (EULA)
 Zero-Touch Provisioning Support
 Transport Layer Security Protocol 1.2 Support
 GREP Support for Netvisor OS
 Running Commands on a Local Switch
 Setting the Date and Time
 Configuring Administrative Session Timeout
 Viewing User Sessions on a Switch
 Confirming Connectivity on the Network
 Adding License Keys to Netvisor OS
 Enabling Administrative Services
 Modifying and Upgrading Software
 Implementing a Fabric Upgrade or a “Rolling” Fabric Upgrade
 Saving and Restoring Netvisor ONE Configurations
 Copying and Importing Configuration Files
 Exporting Configurations Using Secure Copy Protocol (SCP)
 Displaying and Managing Boot Environment Information
 Rolling Back to Previous Versions of Netvisor
 Creating Switch Groups
 Support for Enabling or Disabling LLDP
 Managing RMAs for Switches

Overview
This section contains information about initial configuration of your switch as well as
commands to manage, upgrade, and restoring Netvisor ONE configurations.

10
Pluribus Networks
www.pluribusnetworks.com
Using the Serial Console Port for Initial Configuration
This procedure assumes that you have installed the switch in the desired location and it is
powered on.

Do not connect any ports to the network until the switch is configured.
You can accidentally create loops or cause IP address conflicts on the
network.

If you are going to cable host computers to the switch, there is an option to enable or disable
host ports by default.
1. Connect the console port on the rear or front (depending on the model) of the switch to
your laptop or terminal concentrator using a serial cable.
2. From the terminal emulator application on your computer, log into the switch with the
username network-admin and the default password admin.
3. Begin initial configuration using the initialization procedure displayed:

Informational Note: IPv6 addresses are supported for the in-band interface.

Warning: Be sure to type in a static IP address for the management interface

during the initial configuration. Netvisor OS initially uses DHCP to
obtain an IP address, but DHCP is not supported after the initial
configuration.

switch console login: network-admin

Password: admin
Last login: Fri Oct 3 12:23:04 on console
Command Line Interface v2.5.4
System setup required:
System Name (switch): hostname <return>
network-admin Password: password <return>
Re-enter Password:****** <return>
Enable mgmt link aggregation[disable|enable|active-standby] (disable):
active-standby
This might reset SSH connections after the setup.Are you Sure? (no): yes
<return>
Mgmt IP/Netmask: ip-address/netmask <return>
Mgmt IP Assignment:
Mgmt IPv6: IPv6-address/netmask
Mgmt IPv6 Assignment:
In-band IP/Netmask: ip-address/netmask
In-band IPv6/Netmask:
In-band IPv6 Assignment:
Gateway IP (0.0.0.0): 192.168.100.254 <return> or ip-address
Gateway IPv6 ():
Primary DNS IP (0.0.0.0): 192.168.100.253 <return> or ip-address

Pluribus Networks
www.pluribusnetworks.com
11
Secondary DNS IP (0.0.0.0): 192.168.200.253 <return> or ip-address
Domain name (some-domain.com): domain-name <return>
NTP Server:
Secondary NTP Server:
Timezone:
EULA accepted:
EULA timestamp:
Date:
Automatically Upload Diagnostics (yes): <return>
Enable host ports by default (yes): <return>
nvOS system info:
Switch Setup:
Switch Name: T6001
Switch Mgmt IP: 192.168.100.1/24
Mgmt IP assignment: static
Switch Mgmt IPv6: 2001:0db8:85a3:0000:0000:8a2e:0370:7334
MGMT IPv6 assignment: autoconf
Mgmt Link State: up
Mgmt Link Speed: 1g
Switch In-band IP: 192.168.200.1/24
Switch In-band IPv6: 2001:0db8:85a3:0000:0000:8a2e:0370:7315
Switch Gateway: 192.168.100.254
Switch DNS Server: 192.168.100.254
Switch DNS2 Server: 192.168.100.253
Switch Domain Name: pluribusnetworks.com
Switch NTP Server: 0.north-america.pool.ntp.org
Switch NTP Secondary-server 1.north-america.pool.ntp.org
Switch Timezone: US/Pacific
Switch Date: 2017-05-03, 13:02:39
Phone Home: Yes
HostID: 184551182
Location ID: 1
Upload Diagnostics: yes
Enable host ports: yes
Analytics Store: default
Fabric required. Please use fabric-create/join/show
Connected to Switch; nvOS Identifier:0x000044; Ver: 0.19.3398

Informational Note: In order to use the phone home feature, you must
open ports 8084 and 8843 on your firewall.

When you setup a switch for initial configuration, disable host-facing ports until ready to plug
in host cables to the switch. If Netvisor does not detect adjacency on a port during the
quickstart procedure, the ports remain in the disabled state. To enable the ports after
plugging in cables, use the port-config-modify port port-number host-enable
command. Netvisor enables host ports by default unless you specify no during the
quickstart procedure.
Netvisor OS Command Line Interface 3.0
By ANSWERING "YES" TO THIS PROMPT YOU ACKNOWLEDGE THAT YOU HAVE READ THE
TERMS OF THE PLURIBUS NETWORKS END USER LICENSE AGREEMENT (EULA) AND AGREE TO
THEM. [YES | NO | EULA]?: yes

12
Pluribus Networks
www.pluribusnetworks.com
Switch setup required:
Switch Name (e68-leaf-01):
network-admin Password:
Re-enter Password:
Mgmt IP/Netmask (10.13.25.225/16):
In-band IP/Netmask (192.168.97.2/24):
Gateway IP (10.42.42.1):
Primary DNS IP (10.42.44.1):
Secondary DNS IP:
Domain name (pluribusnetworks.com):
Automatically Upload Diagnostics (yes):
Enable host ports by default (yes): no

LI (network-admin@e68-leaf-01) > port-show

switch port status config
------------ ---- ------------ ------
e68-leaf-01 25 phy-up,host-disabled 10g

CLI (network-admin@e68-leaf-01) >port-config-modify port 25 host-enable

CLI (network-admin@e68-leaf-01) > port-show

switch port status config
------------ ---- ------------ ------
e68-leaf-01 25 up 10g

With switch-setup Enable host ports mode set to no, all ports have this
port-config-setting set to no. This can be viewed using the following command:
CLI (network-admin@Spine1) > port-config-show format port,host-enable,
In this mode, when any port comes up physically, Netvisor OS automatically sends and
receives LLDP packets to look for peer switches. If LLDP packets are received and Netvisor
forms an adjacency, Netvisor OS continues normally. If Netvisor does not detect an
adjacency within in 5 seconds, Netvisor OS flags the port as host-disabled. With this flag set,
Netvisor only accepts LLDP packets and does not initiate packet transmission.
CLI (network-admin@Spine01) > port-show
switch port ip mac hostname status
-------- ---- --------
------------- --------------- -------- ------------------------------
Spine-01 34 192.168.97.4 66:0e:94:cc:ee:fc E68-pa up,PN-switch,PN-other,LLDP,
Spine-01 5 phy-up,host-disabled
config trunk
--------- --------
fd,10g auto-128

After completing switch discovery and fabric creation, enable host, server, or router traffic
switching, and enable the ports using the host-enable option:
CLI (network-admin@Leaf1)>port-config-modify port 5 host-enable
switch-setup-show displays enable-host-ports setting. You cannot change this global
switch setting after the first initial switch-setup is performed. However, configure individual
ports for host-enable or disable by using the port-config-modify command.

Autoconfiguration of IPv6 Addresses on the Management Interface Support

Pluribus Networks
www.pluribusnetworks.com
13
IPv6 Stateless Address Autoconfiguration (SLAAC)
Like IPv4 addresses, you configure hosts in a number of different ways for IPv6 addresses.
Dynamc Host Configuration Protocol (DHCP) assigns IPv4 addresses dynamically and static
addresses assign fixed IP addresses. DHCP provides a method of dynamically assigning
addresses, and provides a way to assign the host devices other service information like DNS
servers, domain names, and a number of different custom information.
SLAAC allows you to address a host based on a network prefix advertised from a local
network router using Router Advertisements (RA). RA messages are sent by default by IPv6
router. These messages are sent out periodically by the router and include information
including:
One or more IPv6 prefixes (Link-local scope)
Prefix lifetime information
Flag information
Default device information (Default router to use and its lifetime)
Netvisor enables SLAAC by default on the switch.
When you configure IPv6 address on the management interface during setup, the parameter,
assignment, has two options:
none — Disables IPv6 addresses.
autoconf — Configure the interface with SLAAC.

Changes to the End User License Agreement (EULA)

Currently, the Netvisor OS displays the EULA during switch setup.
Netvisor OS Command Line Interface 2.6
By ANSWERING "YES" TO THIS PROMPT YOU ACKNOWLEDGE THAT YOU HAVE READ THE TERMS
OF THE PLURIBUS NETWORKS END USER LICENSE AGREEMENT (EULA) AND AGREE TO THEM.
[YES | NO | EULA]?:

When you enter the EULA option, the output displays the complete EULA text. After this
action, you cannot confirm EULA acceptance again. In some cases, an integrator may have
accepted the EULA on behalf of the actual end user.
Netvisor ONE displays the EULA acceptance with a timestamp of the event:

CLI (network-admin@Spine1) >eula-show

End User License Agreement

Pluribus Networks, Inc.'s ("Pluribus", "we", or "us") software products are
designed to provide fabric networking and analytics solutions that simplify
operations, reduce operating expenses, and introduce applications online more
rapidly. Before you download and/or use any of our software, whether alone or
as loaded on a piece of equipment, you will need to agree to the terms of this
End User License Agreement (this
"Agreement").
...
PN EULA v 2.1
accepted: true

14
Pluribus Networks
www.pluribusnetworks.com
Zero-Touch Provisioning Support
Use Zero Touch Provisioning (ZTP) to quickly bring up and deploy a configuration on a
Pluribus switch with no user interaction. Typically used in large-scale data center
deployments where the data center engineers simply rack the equipment and connect it to
the management network.
ZTP leverages an on-premise DHCP server where an administrator configures one or more
vendor-specific DHCP options that Netvisor OS interprets and configures the switch.
ZTP runs when Netvisor is started and is in setup mode. Netvisor searches for vendor specific
DHCP options (236 and 237),in addition to a few commonly used ones.
Phase 2 of ZTP allows you to bring up a new switch and automatically configure the required
switch-setup settings, in-band-ip, or port-specific settings.
As new switches are connected to the DHCP-enabled management network, the new switch
is provided the required configuration using DHCP options to connect and retrieve a script
(ZTP script) interpreted by Netvisor OS.
If the switch is in ‘setup’ mode, Netvisor OS discovers and runs the ZTP script using the
following algorithm:
1. local directory (/sftp/import/nv-ztp-installer)
2. directory of USB drive (i.e. /media/{drive}/nv-ztp-installer)
3. remote webserver (http://<host>/nv-ztp-installer)
In all of the above cases, the script must be named nv-ztp-installer. However, a complete
URL may be specified using DHCP option 236, in which case the complete path to the installer
may be specified. For example,
option Pluribus_ZTP_url “http://<server>/my_script”;

Also, if you use options 66 and/or 67, the script may be named option 67. If you do not use
option 67, Netvisor OS defaults to the name nv-ztp-installer. Additionally, the Pluribus
Networks Cloud developer portal encrypts and signs the script.

Pluribus Networks
www.pluribusnetworks.com
15
Upload the script and click Create Signed Package button. The portal then encrypts, signs, and
downloads to the your switch. Pluribus Network Cloud does not store the script..

Informational Note: Please contact Pluribus Networks for access to the

Developer Portal.

If Netvisor OS mode is in setup mode, ZTP discovery is triggered upon service startup. This is
the default mode for Netvisor OS.
The ZTP script contains a number of CLI commands that are interpreted in the order listed in
the script and issued to Netvisor OS as if you typed them at the CLI prompt.
The following sample script accepts the EULA, sets the inband-ip (based on DHCP option
237), name of the switch, DNS domain, and joins the fabric, corp-fabric:
#
# Configure the setup-related options first
#
switch-setup-modify eula-accepted true
switch-setup-modify in-band-ip %NV_ZTP_INBAND_IP%
--script-password switch-setup-modify password changeme
switch-setup-modify switch-name august

16
Pluribus Networks
www.pluribusnetworks.com
switch-setup-modify domain-name pluribusnetworks.com
#
# At this stage, nvOS is no longer in setup mode, other commands
# may now be used.
#
switch-setup-modify phone-home
--user network-admin:test123 fabric-join name corp-fabric
Any command used at the CLI prompt can also be used in a ZTP script. However, regular Unix
shell commands are not supported at this time and cause the script to fail.
When developing the script, Pluribus Networks recommends validating the script by first
executing the equivalent commands at the CLI prompt to ensure the proper sequence and
syntax. If any command fails, Netvisor ONE terminates the script.
The %NV_ZTP_INBAND_IP%, if used, replace the vendor-specific DHCP option 237. This allows
the DHCP server to control the in-band IP assignment in much the same way as controlling
management IP assignment by MAC. For example, the following DHCP server snippet sends
the inband-ip of 1.1.1.1 to my-switch:

host my-switch {
hardware ethernet 01:02:03:04:05:06;
option host-name "my-switch1";
option Pluribus_ZTP_inband_ip "1.1.1.1/24";
fixed-address 192.168.1.10;

Pluribus Networks
www.pluribusnetworks.com
17
Figure 4:ZTP Script Discovery

DHCP Options
The following options are queried and interrogated during ZTP discovery:
OPTION 54: DHCP server identifier
OPTION 66: TFTP server name
OPTION 67: Boot filename
OPTION 72: WWW server
OPTION 236: Pluribus ZTP URL (string)
OPTION 237: Pluribus ZTP Inband IP (string

SFTP Discovery
SFTP discovery checks for the presence of the ZTP installer (nv-ztp-installer) in the
directory: /sftp/import.

18
Pluribus Networks
www.pluribusnetworks.com
USB Discovery
USB discovery checks for the presence of the ZTP installer (nv-ztp-installer) on the root
directory of a removable drive. For Netvisor OS, USB drives are auto-mounted under
/media/{name of drive}.
HTTP Discovery
HTTP discovery uses the DHCP options above to find the ZTP script by performing a wget to
each of the options.
When performing HTTP discovery, Netvisor OS sends a number of HTTP headers with each
request. Specify the HTTP headers in the request to identify the client and platform to the
server. This allows the server-side to generate a dynamic response based on these client
parameters.
Netvisor OS sends the following HTTP headers during ZTP discovery:
 X-NV-ZTP-HOSTID: <hostid of switch>
 X-NV-ZTP-SERIAL: <serial # of switch>
 X-NV-ZTP-PLATFORM: <platform of switch>
Security Considerations
The script is encrypted and signed in the same way as Netvisor OS packages and can only be
decrypted by Netvisor OS.
Additionally, the signer is also verified and only scripts signed by Pluribus are run.

Transport Layer Security Protocol 1.2 Support

The TLS protocol provides communications security over the Internet. The protocol allows
client and server applications to communicate in a way designed to prevent eavesdropping,
tampering, or message forgery.

GREP Support for Netvisor OS

Netvisor ONE supports filtering output and allows switch administrators to filter output using
“grep|” from the CLI. This functionality is limited t
o the following commands:
running-config-show
tech-support-show
help

CLI > help | grep “openstack” lists all of the commands for OpenStack

Running Commands on a Local Switch

Run commands locally on a switch by using the switch-local parameter. For instance,
using switch-local port-stats-show displays output for the local switch ports only.

Pluribus Networks
www.pluribusnetworks.com
19
Changing Other Switch Setup Parameters
You can also modify other switch parameters including the following:
Switch name
Management IPv4 and IPv6 addresses
Management IPv4 and IPv6 netmasks
Management IPv4 and IPv6 address assignments
In-band IPv4 address
In-band netmask
Gateway IPv4 address
Gateway IPv6 address
Primary and secondary IPv4 addresses for DNS services
Domain name
NTP server
End User License Agreement (EULA) acceptance and timestamp
Password
Date
Phone home for software updates
Analytics store (storage type)
Message of the Day (MOTD)
Banner

Setting the Date and Time

To set the date and time on the switch, modify the switch configuration using the
switch-setup-modify command. To change the date and time to December 25, 2017,
12:00:00, use the following syntax:
CLI (network-admin@Leaf1)>switch-setup-modify date 2017-12-25T12:00:00
Display the configured setting using the switch-setup-show command:
CLI (network-admin@Leaf1)>switch-setup-show
switch-name: switch
mgmt-ip: 10.9.11.211/16
mgmt-ip6: fe80::3617:ebff:fef6:e2c4/64
mgmt-link-state: up
mgmt-link-speed: 1g
in-band-ip: 10.9.11.213/16
gateway-ip: 10.14.2.1
gateway-ip6: 2001:1000:1111:2222:3333:abcd:1000:2
dns-ip: 10.20.4.1
dns-secondary-ip: 10.20.40.1
domain-name: pluribusnetworks.com
ntp-server: 0.ubuntu.pool.ntp.org
timezone: America/Los_Angeles
date: 2017-12-25,01:06:47
phone-home: yes
hostid: 184551447
location-id: 1
analytics-store: default
enable-host-ports: yes

20
Pluribus Networks
www.pluribusnetworks.com
banner:

The analytics-store parameter refers to the storage location for analytics which in this
case is the hard drive on the switch. Netvisor ONE does not support external hard drives.

Changing the Default Timezone

By default, Netvosisor sets the default timezone toUS/Pacific Standard Time (PST). To
change the timezone, use the switch-setup-modify command:
CLI (network-admin@Leaf1)>switch-setup-modify timezone time-zone name

Configuring Administrative Session Timeout

By default, Netvisor sets the administrator sessions to unlimited session time, and set the
unlimited session time by configuirng the timeout to 0 seconds. The session timeout also
applies to using the shell command in Netvisor.
New commands support this feature:
(CLI network-admin@Spine1)>admin-session-timeout-modify
timeout duration: #d#h#m#s Specify Maximum time to wait for user activity
before terminating login session

(CLI network-admin@Spine1)>admin-session-timeout-show
switch: Spine1
timeout: 300s

Viewing User Sessions on a Switch

For security and troubleshooting, view user sessions on the switch. Netvisor now lists all
currently logged-in users and the IP of the connection and the login time when you execute
the command, mgt-session-show.
CLI (network-admin@Leaf1)>mgmt-session-show

user user-string Displays the user name.

cli-user cli-user-string Displays the name used to log into the switch.
pid pid-number Displays the process ID.
terminal terminal-string Displays the terminal.
from-ip ip-address Displays the IP address for the user.
login-time date/time: Displays the time and date that the user logged
yyyy-mm-ddTHH:mm:ss into the switch.
remote-node remote-node-string Displays the name of the remote node.
vnet vnet-string Displays the VNET assigned to the user.
type cli|api|shell Displays the type of login session.

CLI (network-admin@Leaf1)>mgmt-session-show
switch user cli-user pid terminal from-ip login-time
------------- ----- ------------- ----- -------- ------------ --------------
Spine-ext-41 admin network-admin 13805 pts/3 10.60.1.216 11:20:52
Spine-ext-41 root network-admin 8589 pts/2 10.14.20.109 11-15,17:16:17

Pluribus Networks
www.pluribusnetworks.com
21
Spine-ext-41 network-admin 08:24:10
Spine-ext-41 root 19139 pts/1 10.14.22.54 11-15,11:01:08

type
------
cli
cli
api
shell

Confirming Connectivity on the Network

After connecting your switch, take the time to ensure connectivity by pinging an external IP
address, and pinging a domain to ensure domain name resolution.
To ping the external network from the switch, use the ping command:
CLI (network-admin@Leaf1)>ping 98.138.253.109 : 56 data bytes
PING 98.138.253.109 (98.138.253.109) 56(84) bytes of data.
64 bytes from 98.138.253.109: icmp_seq=1 ttl=47 time=51.8 ms
64 bytes from 98.138.253.109: icmp_seq=2 ttl=47 time=51.9 ms
64 bytes from 98.138.253.109: icmp_seq=3 ttl=47 time=53.6 ms
Use the ping command again to ping a domain:
CLI (network-admin@Leaf1)>ping yahoo.com
PING yahoo.com (98.138.253.109) 56(84) bytes of data.
64 bytes from ir1.fp.vip.ne1.yahoo.com (98.138.253.109): icmp_seq=1 ttl=47
time=52.2 ms
64 bytes from ir1.fp.vip.ne1.yahoo.com (98.138.253.109): icmp_seq=2 ttl=47
time=52.5 ms
64 bytes from ir1.fp.vip.ne1.yahoo.com (98.138.253.109): icmp_seq=3 ttl=47
time=51.9 ms
64 bytes from ir1.fp.vip.ne1.yahoo.com (98.138.253.109): icmp_seq=4 ttl=47
time=51.8 ms

Adding License Keys to Netvisor OS

Netvisor binds the license key to the serial number of the switch and when downloading the
Netvisor software, the Pluribus Networks Cloud locates the serial number.
To install the license key, use the following syntax:
CLI network-admin@switch > software-license-install key license-key
The license key has the format of four words separated by commas. For example,

License Key: rental,deer,sonic,solace

Once you install the license key, display information about the key using the following
command:
CLI (network-admin@Leaf1)>software-license-show
switch: T6001-ON
license-id: NVOS-CLD-LIC-60D
description: Pluribus Open Netvisor OS Linux Cloud Edition License
expires-on: never
status: VALID

22
Pluribus Networks
www.pluribusnetworks.com
Enabling Administrative Services
There are many features of the Pluribus Networks fabric that require or can be enhanced
using remote access. For example, when Netvisor writes packets to a log file, transfer the file
from a switch to a different system for analysis. Also, if you create a NetVM environment, you
must load the OS of the guest OS on the switch.
Netvisor supports file transfer method SFTP.
SFTP is enabled by default. Because SFTP relies on Secure Shell (SSH), you must enable SSH
before enabling SFTP.
1. To check the status of SFTP, use the following command:
CLI (network-admin@Leaf1)>admin-service-show
switch: Leaf-2
if: mgmt
ssh: on
nfs: on
web: on
web-ssl: off
web-ssl-port: 443
web-port: 80
web-log: off
snmp: on
net-api: on
icmp: on
switch: techpub-accton-2
if: data
ssh: on
nfs: on
web: on
web-ssl: off
web-ssl-port: 443
web-port: 80
web-log: off
snmp: on
net-api: on
icmp: onf

2. To enable SSH, use the following command:

CLI (network-admin@Leaf1)>admin-service-modify nic mgmt ssh
admin-sftp-modify enable
sftp password: <password>
confirm sftp password: <password>

The default SFTP username is sftp and change the password using the
admin-sftp-modify command:
CLI (network-admin@Leaf1)>admin-sftp-modify
sftp password: <password>
confirm sftp password: <password>
CLI (network-admin@Leaf1)>admin-service-show
switch nic ssh nfs web web-port snmp net-api icmp
------ --- --- --- --- -------- ---- ------- ----
pleiades24 mgmt on on off 80 off off off

CLI (network-admin@Leaf1)>admin-sftp-show

Pluribus Networks
www.pluribusnetworks.com
23
switch: pleiades24
sftp-user: sftp
enable: yes

Use SFTP from a host to the switch, and login with the username sftp and the password
configured for SFTP. Then you can download the available files or upload files to the switch.
CLI (network-admin@Leaf1)>admin-service-show
switch nic ssh nfs web web-port snmp net-api icmp
------ --- --- --- --- -------- ---- ------- ----
pleiades01 mgmt on off on 80 off on on

Modifying and Upgrading Software

A switch contacts an upgrade server, either directly or through a proxy, to download and
upgrade to a newer version of Netvisor OS. Modify the upgrade process for the switch and
add a proxy host.

Informational Note:This upgrade procedure applies to only one switch. To

upgrade switches on the fabric or to create a “rolling upgrade” on
the fabric, see

What are Software Tracks?

Pluribus Networks manages different software releases using software tracks. By default, the
software track, release, is the standard track, but other tracks, such as Beta, may be
available for download.
CLI (network-admin@Leaf1)>software-modify phone-home

Updating Netvisor ONE on the Switch

Pluribus Networks switches send “phone home” messages to the Pluribus Networks update
servers to determine the availability of a new release of software.
1. To view the current version of Netvisor OS on the switch, use the following command:
CLI (network-admin@Leaf1)>software-show
version: 2.2.1-202016524
track: 2.2-release
upgrade-status: available
version-available: 2.2.0-202006524 -> 2.2.1-202016554
auto-upgrade: disable
use-proxy: no

2. If the upgrade status indicates the availability of a newer version of Netvisor ONE, request
an update from the server:
CLI (network-admin@Leaf1)>software-upgrade

upgrade successful. rebooting...

Check the status while the switch is upgrading, use the software-upgrade-status-show
command.
3. Check the status of the switch after upgrading, reconnect to the switch, and enter the fol-
lowing command:

24
Pluribus Networks
www.pluribusnetworks.com
CLI (network-admin@Leaf1)>software-show
version: 2.2.1-202016554
track: 2.2-release
upgrade-status: up-to-date
auto-upgrade: disable
use-proxy: no

Informational Note: Allow plenty of time for the switch to download and
install the new version of software. Do not interrupt the operation while the
upgrade is in progress. After completing the upgrade, the switch reboots
and loads the latest version of the software.
If you encounter any problems with the new version of the software, select
a previous versionof the boot software.

Informational Note: Upgrading without an Internet connection - If the

switch does not have direct access to the Internet but uses a proxy server,
enter the software-modify use-proxy command to configure the proxy
and then check for software upgrade availability. If no access to the
Internet from the switch, contact Pluribus Technical Support for instructions
on upgrading a switch offline.

To upgrade the current Netvisor OS to a later release, use the software-upgrade command.
CLI (network-admin@Leaf1)>software-upgrade package
nvos-2.3.1-203018600.tgz
The parameter package allows you to specify the name of the upgrade file.
To display information about the software upgrade path, you can use the
software-track-show command.

Implementing a Fabric Upgrade or a “Rolling” Fabric Upgrade

Netvisor implements a fabric-wide upgrade and reboot the switches at the same time or in a
sequential order. A fabric upgrade requires downloading the new Netvisor software package
to each switch, and rolling upgrade downloads the software packages from the update server
and then copies the software to each switch as the upgrade proceeds.
Issue the fabric-upgrade-start command on the upgrade controller. You must execute all
upgrade commands from the upgrade controller.
The fabric upgrade feature has two phases:
Upgrade — start the upgrade to create and update Netvisor to new boot
environments but does not reboot the fabric.
Reboot — reboots the entire fabric after all server-switches are upgraded to new boot
environments. Abort the process and discard the new boot environments during the
phase, if necessary.
Netvisor locks the fabric during the entire process and you cannot change any configurations
during the process.

Pluribus Networks
www.pluribusnetworks.com
25
Before You Begin the Fabric Upgrade
Before you begin, review the following options for the fabric-upgrade-start command:
auto-finish — specify to automatically reboot the entire fabric after completing the
upgrade.
rolling — specify if performing a rolling fabric upgrade. A rolling fabric upgrade
performs the upgrade procedure on a switch-by-switch basis and copies the software
package from the controller to other switches in the fabric. If you specify no-rolling,
Netvisor reboots all switches after the upgrade.
abort-on-failure — specify if the upgrade to stop if a failure occurs during the
process.
manual-reboot — specify if to manually reboot individual switches after the upgrade
process. If you specify no-manual-reboot, all switches reboot automatically after
completing the upgrade.
prepare — specify if to perform setup steps prior to performing the upgrade. This step
copies the offline software package and then extracts and prepares it for the final
upgrade process. Once you begin the prepare process, you cannot add new switches
to the fabric.
reboot-parallel — specify to reboot switches in parallel for a cluster configuration.
Or, you can reboot them one at time using the reboot-single option.
reboot-group — specify the number of switches to reboot as a group in parallel
mode. By default, a fabric consists of up to 100 switches msximum.

Starting the Fabric Upgrade

1. Download the latest Netvisor software from the update server onto a switch in the fabric.
2. Copy the Netvisor software package to each switch in the fabric.
3. Select a switch in the fabric to act as the upgrade controller switch, and use the fab-
ric-upgrade-start command to begin the upgrade.
4. Depending on the options selected, the upgrade completes by reboot the fabric or reboot-
ing all of the switches.

Starting the Rolling Fabric Upgrade

If you opted for a rolling fabric upgrade, then the upgrade controller switch begins copying to
software packages to other switches in the fabric. Other than this step, the rolling fabric
upgrade functions the same as a fabric upgrade depending on the selected options.
You can check the status of the upgrade using the fabric-upgrade-status-show command:
CLI (network-admin@sw1) > fabric-upgrade-status-show
log switch state
----------------------------------------------- -------- ------------------
(0:00:36)Upgrading software upgrade framework sw3 Running
(0:00:08)Computing package update requirements. sw2 Running
(0:00:12)Agent needs restart sw1* Agent restart wait

The first entry in the log is the duration of the upgrade process. It does not include waiting
time. The switch with the asterisk (*) is the controller server-switch where the
fabric-upgrade-start command was issued.

26
Pluribus Networks
www.pluribusnetworks.com
Additional commands for the fabric upgrade feature:
fabric-upgrade-finish — you can issue this command at any time during the fabric
upgrade to reboot all nodes in the fabric and complete the upgrade. Once the upgrade
phase is complete, all server-switches display the “Upgrade complete” message in the
log field. You can then safely reboot the fabric.
fabric-upgrade-abort — aborts the software upgrade process. All changes to the
server-switches are cleaned up and the server-switches do not reboot. The
configuration lock on the fabric is also released.
If you issue the fabric-upgrade-abort command during the upgrade process, it may take
some time before the process stops because the upgrade has to reach a logical
completion point before the changes are rolled back on the fabric. This allows the proper
cleanup of the changes.
fabric-upgrade-prepare-cancel — cancels a fabric upgrade prepared earlier.
fabric-upgrade-prepare-resume — resume a fabric upgrade prepared earlier.
fabric-upgrade-prepare-show — displays the status of prepared upgrades on the
fabric nodes.

Saving and Restoring Netvisor ONE Configurations

A switch contains local configuration information such as port settings as well as fabric
configuration information. Fabric configurations are stored on every switch in the fabric and
does not require that you save and restore before replacing a switch. When a switch is
replaced, removed, or otherwise disrupted, you can save and restore the local configuration
information.
The information that is saved and restored on the local switch includes the following:
VNETs with VNET manager running on the switch
Port VLAN associations
Network services running on the switch
To display a full list of the current configuration details for a switch, use the
running-config-show command.
Use SFTP to transfer the configuration file, but you must enable the features:

Caution! There is a potential for data loss when restoring a

configuration. The configuration on the switch is replaced by the
configuration stored in the import file. Although ISO images and
disk-library images are not likely to disappear, you should only perform
switch-config-import on a switch that doesn’t have important data
stored on it.
As a precaution, use the command switch-config-export to save the
data on the switch to import the configuration file.

1. Use the following command to save the switch configuration to a file:

CLI (network-admin@Leaf1)>switch-config-export export-file pleiades24
Exported configuration to /nvOS/export/pleiades24.2013-11-04T22.33.31.tar.gz

2. Use the following command to display the files available for import and export:
CLI (network-admin@Leaf1)>switch-config-show

Pluribus Networks
www.pluribusnetworks.com
27
switch export-file
pleiades24 pleiades24.2013-11-04T22.33.31.tar.gz

Now copy the configuration file to a different host using SFTP or NFS. For example, SFTP to
the switch-ip-address, and login using the SFTP password. Then use cd/nvOS/import,
and use get to download the configuration file.
The Netvisor command, switch-config-export exports the configuration of the local
switch. The file created is a tar file that includes a number of configuration files for the
switch. The file created under /nvOS/export. Also, each time you reset the switch using
the command, switch-config-reset, Netvisor retains a backup of the configuration and
places a file in the same location.
Once you export the switch configuration, use it to import on the same switch, by executing
the switch-config-copy-to-import command. Netvisor copies the configuration tar file
from the /nvOS/export to the /nvOS/import directory. Once in the /nvOS/import
directory, use the switch-config-import command to import the switch configuration.
The switch-config-import command is used to import a configuration on the local
switch. When using that command, the intention is to import a switch configuration t
previously exported by the same switch.
The switch-config-import command has a few parameters to it. The
ignore-system-config and the apply-system-config parameters are 2 parameters
that allow the imported configuration of the switch to override or not override the
currently configured information found under the switch-setup-show command. When
you select the ignore-system-config parameter, Netvisor s to an archive. If you select
apply-system-config, Netvisor applies the settings in the tar file to the local switch.
When you import a configuration using the switch-config-import command, Netvisor
over writes the current configuration on the switch with the imported configuration file.
The skip-fabric-join option imports the fabric configuration from the tar file.
However, this information may be out of date with respect to the fabric if transactions
occurred on the fabric since exporting the file causes the imported configuration to be
out-of-sync with the current fabric. Specify do-fabric-join, which extracts the fabric
name from the tar file, and attempts to join the fabric and download the current fabric
configuration, so that it synchs with the rest of the fabric. Netvisor ignores the fabric
configuration in the tar file, but imports cluster and local configurations from the tar
file.
When a switch that was part of a cluster is replaced, use the fabric-join
repeer-to-cluster-node command for the new switch to receive all required switch
configuration, including the local configuration.
To upload a configuration file to a switch and set the configuration for the switch using the
configuration file, you must transfer the configuration file to the target switch using the
following sequence of commands:
sftp sftp@<switch-ip-address>
Connecting to switch-ip-address
Password: <password>
sftp> cd nvOS/import
sftp> put pleiades24.2013-11-04T22.33.31.tar.gz

28
Pluribus Networks
www.pluribusnetworks.com
 Informational Note: The configuration file must use the *.tar.gz
extension to be recognized by nvOS.

CAUTION! Loading the configuration file causes nvOS to restart which results in
a brief interruption to switch traffic flow.

Now load the configuration file which replaces the current configuration on the switch with
the information in the file.
CLI (network-admin@Leaf1)>switch-config-import import-file
pleiades24.2013-11-04T22.33.31.tar.gz
New configuration imported. Restarting nvOS...
Connected to Switch pleiades24; nvOS Identifier:0xb000011; Ver: 0.19.3747

Netvisor provides many options to allow you to control how the switch-config-import
modifies the switch, including the following:
ignore-system-config - ignore the current system configuration.The settings in the
*.tar file are not applied to the local switch.
apply-system-config — apply the system configuration in the imported file. The
settings in the *.tar file are applied to the local switch. You typically do not want to
use this option as it changes the in-band IP address and other settings.
skip-fabric-join — opt out of joining the fabric. This setting imports the fabric
configuration from the *.tar file, but this information may be out of date with respect
to the fabric if additional transactions occur on the fabric since the file was exported.
do-fabric-join — join the current fabric. This setting extracts the fabric name from the
*.tar file and attempts to join the fabric. Then the switch contacts the current fabric to
download the configuration so that the switch is in sync with the rest of the fabric.
Cluster and local configurations are imported from the *.tar file.
no-replace-switch — do not replace the current switch.
replace-switch — replace the current switch. Use this setting to replace a faulty switch
and after importing the file, has the same configuration as the replaced switch. This
replaces all of the local, cluster, and fabric configuration by downloading the
configurations from peer switches. No configuration is necessary or advised before
running this command. However, you need to run the initial quickstart to obtain an
in-band IP address.

Pluribus Networks
www.pluribusnetworks.com
29
By default, the initial switch system configuration, management IP addresses and other
parameters, are not applied if there is another switch in the fabric with the same settings. To
apply the initial settings, use the apply-system-config option. Also, by default, the
imported configuration attempts to join the same fabric that the original switch was a
member. If that join fails, then the import fails. You can avoid this issue by using the
skip-fabric-join option. Finally, if the original switch is still on the network and you want
to copy the configuration to a new switch, but you want to prevent the new switch from
taking ownership of any objects specific to the original switch, such as VNET services, or
VLAN port settings, you must use the no-replace-switch option.

Copying and Importing Configuration Files

Create a configuration file to import to another switch by using the
switch-config-copy-to-import command. To create a configuration file with the name
config-092613 to import on another switch, use the following syntax:
CLI (network-admin@Leaf1)>switch-config-copy-to-import export-file
config-092613
After you create the configuration file, you can export it to /nvOS/export/ directory, and
SFTP to it from the target switch.
To review the available files for import and export, use the following syntax:
CLI (network-admin@Leaf1)>switch-config-show
switch export-file
pbg-nvos config-092613.tar.gz

Depending on the available remote access services, you copy the configuration file to a
different switch. For example, SFTP to another switch using the IP address of the switch,
login as SFTP with the previously set password, cd /nvOS/import and get the configuration
file.
To upload the configuration file to the target switch and set the configuration from the
configuration file, transfer the configuration file to the target switch with the IP address,
192.168.3.35.
To export a configuration to a server, use the switch-config-export command:
CLI (network-admin@Leaf1)>switch-config-export

Exporting Configurations Using Secure Copy Protocol (SCP)

The SCP is a network protocol based on the BSD RCP protocol supporting file transfers
between hosts on a network. SCP uses Secure Shell (SSH) for data transfer and uses the
same mechanisms for authentication, and ensures the authenticity and confidentiality of the
data in transit. A client uploads files to a server, optionally including basic attributes such as
permissions or timestamps. Clients also download files or directories from a server. SCP runs
over TCP port 22 by default. Like RCP, no RFC defines the specifics of the protocol.
In Netvisor, the CLI prompts for a password when you provide the upload-server option.
During the software upgrade process, Netvisor exports the switch configuration and moves it
to a shared directory. Access the exported configuration archive from all boot environments.
Netvisor exports the configuration before the start of the software upgrade.
Netvisor stores a maximum of three configuration archives on the switch and deletes older
configurations.

30
Pluribus Networks
www.pluribusnetworks.com
New parameters in Netvisor support this feature:
CLI (network-admin@Leaf1)>switch-config-export

export-file switch-config Specify the name of the file to export.

export-file
upgrade-location-mappings Specify the upgrade location mappings.
upgrade-location-mappings-string
Specify any of the following options:
upload-server upload-server-string Specify the name of the upload server.
server-password Specify the password for the upload
server-password-string server.

If you specify an upload server and password, Netvisor OS prompts you for that information
when you execute the software-upgrade command.

Displaying and Managing Boot Environment Information

Display information about the different boot environments on the switch. There are two boot
environments: the current boot environment, and the previous boot environment. To display
boot environment information, use the following command:
CLI (network-admin@Leaf1)>bootenv-show
name version current reboot space created
----------- ---------- ------- ------ ----- -------------------
netvisor-22 2.2.7-7356 no no 58.5M 2015-12-07,09:55:58
netvisor-23 2.3.1-8600 yes yes 27.4G 01-06,09:13:11

To reset the boot environment and reboot using the previous environment, use the following
syntax:
CLI (network-admin@Leaf1)>bootenv-activate-and-reboot name netvisor-22
To delete a boot environment, use the following syntax:
CLI (network-admin@Leaf1)>bootenv-delete name netvisor-22

Rolling Back to Previous Versions of Netvisor

After upgrading to a newer version of Netvisor, you can rollback to an earlier version and
preserve the current configuration. Netvisor applies the new configuration before booting into
the previous environment so Netvisor retains critical ACLs and security vFlows when Netvisor
restarts.
A new parameter, apply-current-config, for the command, bootenv-active-and-reboot,
provides support for this feature.
Before rebooting, Netvisor copies the current boot environment transaction logs into the
target boot environment.
After rebooting, Netvisor performs the following:
Reads the copied transaction logs, and sorts all transactions by time, then scope,
fabric>cluster>local, and then the transaction ID.
Parses the list of all transactions from oldest to newest.

Pluribus Networks
www.pluribusnetworks.com
31
 If the current transaction ID for the scope is less, Netvisor rolls the transaction
forward, and deletes the files when done.

Retaining the current configuration when booting to an older version of Netvisor

is best-effort. Some transaction IDs from the newer (or current) version may not
properly apply due to feature incompatibility. It is not guaranteed that all changes
are applied.

You must apply the parameter, apply-current-config, on all nodes in the

fabric. There is no coordination across the fabric for this process, therefore the
commitment of fabric transactions on one node but not another using this process
causes the fabric to go out of sync and may result in unrecoverable errors.

Creating Switch Groups

Create switch groups on your network, and you create as many switch groups as needed.
Provide a name to a group of switches, and a switch can be a member of more than one
group.
When you add an offline switch to a group,the configuration fails for that switch. Netvisor
adds online switches normally.
Switch groups are static and you must manually remove a switch from a group. You cannot
use a switch name for the switch group name and Netvisor displays a warning message due
to the invalid configuration.
New Commands
(CLI network-admin@Spine1)>switch-group-create

name name-string Specify a name for the switch group.

description description-string Specify a description for the switch group.
To create a switch-group with the name, rack-1-row-1, use the following syntax:
CLI (network-admin@Leaf1)>switch-group-create name rack-1-row-1
description datacenter rack 1
(CLI network-admin@Spine1)>switch-group-delete

name name-string Specify a name for the switch group.

description description-string Specify a description for the switch group.
To delete a switch-group with the name, rack-1-row-1, use the following syntax:
CLI (network-admin@Leaf1)>switch-group-delete name rack-1-row-1
description datacenter rack 1
(CLI network-admin@Spine1)>switch-group-modify

name name-string Specify a name for the switch group.

description description-string Specify a description for the switch group.
To modify a switch-group with the name, rack-1-row-1, and change the description, use the
following syntax:

32
Pluribus Networks
www.pluribusnetworks.com
CLI (network-admin@Leaf1)>switch-group-modify name rack-1-row-1
description datacenter
(CLI network-admin@Spine1)>switch-group-show

name name-string Displays the name of the switch group.

description description-string Displays a description of the switch group.
To display a switch-group with the name, rack-1-row-1, use the following syntax:
CLI (network-admin@Leaf1)>rack-1-row-1 datacenterswitch-group-show
name description
------------ -----------
rack-1-row-1 datacenter

Adding Switches to Switch-Groups

(CLI network-admin@Spine1)>switch-group-member-add

name name-string Specify the name of the switch group to add the
member.
member fabric-node name Specify the name of the switch to add as a member.
To add switch, Leaf-1, to switch-group, rack-1-row-1, use the following syntax:
CLI (network-admin@Leaf1)>switch-group-member-add name rack-1-row-1
member Leaf-1
(CLI network-admin@Spine1)>switch-group-member-remove

name name-string Specify the name of the switch group to remove the
member.
member fabric-node name Specify the name of the switch to remove as a
member.
To remove switch, Leaf-1, from switch-group, rack-1-row-1, use the following syntax:
CLI (network-admin@Leaf1)>switch-group-member-remove name rack-1-row-1
member Leaf-1
(CLI network-admin@Spine1)>switch-group-member-show

name name-string Displays the name of the switch group.

member fabric-node name Displays the name of the switches in a group.
To display switch-group, use the following syntax:
CLI (network-admin@Leaf1)>switch-group-member-show
switch name member
-------- ------------ -------------
Spine-1 rack-1-row-1 Leaf-1

Support for Enabling or Disabling LLDP

This feature provides for a generic LLDP ON/OFF toggle function set at the system level.
Currently, to disable LLDP on a switch you must disable the LLDP configuration on all ports.
This resets all related configurations of LLDP protocol setting and LLDP vFlows.
Use the following CLI command to enable and disable the protocol:

Pluribus Networks
www.pluribusnetworks.com
33
CLI (network-admin@Leaf1)>system-settings-modify [lldp|no-lldp]
LLDP packets are executed on the CPU with the help of LLDP vFlows.
To clear all LLDP protocol system flows use the parameter no-lldp.
To add all LLDP protocol system flows use the parameter lldp.
This approach does not disturb port LLDP configurations
CLI (network-admin@Leaf1)>system-settings-show
switch: Spine1
optimize-arps: on
lldp: on

Managing RMAs for Switches

RMA Use Case

Informational Note: This process applies to Version 2.5.4 and earlier.

A primary case for an RMA is a failed switch in the network. Netvisor restores the
configuration to a replacement switch using the following commands:
 fabric-join

 fabric-join repeer-to-cluster-node
 switch-config-import

RMA Process
This procedure assumes a failed switch is part of a HA pair (cluster). Nodes part of a cluster
automatically back up the other configuration.
For an RMA case, the host ID differs between the new switch and the old failed switch.
Netvisor ties both cluster membership and service object locations to the host ID.
1. Retrieve the host id of the old node:
CLI> fabric-node-show name <old-hostname> format name,id
2. Evict the old node from the fabric. This allows Netvisor to process fabric provisioning oper-
ations before completing the RMA. Additionally, the presence of the old node ID interferes
with subsequent steps.
CLI> fabric-node-evict name <old-hostname>
3. Setup the new switch with basic settings, such as hostname and IP address.
Perform this step at the console when booting the switch for the first time:
CLI> switch-setup-modify
4. Configure the new switch to rejoin the fabric. As it is part of a cluster, use the
repeer-to-cluster-node option.
CLI> fabric-join name <fabric-name> repeer-to-cluster-node
<existing-peer-name>
Netvisor downloads the entire backed up configuration from the cluster peer and restarts
Netvisor ONE to apply it. The process restores local, cluster, and fabric scoped configuration.

34
Pluribus Networks
www.pluribusnetworks.com
5. After restart, any service objects present on the failed switch, must be migrated to the
new host. Use the value retrieved in Step 1 for the location parameter:

CLI network-admin@switch > object-location-modify location <old-hostid>

new-location <new-hostname>

The above command executes a bulk migration of all service objects (vRouters, VNET
managers, OVSDB Interfaces) and sub-objects

RMA Process for Version 2.6.0 and Later

Netvisor OS fabric creates objects such as vRouters, VLAGs, clusters, and others on a switch
in the fabric. Netvisor OS tracks the switch using a location field, currently the host ID of the
switch where the fabric objects are configured.
This presents various issues when replacing a faulty switch with a new switch and a new host
ID. Fabric-wide configurations that reference the old host ID requires updating to the new
host ID. These updates require a few manual extra steps and are either confusing, or it isn’t
clear what commands need execution.
Netvisor changes the location from a host ID to a fabric-specific location id assigned to each
switch as the switch joins the fabric. Netvisor OS keeps the same ID during the RMA process
and reduces the RMA process to a single command.
Netvisor supports a new parameter, location-id, unique among the fabric nodes. Each node
that joins the fabric is assigned a new location ID when it joins. All configurations require a
location tied to the location ID instead of the host ID. When Netvisor OS executes the
command, switch-config-import, the location ID inherits the ID from the imported
configuration. Therefore, no updates required across the fabric because all configurations
refer to the correct location ID.
The following commands no longer restore an imported configuration on a new switch:
cluster-membership-modify
fabric-node-evict
object-location-modify

A new parameter, location-id, added to the commands, node-info and fabric-node-show

output. This displays the location of the node.
A new command, fabric-node-location-mappings, displays the current fabric host ID to
the location ID mappings. The location uses the input for the command,
switch-config-import, when importing configurations from earlier versions of software.
If you import a configuration from an earlier version of software, use the following syntax:
(CLI network-admin@Spine1)>switch-config-import upgrade-location-mappings

If the imported configuration already has location IDs, Netvisor ignores the parameter.

Pluribus Networks
www.pluribusnetworks.com
35
Configuring Port Attributes
 Displaying Port Numbering
 Configuring Ports for Different Throughput
 Displaying Port Status
 Displaying Port Statistics
 Auto-Recovery of a Disabled Port
 Loop-Free Layer 2 Topology
 Managing Control Plane Traffic Protection (CPTP)
 Enhancements for Control Plan Traffic Protection
 Additional Control Plane Traffic Protection Enhancements
 Display Physical Port Layer 2 Information
 Displaying Transceiver Information
 Configuring Minimum and Maximum Bandwidth on Ports
 Changes to Class of Service (CoS) Behavior
 Configuring Port Storm Control
 Enabling Jumbo Frame Support
 About Port Isolation
 Support for Priority-based Flow Control
 Support for Priority-based Flow Control Port Statistics


 Limiting the Number of MAC Addresses per Port

 Support for Fabric Guard

Displaying Port Numbering

To display port numbering on white box Netvisor ONE platforms, use the
bezel-portmap-show command:

CLI network-admin@Leaf1 > bezel-portmap-show

switch port bezel-intf

-------------- ---- ----------
Leaf1 1 1
Leaf1 2 2
Leaf1 3 3
Leaf1 4 4
Leaf1 5 5
...
Leaf1 49 49
Leaf1 50 49.2

36
Pluribus Networks
www.pluribusnetworks.com
Leaf1 51 49.3
Leaf1 52 49.4
Leaf1 53 50
Leaf1 54 50.2
Leaf1 55 50.3
Leaf1 56 50.4

Configuring Ports for Different Throughput

By default, ports on the switches are configured as 40GbE ports. You use them as 4 x 10GbE
with the right transceiver. To refer to the 40Gb port, use the last port number of the port
group. For example, the first 40Gb port, in the example above, Netvisor refers to as port 49
for 40GbE use and as ports 49, 50, 51, and 52 for 4/10Gb use.
To change the 40Gb port to 4x10Gb functionality, use the following command sequence:

CLI network-admin@Leaf1 > port-config-modify port 49-52 speed 10g

To change the port back to 40Gb operation, use the following command sequence:

CLI network-admin@Leaf1 > port-config-modify port 49 speed 40g

Netvisor sets the default port speed to 10G and you can modify the parameters of a port:
Speed - you can disable the port or set the speed to 10m, 100m, 1g, 2.5g, 10g, or
40g.
Egress rate — limit the egress rate or set to unlimited.
Ethernet mode type — set the mode type to 1000base-x, sgmii, autonegotiate
Enable or disable a port
LACP priority — between 1 and 65535
Reflect — received frames are reflected for loopback testing.
Edge-switch — Specify if the port connects to another Netvisor ONE device or
uplinks to a third-party switch or host.
Pause — pause traffic on the port.
Description — description of the port
Loopback — specify loopback
Mirror-receive — receive mirrored traffic only.
MAC address — specify a MAC address for the port.
VLAG failover — specify if the port is used in VLAG failover.
Sending port number — specify if the port number sends traffic.

Pluribus Networks
www.pluribusnetworks.com
37
Displaying Port Status
Use the port-show command to display status information on all ports with active links.
Details for each port include the IP addresses and MAC addresses of hosts connected to that
port. More than one host if a network device such as a switch connects. The command also
displays the VLAN of the port, port status, and configuration details.
To display all port information for ports 1-6 on the switch, use the command, port-show
port 1-6:

CLI network-admin@Leaf1 > port-show port 1-6

switch port bezel-port vnet l2-net hostname status config

-------- ---- ---------- ---- ------ -------- ------ ------
Leaf1 1 1 10g
Leaf1 2 2 10g
Leaf1 3 3 10g
Leaf1 4 4 10g
Leaf1 5 5 10g
Leaf1 6 6 10g

Displaying Port Statistics

Display statistics for all active ports on the switch. This information is useful for
understanding the traffic on the ports.
Use the port-stats-show command to display the information:

CLI network-admin@Leaf1 > port-stats-show port 0 format all layout vertical

switch: dorado05
time: 11:32:41
port: 0
description:
counter: 0
ibytes: 2.82G
ibits: 24.2G
iUpkts: 176M
iBpkts: 0
iMpkts: 0
iPauseFs: 0
iCongDrops: 0
idiscards: 7
ierrs: 0
obytes: 884M
obits: 7.42G
oUpkts: 13.0M
oBpkts: 0
oMpkts: 0
oPauseFs: 0
oCongDrops: 1.89G
odiscards: 1.89G
oerrs: 0
mtu-errs: 0
HER-pkts: 0
HER-bytes: 0
port-speed: disable

38
Pluribus Networks
www.pluribusnetworks.com
The output headers have the following meaning:
switch — switch name
time — the time of the command
port — port number
counter — number of counters for the port
ibytes — number of incoming bytes in K (Kilobytes), M (Megabytes), or G (Gigabytes)
iUpkts — number of incoming unicast packets
iBpkts — number of incoming broadcast packets
iPauseFs — number of incoming paused fragmented packets
iCongDrops — number of incoming packets dropped due to congestion
idiscards — number of discarded incoming packets
ierrs — number of incoming packets with errors
obytes — number of outgoing bytes K (Kilobytes), M (Megabytes), or G (Gigabytes)
oUpkts — number of outgoing unicast packets
oBpkts — number of outgoing broadcast packets
oMpkts — number of outgoing multicast packets
oPauseFs — number of outgoing paused fragmented packets
oCongDrops — number of outgoing packets dropped due to congestion
odiscards — number of discarded outgoing packets
oerrs — number of outgoing packets with errors
mtu-errs — number of MTU errors
HER-pkts — number of VLE port packets
HER-bytes — number of VLE bytes

Using Port Buffering

Modify and display the port buffering settings for the switch ports. To display the port
buffering settings, use the port-buffer-settings-show command:

CLI network-admin@Leaf1 > port-buffer-settings-show

switch: Spine1
enable: yes
interval: 1m
disk-space: 50M

To modify port buffering settings, use the port-buffer-settings-modify command:

CLI network-admin@Leaf1 > port-buffer-settings-modify interval 2m

Modify the buffer interval, duration, disk space, and enable or disable port buffering on the
switch.
To display the port buffer, use the port-buffer-show command:

CLI network-admin@Leaf1 > port-buffer-show

switch: Spine1

Pluribus Networks
www.pluribusnetworks.com
39
port: 0
ingress-used-buf: 0%
ingress-used-buf-val: 0
egress-used-buf: 0%
egress-used-buf-val: 0
switch: Spine1
port: 3
ingress-used-buf: 0%
ingress-used-buf-val: 0
egress-used-buf: 0%
egress-used-buf-val: 0
switch: Pleiades24
port: 57
ingress-used-buf: 0%
ingress-used-buf-val: 0
egress-used-buf: 0%
egress-used-buf-val: 0
switch: Spine1
port: 65
ingress-used-buf: 0%
ingress-used-buf-val: 0
egress-used-buf: 0%
egress-used-buf-val: 0
switch: Spine2
port: 0
ingress-used-buf: 0%
ingress-used-buf-val: 0
egress-used-buf: 0%
egress-used-buf-val: 0
switch: Spine2
port: 1
ingress-used-buf: 0%
ingress-used-buf-val: 0
egress-used-buf: 0%
egress-used-buf-val: 0

Auto-Recovery of a Disabled Port

Netvisor automatically disables physical ports due to certain violations. For example, if a port
receives BPDU messages from an edge port, Netvisor ONE disables the port because
receiving BPDUs on a edge port becomes a security violation. However, Netvisor does not
indicate that the shut down port i because of a violation and not because of physical link
issues.
The port may be disabled due to the following errors:
BPDU Guard Messages

The port is set to Err disabled when a BPDU Guard message is received on an Edge
port.
The port is configured with BPDU Guard enabled.
Pluribus Networks switches supports BPDU Guard on Edge ports.
Link Flaps
There are too many link flaps for a configured interval of time.
MAC address Security Violation

40
Pluribus Networks
www.pluribusnetworks.com
 The number of MAC addresses on an interface is greater than the configured limit.
This feature allows you to configure an automatic retry to enable the port after a
configured timeout.

CLI network-admin@Leaf1 > err-disable-counters-clear

bpduguard|no-bpduguard Specify if you want BPDU guard enabled.

macsecurity|no-macsecurity Specify if you want MAC recovery enabled.
recovery-timer duration: Specify the recovery time value. The default timer
#d#h#m#s value is 5 minutes

CLI network-admin@Leaf1 > err-disable-modify

bpduguard|no-bpduguard Specify if you want BPDU guard enabled.

macsecurity|no-macsecurity Specify if you want MAC recovery enabled.
recovery-timer duration: Specify the recovery time value. The default timer
#d#h#m#s value is 5 minutes.

CLI network-admin@Leaf1 > err-disable-show

bpduguard|no-bpduguard Display BPDU guard settings.

macsecurity|no-macsecurity Display MAC recovery settings.
recovery-timer duration: Display the recovery time value.
#d#h#m#s

CLI network-admin@Leaf1 > err-disable-show

switch: Leaf1
bpduguard: off
macsecurity: off
recovery-timer: 5m

Loop-Free Layer 2 Topology

Netvisor ONE Loop Detection operates in conjunction with Rapid Spanning Tree Protocol
(RSTP) and Multiple Spanning Tree Protocol (MSTP). Netvisor uses RSTP and MSTP to ensure
loop free topology of the VLANs in the Layer 2 network.
RSTP prevents loops in the network caused by miscabled networking equipment, but does
not address misconfigured hosts. Netvisor ONE Loop Detection goes beyond STP to protect
the network from misconfigured or miscabled hosts attached to the network
Netvisor ONE Control Plane — The Netvisor ONE control plane includes information about
every MAC address attached to the Layer 2 network in a vPort database. Netvisor distributes
the vPort database throughout the fabric so that each Netvisor ONE switch contains a copy of
the vPort database for the entire fabric.
A MAC address stored in a vPort, includes the following information:
MAC address, VLAN ID, and VXLAN ID
Owner-port and local-port

Pluribus Networks
www.pluribusnetworks.com
41
 Migration history including owner, time, and port
vPort state as active, static, moving, or loop-probe
Access to the Netvisor ONE fabric goes through the Netvisor ONE software. Netvisor ONE
determines if endpoints access the network based on control plane data structures including
the vPort database.

Detecting Loops
Netvisor implements Netvisor ONE Loop Detection as part of Netvisor ONE source MAC
address miss handling. Netvisor ONE disables hardware learning of MAC addresses, when a
packet arrives with an unknown MAC address, the switch sends the packet to Netvisor ONE
rather than switching the packet normally. Netvisor ONE examines the vPort table to
determine if a packet with an unknown MAC indicates a loop.
Netvisor ONE uses two criteria to detect a loop on the network:
A MAC address associated with an in-band NIC of a node in the fabric appears as the
source MAC on a packet that ingresses on a host port. Netvisor ONE detects this
situation by noting the PN-internal status of a vPort migrating to a host port. Netvisor
ONE prevents the migration to take place and starts loop mitigation.
For the purposes of Netvisor ONE Loop Detection, Netvisor defines a host port as a port not
connected to another Pluribus switch, not an internal port, and disables participation in STP
with Netvisor ONE. Netvisor ONE disables STP on the switch and the device connected on the
port.
Packets with the same source MAC address arrive on multiple host ports in the fabric
at approximately the same time. In order to support VM and host migration, Netvisor
tolerates some rapid movement of MAC addresses through the fabric. When the same
MAC address moves rapidly back and forth between two ports, Netvisors detects a
loop and loop mitigation starts.
VRRP MAC addresses do not participate Loop Detection and Mitigation, and migrate freely.
Netvisor detects loops on a port by port basis. A single loop typically involves two ports,
either on the same switch or on two different switches. When multiple loops occur with more
than two ports then Netvisor responds to each port separately.

Loop Mitigation
When Netvisor detects a loop, a message appears in the system log indicating the host port
and VLAN involved in the loop. In addition the host port involved in the loop has the "loop"
status added and Netvisor ONE adds the VLAN to the host port loop-vlans VLAN map.
Looping ports and VLANs are displayed in the port-show output.
At the start of loop mitigation, Netvisor ONE creates vPorts to send loop probe packets. The
vPorts use the port MAC address for the in-band NIC port, status of PN-internal, and a state
of loop-probe. Netvisor propagates Loop-probe vPorts throughout the fabric. Netvisor ONE
creates a loop-probe vPort for each looping VLAN.
At the start of loop mitigation Netvisor ONE deletes all vPorts from the looping host port and
VLAN. This prevents the hardware from sending unicast packets to the looping port, and
causes every packet arriving on the looping port to appear in the software as a source MAC
miss. During loop mitigation, Netvisor drops all packets arriving on the looping port.

42
Pluribus Networks
www.pluribusnetworks.com
During loop mitigation, Netvisor ONE sends loop probe packets on the looping VLANs every 3
seconds. As long as the loop persists, Netvisor ONE receives the probe packets as source
MAC miss notification on the looping ports, so Netvisor ONE can determine if the loop is still
present. If 9 seconds elapse with no received probe packets, Netvisor ONE detects the loop is
resolved and ends loop mitigation.
At the end of loop mitigation, log messages are added the system log, loop-probe vPorts are
removed, and loop stats and loop VLANS are removed from the looping port.
To view affected ports, use the port-show command and add the parameter, status loop:
network-admin@switch-31>port-show status loop
switch port hostname status config
---------- ---- -------- --------------------- ------
switch-31 9 up,stp-edge-port,loop fd,10g
switch-32 9 up,stp-edge-port,loop fd,10g
Note the new status, loop, in the status column.
During loop mitigation, the MAC addresses for loop probes are displayed in the vPort table:
CLI (network-admin@switch-31) > vport-show state loop-probe
owner mac vlan ports state hostname status
---------- ----------------- ---- ----- ---------- ---------- -----------
switch-32 06:c0:00:16:f0:45 42 69 loop-probe leo-ext-32 PN-internal
switch-31 06:c0:00:19:c0:45 42 69 loop-probe leo-ext-31 PN-internal

Note the loop-probe state as well as the PN-internal state. The loop probes use the port
MAC address format, and use the internal port for the in-band NIC.
If you notice a disruption in the network, use the port-show command to find the looping
ports, and fix the loop. Fixing the loop typically involves correcting cabling issues, configuring
virtual switches, or as a stop-gap measure, using the port-config-modify command to
change port properties for the looping host ports. Once you resolve the loop, Netvisor ONE
no longer detects probes and leaves the loop mitigation state, while logging a message:
2016-01-12,12:18:41.911799-07:00 leo-ext-31 nvOSd(25695) system
host_port_loop_resolved(11381) : level=note : port=9 :
Traffic has stopped looping on host-port=9
At this point Netvisor removes the loop status from the port-show output for port 9 and
deletes the loop-probe vPorts.
Netvisor ONE Loop Detection exposes loops using system log messages, port-show output,
and vport-show output. Enable or disable Netvisor ONE Loop Detection by using the
system-settings-modify command:
network-admin@e68-leaf-01>system-settings-modify block-loops
network-admin@e68-leaf-01>system-settings-modify no-block-loops

When Netvisor ONE detects an internal port MAC address on a host port, Netvisor ONE prints
a log message:
system 2016-01-19,15:36:40.570184-07:00 mac_move_denied
11379 note MOVE DENIED mac=64:0e:94:c0:03:b3 vlan=1 vxlan=0
from switch=leo-ext-31 port=69 to deny-switch=leo-ext-31 deny-port=9
reason=internal MAC of local switch not allowed to change ports

Netvisor ONE starts Loop Mitigation by logging a message:

system 2016-01-19,15:36:40.570334-07:00 host_port_loop_detected
11380 warn Looping traffic detected on host-port=9
vlan=1. Traffic on this port/VLAN will be ignored until loop resolved

Pluribus Networks
www.pluribusnetworks.com
43
During Loop Mitigation, Netvisor ONE sends loop probes. When these probes, as well as any
other packets received on a looping host port, Netvisor ONE logs a message:

system 2016-01-19,15:59:54.734277-07:00 mac_move_denied

11379 note MOVE DENIED mac=06:c0:00:19:c0:45 vlan=1 vxlan=0
from switch=leo-ext-31 port=69 to deny-switch=leo-ext-31
deny-port=9 reason=port is looping

Netvisor ONE limits mac_move_denied messages to one every 5 seconds for each vPort. This
prevents the system log from filling up with mac_move_denied messages during loop
mitigation.
During loop mitigation, use the port-show command to see ports involved in the loop:

CLI network-admin@Leaf1 > port-show status loop

switch port hostname status loop-vlans config

---------- ---- -------- --------------------- ---------- ------
e68-leaf-01 9 up,stp-edge-port,loop 1 fd,10g
e68-leaf-01 9 up,stp-edge-port,loop 1 fd,10g

Note the loop status in the status column and the loop-vlans column.
During loop mitigation the MAC addresses for loop probes Netvisor displays the vPort table:

CLI network-admin@Leaf1 > vport-show state loop-probe,

owner mac vlan ports state hostname status

---------- ----------------- ---- ----- ---------- -------- ---------
--------
e68-leaf-01 06:c0:00:16:f0:45 42 69 loop-probe leo-ext-32
PN-internal
e68-leaf-01 06:c0:00:19:c0:45 42 69 loop-probe leo-ext-31
PN-internal

Managing Control Plane Traffic Protection (CPTP)

This feature is supported on the following

platforms:
• F9272-X • AS5712-54X
• F9232-C • AS6712-32X
• FF9372-T • AS5812-54T

44
Pluribus Networks
www.pluribusnetworks.com
 Nevisor supports this feature on the following
platforms:
• S4048-ON • Z9100-ON

Control Plane Traffic Protection (CPTP) applies to the internal control, data, and span ports
which all connect to the CPU, so CPTP protects the CPU resources from large quantities of
traffic arriving from different sources such as control packets, cluster communication, fabric
updates as well as the regular flood traffic, learning packets and copy-to-cpu packets.
CPTP classifies the traffic on the hardware to different Class of Service (CoS), and performs
priority scheduling between them, and also applies a rate limit for each of the CoS, to protect
the CPU resources and at the same time, provide a Service Level Agreement (SLA) for critical
traffic.

CLI network-admin@Leaf1 > port-cos-rate-setting-show

switch port port-number cos0-rate cos1-rate cos2-rate cos3-rate cos4-rate cos5-rate cos6-rate
--------- ----- ----------- --------- --------- --------- --------- --------- --------- --------
Spine1 pci-e 0 100 100 1000000 1000000 1000000 1000000 1000000
Spine1 data 65 100 100 1000000 1000000 1000000 1000000 1000000
Spine1 span 66 100 100 1000000 1000000 1000000 1000000 1000000

On ONVL switches, display the following output :

CLI network-admin@switch > port-cos-rate-setting-show

switch: Leaf1
port: control-port
ports: 0
cos0-rate(pps): 5000
cos1-rate(pps): 5000
cos2-rate(pps): 5000
cos3-rate(pps): 5000
cos4-rate(pps): 5000
cos5-rate(pps): 5000
cos6-rate(pps): 5000
cos7-rate(pps): 5000

Modify the CoS rate settings using the port-cos-rate-setting-modify command. Netvisor
sets the rate limits in packets per second.

CLI network-admin@Leaf1 > port-cos-stats-show

switch: Spine1
time: 11:59:15
port: 0
cos0-out: 58.8M
cos0-drops: 180M
cos1-out: 58.8M
cos1-drops: 185M
cos2-out: 0
cos2-drops: 0
cos3-out: 0
cos3-drops: 0
cos4-out: 0

Pluribus Networks
www.pluribusnetworks.com
45
cos4-drops: 0
cos5-out: 0
cos5-drops: 0
cos6-out: 65.5M
cos6-drops: 1.06G
cos7-out: 483K
cos7-drops: 493MTo clear the statistics for CoS on the ports, use the
port-cos-stats-clear command.

Enhancements for Control Plan Traffic Protection

This enhancement to Control Plane Traffic Protection (CPTP) provides 44 queues to further
strengthen CPU protection and limits the traffic going to the CPU. Currently, only 8 Class of
Service (CoS) queues are supported for flow control on a physical port. Each traffic class with
a CPU destination has a separate vFlow. All system vFlows with the parameters, to-cpu or
copy-to-cpu, now have an additional cpu-cos value.

CLI network-admin@Leaf1 > cpu-class-show

switch name scope rate-limit queue

-------- ------------- ----- ---------- -----
Spine1 stp local 1000 8
Spine1 lacp local 1000 9
Spine1 system-d local 1000 10
Spine1 igmp local 1000 11
Spine1 bcast local 1000 12
Spine1 icmpv6 local 1000 13
Spine1 tcp-analytics local 1000 14
Spine1 fabric local 1000 15
Spine1 kpalv local 1000 16
Spine1 ecp local 1000 17
Spine1 arp local 1000 18
Spine1 lldp local 1000 19
Spine1 vport-stats local 1000 20
Spine1 dhcp local 1000 21
Spine1 pim local 1000 22
Spine1 local-subnet local 1000 23
Spine1 bgp local 1000 24
Spine1 ospf local 1000 25

Netvisor assigns all DHCP traffic to a separate CoS queue, 21, and reserves CoS 0-7 CPU
queues. Any traffic not in one of the listed classes uses queue 0.
Netvisor ONE assigns a default rate-limit of 1000 to each queue, and you modify the rate
using the following syntax:

CLI network-admin@Leaf1 > cpu-class-modify cpu-class-name DHCP rate-limit 2000

Restart Netvisor ONE for the change to take effect on the switch. Modify any or all traffic
classes at one time and then reboot the switch once.

46
Pluribus Networks
www.pluribusnetworks.com
Configuring User-defined Classes
1. Create a CPU class and specify the rate-limit:

CLI network-admin@Leaf1 > cpu-class-create name ftp rate 1000

Netvisor ONE assigns a CoS class to the new CPU class.

2. Display the CPU class configuration:

CLI network-admin@Leaf1 > cpu-class-show name ftp

name queue rate

----- ----- -----
ftp 17 1000

3. Create a vFlow using the ftp class:

CLI network-admin@Leaf1 > vflow-create name ftp scope local proto ftp cpu-class
ftp action copy-to-cpu

Netvisor validates the vFlow only if you add the cpu-class parameter and specify the
action copy-to-cpu or to-cpu.

Display statistics for each vFlow using the command, cpu-cos-stats-show:

CLI network-admin@Leaf1 > cpu-cos-stats-show

switch name cos out-pkts drop-pkts

-------- ------------- --- -------- ---------
Spine1 class0 0 0 0
Spine1 class1 1 0 0
Spine1 class2 2 0 0
Spine1 class3 3 0 0
Spine1 class4 4 0 0
Spine1 class5 5 0 0
Spine1 class6 6 0 0
Spine1 class7 7 0 0
Spine1 stp 8 298K 0
Spine1 lacp 9 0 0
Spine1 system-d 10 0 0
Spine1 igmp 11 35.1K 0
Spine1 bcast 12 0 0
Spine1 icmpv6 13 0 0
Spine1 tcp-analytics 14 0 0
Spine1 fabric 15 5.02K 0
Spine1 kpalv 16 75.4K 0
Spine1 ecp 17 0 0
Spine1 arp 18 3.02K 0
Spine1 lldp 19 15.1K 0
Spine1 vport-stats 20 0 0
Spine1 dhcp 21 0 0
Spine1 pim 22 0 0
Spine1 local-subnet 23 31.0K 0
Spine1 bgp 24 0 0

Pluribus Networks
www.pluribusnetworks.com
47
Spine1 ospf 25 0 0
Spine1 ftp 26 0 0

Additional Control Plane Traffic Protection Enhancements

Additional Control Plane Traffic Protection (CPTP) enhancements to a new feature allowing
you to impose rate limits on the flow of traffic arriving on the CPU management port.
Currently, Netvisor does not protect control plane traffic arriving out-of-band on the
management NIC of the switch. Receiving excessive control plane traffic may saturate the 1G
management port or starve the CPU of other critical traffic.
Restrict the ingress traffic types on a port used as a management interface, and drop packets
exceeding a configured bandwidth limit.
Modify the settings for traffic sent to the management NIC. Currently, you can manage the
following types of traffic:
ARP
ICMP
SSH
SNMP
Fabric
NFS
Web
Web-SSL
NET-API
Netvisor disables this feature by default.
Manage the settings using the following Netvisor ONE commands:

CLI network-admin@Leaf1 > cpu-mgmt-class-modify

name Select the class of traffic to modify.

arp|icmp|ssh|snmp|fabric|bcast|
nfs|web|web-ssl|net-api
one or more of the following options:
rate-limit unlimited Specify the ingress rate limit on the
management port in bps or unlimited.
burst-size default Specify the ingress traffic burst size in bytes
or default.

CLI network-admin@Leaf1 > cpu-mgmt-class-show

name Displays the class of traffic.

arp|icmp|ssh|snmp|fabric|bcast|
nfs|web|web-ssl|net-api
one or more of the following options:

48
Pluribus Networks
www.pluribusnetworks.com
 rate-limit unlimited Displays the ingress rate limit on the
management port in Bps or unlimited.
burst-size default Displays the ingress traffic burst size in bytes
or default.

CLI network-admin@Leaf1 > cpu-mgmt-class-stats-settings-modify

enable|disable Specify if you want to enable statistics

collection.
interval duration: #d#h#m#s Specify the interval duration.
disk-space disk-space-number Specify the amount of disk space for the
statistics.

CLI network-admin@Leaf1 > cpu-mgmt-class-stats-settings-show

enable|disable Displays if statistics collection is enabled or

disabled.
interval duration: #d#h#m#s Displays the interval duration.
disk-space disk-space-number Displays the amount of disk space for the
statistics.

CLI network-admin@Leaf1 > cpu-mgmt-class-stats-show

time date/time: Displays the time to start collection.

yyyy-mm-ddTHH:mm:ss
start-time date/time: Displays the start time of collection.
yyyy-mm-ddTHH:mm:ss
end-time date/time: Displays the end time of collection.
yyyy-mm-ddTHH:mm:ss
duration duration: #d#h#m#s Displays the duration of collection.
interval duration: #d#h#m#s Displays the interval between collection.
since-start Displays the statistics collected since the
start time.
older-than duration: #d#h#m#s Displays the statistics older than the
specified time.
within-last duration: #d#h#m#s Displays the statistics collected within last
time.
name Displays the CPU management class.
arp|icmp|ssh|snmp|fabric|bcast|
nfs|web|web-ssl|net-api
in-bytes in-bytes-number Displays the ingress bytes processed.

Pluribus Networks
www.pluribusnetworks.com
49
 in-pkts in-pkts-number Displays the ingress packets processed.
drop-pkts drop-pkts-number Displays the number of ingress packets
dropped.

CLI network-admin@Leaf1 > cpu-mgmt-class-show

name rate-limit
------- ----------
arp unlimited
icmp unlimited
ssh unlimited
snmp unlimited
fabric unlimited
bcast unlimited
nfs unlimited
web unlimited
web-ssl unlimited
net-api unlimited

CLI network-admin@Leaf1 > cpu-mgmt-class-stats-settings-show

switch name in-bytes in-pkts drop-pkts

-------- ------- -------- ------- ---------
dorado05 arp 0 0 0
dorado05 icmp 0 0 0
dorado05 ssh 0 0 0
dorado05 snmp 0 0 0
dorado05 fabric 0 0 0
dorado05 bcast 0 0 0
dorado05 nfs 0 0 0
dorado05 web 0 0 0
dorado05 web-ssl 0 0 0
dorado05 net-api 0 0 0

Display Physical Port Layer 2 Information

Display physical port information at Layer 2 using the port-phy-show command. The
command displays information about the default VLAN, link quality, maximum frame size,
Ethernet mode, speed, and status. You can display the default VLAN for a port.

CLI network-admin@Leaf1 > port-phy-show

port state speed eth-mode max-frame learning def-vlan

---- ----- ----- -------- -------- -------- --------
17 up 1000 1000base-x 1540 on 1
19 up 10000 10Gbase-cr 10232 on 1

50
Pluribus Networks
www.pluribusnetworks.com
Displaying Transceiver Information
Display information about the transceivers connected to the switch using the
port-xcvr-show command:

CLI network-admin@Leaf1 > port-xcvr-show

switch port vendor-name part-number serial-number

-------------- ---- ---------------- ---------------- ----------------
Spine1 3 3M 1410-P17-00-0.50
Spine1 4 3M 1410-P17-00-0.50
Spine1 57 3M 9QA0-111-12-1.00 V10B9252
Spine1 65 3M 9QA0-111-12-1.00 V10B9614

Configuring Minimum and Maximum Bandwidth on Ports

This feature introduces bandwidth guarantees on switch ports. Currently, Pluribus Networks
switches allow rate limiting only for CPU facing (PCIe, data and span) ports. Using this
feature, you configure bandwidth guarantees at egress CoS (Class of Service) queue level
and manage prioritized traffic. Use this feature for setting SLAs (Service Level Agreements).
Currently, Netvisor ONE provides maximum bandwidth policing at the vFlow level,but youo
cannot set guaranteed minimum bandwidth. This feature addresses the limitation.
Switch hardware supports minimum and maximum bandwidth guarantees configured at a
port and CoS queue level. Netvisor supports a per port configuration. Netvisor allows the
settings as a percentage of port speed, and determines the data rate internally on command
execution. Additionally, you update port speed , the port configuration internally re-adjusts
the minimum or maximum bandwidth rates for the applicable ports. The port-config-show
command displays 100% allocations by default. Netvisor displays new configurations as
additional elements, sorted by CoS queue.
Netvisor modifies and shows ONLY modified port configurations. Ports not displayed in the
show command output default to the settings 100% link capacity, and no minimum
guarantee for each CoS queue.
(CLI network-admin@Spine)>port-cos-bw-modify

cos integer Specify the CoS priority between 0 and 7.

port port-list Specify the physical port(s).
min-bw-guarantee Specify the minimum bandwidth as a percentage.
min-bw-guarantee-string
max-bw-limit Specify the maximum bandwidth as a percentage.
max-bw-limit-string

(CLI network-admin@Spine)>port-cos-bw-show

cos integer Specify the CoS priority.

port port-list Specify the physical port(s).

CLI (network-admin@Spine1) > port-cos-bw-modify port 2-5 cos 5

min-bw-guarantee 10

Pluribus Networks
www.pluribusnetworks.com
51
CLI (network-admin@Spine1) > port-cos-bw-show

switch cos port min-bw-guarantee max-bw-limit weight

---------------- --- ------ ---------------- ------------ ------
Spine1 0 1-72 0% 100% 16
Spine1 1 1-72 0% 100% 32
Spine1 2 1-72 0% 100% 32
Spine1 3 1-72 0% 100% 32
Spine1 4 1-72 0% 100% 32
Spine1 5 1,6-72 0% 100% 32
Spine1 5 2-5 10% 100% 32
Spine1 6 1-72 0% 100% 64
Spine1 7 1-72 0% 100% 127
CLI (network-admin@Spine1) > port-cos-bw-modify port 2-511,13 cos 4 min-bw 20
max-bw 80
CLI (network-admin@Spine1) > port-cos-bw-show

switch cos port min-bw max-bw

------- ---- ------ ------ -----
Spine1 0 0-72 100% 100%
Spine1 1 0-72 100% 100%
Spine1 2 0-72 100% 100%
Spine1 3 0-72 100% 100%
Spine1 4 0-1,11-72 100% 100%
Spine1 4 11-13 20% 80%
Spine1 5 2-10 10% 100%
Spine1 6 0-72 100% 100%
Spine1 7 0-72 100% 100%

Changing the port settings to new values overrides the previous settings.

Changes to Class of Service (CoS) Behavior

Netvisor ONE automatically weights Class of Server (CoS) queues with minimum bandwidth
guarantees. When you configure minimum bandwidth on a port queue using CoS, Netvisor
assigns the remaining bandwidth to the rest of the queues in the same ratio as the minimum
bandwidth.
Once you configure minimum bandwidth on a CoS port using the command,
port-cos-bw-modify, the command, port-cos-weight-modify, no longer works to
configure the CoS queue weight.
When you configure a minimum bandwidth without specifying a weight value, the weight for
the port and CoS is automatically set on a scale of 1 to 100. For example, if you configure the
minimum bandwidth as 10%, Netvisor ONE automatically assigns the queue a weight value
of 1. You can also assign a specific weight value to the queue.
Additionally, you can configure strict priority scheduling for any of the queues. By default,
CoS 7 assigns strict priority scheduling for the queue.
To allow automatic weight assignment for CoS queues, use the following syntax:

CLI network-admin@Leaf1 > system-settings-modify

cosq-weight-auto|no-cosq-weight-auto

52
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@Leaf1 > port-cos-bw-modify

cos integer Specify the CoS priority between 0

and 7.
port port-list Specify the physical port(s).
min-bw-guarantee Specify the minimum bandwidth as
min-bw-guarantee-string a percentage.
max-bw-limit max-bw-limit-string Specify the maximum bandwidth as
a percentage.
weight|no-weight Specify if the scheduling weight
after the bandwidth guarantee is
met.

CLI network-admin@Leaf1 > port-bw-show

switch cos port min-bw-guarantee max-bw-limit weight

------- --- ---- ---------------- ------------ ------
Spine-1 0 1-72 0% 100% 0
Spine-1 1 1-72 0% 100% 0
Spine-1 2 1-72 0% 100% 0
Spine-1 3 1-72 0% 100% 0
Spine-1 4 1-72 0% 100% 0
Spine-1 5 1-72 0% 100% 0
Spine-1 6 1-72 0% 100% 0
Spine-1 7 1-72 0% 100% 0

To auto-configure bandwidth, use the following syntax:

CLI network-admin@Leaf1 > port-cos-bw-modify cos 1 port 1 min-bw-guarantee 20

CLI network-admin@Leaf1 > port-cos-bw-show

switch cos port min-bw-guarantee max-bw-limit weight

------- --- ---- ---------------- ------------ ------
Spine-1 0 1-72 0% 100% 0
Spine-1 1 2-72 0% 100% 0
Spine-1 1 1 20% 100% 2
Spine-1 2 1-72 0% 100% 0
Spine-1 3 1-72 0% 100% 0
Spine-1 4 1-72 0% 100% 0
Spine-1 5 1-72 0% 100% 0
Spine-1 6 1-72 0% 100% 0
Spine-1 7 1-72 0% 100% 0

To configure a specific weight, use the following syntax:

CLI network-admin@Leaf1 > port-cos-bw-modify cos 1 port 1 weight 6

CLI network-admin@Leaf1 > port-cos-bw-show

Pluribus Networks
www.pluribusnetworks.com
53
switch cos port min-bw-guarantee max-bw-limit weight
------- --- ---- ---------------- ------------ ------
Spine-1 0 1-72 0% 100% 0
Spine-1 1 2-72 0% 100% 0
Spine-1 1 1 20% 100% 6
Spine-1 2 1-72 0% 100% 0
Spine-1 3 1-72 0% 100% 0
Spine-1 4 1-72 0% 100% 0
Spine-1 5 1-72 0% 100% 0
Spine-1 6 1-72 0% 100% 0
Spine-1 7 1-72 0% 100% 0

Configuring Port Storm Control

Port Storm Control prevents traffic on a LAN from disruption by a broadcast, multicast, or
unicast storm on a port. A LAN storm occurs when packets flood the LAN, creating excessive
traffic and degrading network performance.
Use port-storm-control-modify to modify the percentage of total available bandwidth that
can be used by broadcast, multicast, or unicast traffic.

CLI network-admin@Leaf1 > port-storm-control-modify port 11

unknown-ucast-level 1.1

Use the port-storm-control-show command to display the configuration:

CLI network-admin@Leaf1 > port-storm-control-show

switch port speed unknown-ucast-level unknown-mcast-level broadcast-level

vlag trunk
-------- ---- ----- ------------------- ------------------- ---------------
---------- ------
draco01 1 40g 30% 30% 30%
draco01 2 10g 30% 30% 30%
draco01 3 10g 30% 30% 30%
draco01 4 10g 30% 30% 30%
draco01 5 40g 30% 30% 30%
draco01 6 40g 30% 30% 30%
draco01 7 10g 30% 30% 30

Enabling Jumbo Frame Support

Jumbo frames includes frames bigger than the standard Ethernet frame size, 1518 bytes
(including Layer 2 (L2) header and FCS). Different vendors assign different frame sizes as
the IEEE standard does not include jumbo frames.
When you enable the jumbo frame feature on a port, the port switches large or jumbo
frames. This feature optimizes server-to-server performance. The default Maximum
Transmission Unit (MTU) frame size is 1548 bytes for all Ethernet ports. The MTU size is
increased to 9216 bytes when you enable the jumbo frame feature on a port.
Netvisor disables jumbo frame support by default.
To enable jumbo frame support, add the jumbo parameter to the port-config-modify
command:

CLI network-admin@Leaf1 > port-config-modify jumbo

54
Pluribus Networks
www.pluribusnetworks.com
About Port Isolation
Port Isolation prevents local switching among ports on a Netvisor ONE switch or on a pair of
Netvisor ONE switches configured as a cluster. With Port Isolation, Netvisor disables direct
communication with hosts part of same Layer 2 domain connected to isolated ports or to
mutually learn the other MAC address. Communication between these hosts occurs through a
Layer 3 device. Use this feature to secure bridged east-west traffic through a firewall.
When using this feature on ports within a cluster, you must configure the port-link state
association rules between the uplink ports and the downlink isolated ports.

Example Use-Case Configuration

In a typical scenario, as shown in Figure 4, ports 1, 2, and 3 configured as isolated ports so
the hosts attached to these ports cannot communicate with each other directly, but only
through the upstream firewall or router connected to port 64.
Figure 4:Port Isolation scenario

As shown in Figure 4, create the configuration as follows:

PN-HA1

CLI network-admin@Leaf1 > port-config-modify port 1 no-local-switching

CLI network-admin@Leaf1 > port-config-modify port 2 no-local switching

Pluribus Networks
www.pluribusnetworks.com
55
PN-HA2

CLI network-admin@Leaf1 > port-config-modify port 2 no-local-switching

CLI network-admin@Leaf1 > port-config-modify port 3 no-local-switching

Typically, you configure the upstream router or firewall to perform local proxy ARPs and/or
NDP proxy and respond to all ARP requests and/or Neighbor Solicitations coming from
isolated hosts. To avoid interfering with local proxy ARPs and NDP proxy, disable ARP and ND
Optimization as follows:

CLI network-admin@Leaf1 > system-settings-modify no-optimize-arps

CLI network-admin@Leaf1 > system-settings-modify no-optimize-nd

Configuring Port Isolation

To configure Port Isolation, use the following steps:
1. Configure the isolated ports. In this example, ports 1 and 2:

CLI network-admin@Leaf1 > port-config-modify port 1,2 no-local-switching

2. Optionally, configure the port link state association. A port association requires matching
the link state of downlink isolated ports with the one of uplink ports. When all uplink ports
detect the down state, Netvisor disables downlink isolated ports until one of the uplinks
becomes operational again. In this example, the port association name ,PA, uplink (mas-
ter), ports value 64, and isolated downlink (slave) ports value as 1, 2.

CLI network-admin@Leaf1 > port-association-create-name PA master-ports 64

slave-ports 1,2 policy any-master

3. Optionally, disable ARP and ND optimization.

CLI network-admin@Leaf1 > system-settings-modify no-optimize-arps

CLI network-admin@Leaf1 > system-settings-modify no-optimize-nd

This feature uses the command no-local-switching for the port-config-modify command.
To configure one or more isolated ports:

CLI network-admin@Leaf1 > port-config-modify port port-list

no-local-switchingg

To view ports that are impacted by the no-local-switching command, use the
port-egress-show command:
switch port egress rx-only active-active-vlags loopback
------ --------- ------- ------------------- -------------------- --------
1 0-72 none none none none
2 0-72 none none none none
3 0-72 none none none none
4 0-72 none none none none
5 0-4,11-72 none none none none
6 0-4,11-72 none none none none
7 0-4,11-72 none none none none
8 0-4,11-72 none none none none

56
Pluribus Networks
www.pluribusnetworks.com
mir_prevent_out no-local-switching-out
------------------------ ----------------------
none none
none none
none none
none none
none 5-10
none 5-10
none 5-10
none 5-10

The following Port Isolation options for the trunk-create, trunk-modify, and trunk-show
commands are as follows:

CLI network-admin@Leaf1 > trunk-create

trunk-create Create a trunk configuration for link

aggregation
one or more of the following options:
local-switching|no-local-switching Specify no-local-switching if you do not
want the port to bridge traffic to another
no-local-switching port.

CLI network-admin@Leaf1 > trunk-modify

trunk-modify Modify a trunk configuration for link

aggregation
one or more of the following options:
reflect|noreflect Specify if physical port reflection is enabled
or not.

CLI network-admin@Leaf1 > trunk-show

trunk-show Display trunk configuration

one or more of the following options:
reflect|noreflect Displays if physical port reflection is
enabled or not.

Pluribus Networks
www.pluribusnetworks.com
57
Support for Priority-based Flow Control

Informational Note: This feature is supported on the following platforms:

S4048-ON, S6000-ON, AS5712-54X, AS6712-32X, F9272-X, and F9232-Q.

Priority Flow Control (PFC) is an IEEE standard (802.1qbb) for link level flow control on
Ethernet networks. Functionally, this feature is similar to the IEEE standard 802.3 for PAUSE
mechanism, except that it operates at the granularity of individual packet priorities or traffic
class, instead of port level. When a queue corresponding to traffic with a particular traffic
class reaches a predetermined, either auto or statically set, threshold, the switch chip
generates a PFC frame and sends it back to the sender. For PFC to work effectively end to end
on the network, all switches and hosts in the traffic path are configured to enable PFC, and
configured for traffic class to priority mappings.
Netvisor ONE commands configure priorities, or traffic classes, for PFC. The configuration
allows you to add ports where PFC is enabled. When enabled, traffic class to CoS queue
mappings, as well as to packet priorities, Netvisor performs the configuration in the
background.
Netvisor accepts the following mappings:
1 to 1 traffic class to CoS queue mapping (0 through 7
1 to 1 packet priority (802.1p) mapping
You enable PFC for both transmitting and receiving on the selected port. For transmit,
Netvisor ONE pauses traffic corresponding to the traffic class indicated in the received PFC
frame. For receive, Netvisor ONE generates a PFC frame when a queue corresponding to a
traffic class reaches the pause threshold. Netvisor ONE auto-configures parameters such
buffer threshold, and pause timer value. Disabling PFC turns off PFC for receive and transmit,
although the traffic class priority and queue mappings remain.
On supported switches, even with ingress admission control enabled (in lossless mode), by
default, Netvisor supports only the traffic class or priority group 7 with the memory
management unit (MMU) buffer resources. Packets of all priorities utilize the resources of the
default priority group unless specifically configured. This implies when enabling a new priority
group for PFC, Netvisor generates the buffer configuration and saves in the chip configuration
file, which is read during system initialization for MMU setup. AS a result, when you enable a
new priority for PFC, you must restart Netvisor ONE. Adding new ports to an existing priority
group setting, for another port or ports, does not require restarting Netvisor ONE.
You configure Up to three priority group buffer settings on switches in Netvisor ONE. If you
attempt to configure more than three, Netvisor ONE returns an error message.
To create a new PFC configuration on port 2 with a priority group of 2, use the following
command:
CLI (network-admin@Spine1)port-pfc-create priority 2 port 1-10
Priority configuration will be effective after restart.

To modify the ports and change them to 11-15, use the following command:
CLI (network-admin@Spine1)port-pfc-modify priority 2 port 11-15
Priority configuration will be effective after restart.

58
Pluribus Networks
www.pluribusnetworks.com
To delete the configuration, use the following command:
CLI (network-admin@Spine1)port-pfc-delete priority 2 port 11-15

To display the configuration, use the port-pfc-show command:

CLI (network-admin@Spine1)port-pfc-show
switch priority port error
------- -------- ----- -----
Spine1 2 11-20
Spine1 3 11-20

Support for Priority-based Flow Control Port Statistics

Display the stats to confirm or debug traffic characteristics when you configure PFC. New
commands allow you to display PFC stats per port and adjust the statistics collection.

CLI network-admin@Leaf1 > port-pfc-clear

port port-list Specify the ports to delete PFC statistics.

CLI network-admin@Leaf1 > port-pfc-stats-settings-modify

enable|disable Specify if you want to enable or disable PFC statistics

collection.
interval duration: #d#h#m# Specify the interval between statistics collection.
disk-space disk-space-number Specify the amount of disk space for statistics collection.

CLI network-admin@Leaf1 > port-pfc-stats-settings-show

enable|disable Specify if you want to enable or disable PFC statistics

collection.
interval duration: #d#h#m# Specify the interval between statistics collection.
disk-space disk-space-number Specify the amount of disk space for statistics collection.

CLI network-admin@Leaf1 > port-pfc-stats-show

time date/time: Displays the date and time for statistics collection.
yyyy-mm-ddTHH:mm:ss
start-time date/time: Displays the start date and time for statistics collection.
yyyy-mm-ddTHH:mm:ss
end-time date/time: Displays the end date and time for statistics collection.
yyyy-mm-ddTHH:mm:ss
duration duration: #d#h#m#s Displays the duration for statistics collection.
interval duration: #d#h#m#s Displays the interval between statistics collection.
since-start Displays the statistics since the start time.
older-than duration: #d#h#m#s Displays the statistics older than the specified time.
within-last duration: #d#h#m#s Displays the statistics within a specified time.
port port-list Displays the port list.

Pluribus Networks
www.pluribusnetworks.com
59
Support for Fabric Guard
Currently, Netvisor detects a Layer 2 loop using by STP, LLDP, or loop detect code. However if
a third party device connected to a Pluribus Networks switch consumes LLDP such as a
hypervisor vSwitch, and you configure the port as an edge port, Netvisor ONE cannot detect
loops on the network.
If you configure a port as fabric-guard port, Netvisor ONE triggers sending global discovery
multicast packets on this port after the port is physically up and in an adjacency wait state. If
a port with fabric-guard configuration receives a global discovery packet, Netivsor disables
the port in the same way LLDP disables the port when receiving messages from the same
switch. In order to re-enable the port once the loop is fixed, you must manually enable the
port using the command, port-config-modify port port-number enable.
To enable fabric guard, use the following syntax:

CLI network-admin@Leaf1 > port-config-modify port port-number fabric-guard

To disable fabric guard, use the following syntax:

port-config-modify port port-number no-fabric-guard

60
Pluribus Networks
www.pluribusnetworks.com
Introducing Netvisor ONE Foundational Objects
 About the Netvisor ONE Fabric
 Adding Switches to an Existing Fabric
 Directly Connected Switches in a Fabric
 Fabric Over Management Interface
 Displaying Information about Nodes in the Fabric
 Simplifying Netvisor OS VXLAN Fabric Configuration
 Configuring Link Aggregation Control Protocol (LACP)
 Active-Standby Link Aggregation on Management Interfaces
 Configuring Trunking for Link Aggregation (LAG)
 High Availability
 Modifying a Trunk or VLAG Configuration by Changing the LACP Mode
 Safely Restoring Ports for Cluster Configurations
 Configuring Layer 2 Multipathing for Virtual Chassis Link Aggregation (VLAG)
 VLAG Topology Examples
 Configuring Active-Active VLAG


About the Netvisor ONE Fabric

Netvisor defines a fabric as a distributed architecture based on a collection of compute
clustering techniques to present an open, standard-based Ethernet fabric as one logical
switch. Every node shares the same view of the fabric including MAC and IP addresses,
connections, and application flows.
When you add switches to the fabric, all switches acknowledges a single management
domain highly available through multiple link aggregation and load balancing between
network resources.
The fabric performs a classic database 3-phase commit for configuration changes. All
members of the fabric must accept the configuration changes before the change occurs on
the fabric. Figure 1 Fabric Architecturedisplays the fabric architecture of Netvisor ONE.

Pluribus Networks
www.pluribusnetworks.com
61
Figure 1: Fabric Architecture

Creating an Initial Fabric

After you complete the initial setup on the switch, you must create a new fabric for the
switch or join an existing fabric. When switches form a fabric, the fabric becomes one logical
switch, and shares state as well as communicates commands so that any scope of a fabric-
command executes on each switch in the fabric. A switch must be in a fabric in order to keep
track of the fabric state. However, a switch can be a member of fabric, and consist of a single
switch. A switch leaving one fabric and joining another loses the fabric state of the first fabric
and learns the fabric state of the second fabric.
1. Create a new fabric , use the following command:

CLI network-admin@switch > fabric-create name name-string

2. Create a name for the new fabric.

If you want to assign a password to the fabric that allows other switches to securely join
the fabric only if the administrator knows the password, use the password option. Press the
return key after typing the password parameter:

CLI network-admin@switch > fabric-create name name-string <return>

password:*******
Re-enter password:*******

By default, the fabric is created on VLAN1. You can specify a different VLAN, but if you
change the VLAN, you must recreate the fabric.
To join a remote fabric use the fabric-join command and the switch IP address. For
example,

CLI network-admin@switch > fabric-join switch-ip 192.168.2.2 vlan 20

3. To show fabric details, use the fabric-show command:

62
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > fabric-show

name id vlan fabric-network control-network tid

fabric-advertisement-network
---------------------- -------------------------------- -------- ---------------------------- ------
------------------------- ------- -------------------------------------------------------
Leaf1 b000707:59b6a9ef 0 mgmt mgmt 48 inband-mgmt
Leaf2 90004eb:59b7da05 0 mgmt mgmt 8 inband-mgmt

Adding Switches to an Existing Fabric

For this example, the switches connect as in Figure 4 Directly Connected Switches in a
Fabric:
Figure 4: Directly Connected Switches in a Fabric

Netvisor ONE accepts global discoveries for identifying fabrics in the network as joinable
fabrics for the fabric-join command. Upon receipt, the receiver sends back a global
keep-alive with node and fabric information.
When you have more than one switch, you must add it to the fabric to take advantage of the
features offered by Netvisor ONE. To add the new switch, use the following command on one
of the switches:

CLI network-admin@switch > fabric-join name corp-fabric

You can join the fabric using either the fabric name or the switch IP address. If you use the
Tab key to display the available options, Netvisor ONE displays all fabrics configured on the
network as options.

Pluribus Networks
www.pluribusnetworks.com
63
If you specify a password for the fabric, you must type it in twice. Netvisor ONE uses the
password to encrypt communication between the nodes in the fabric. When you join the
fabric from a node, you must type in the password to join it.
You can specify a specific VLAN for the fabric when you create a new one, or by default, the
fabric uses VLAN1. However, you cannot change the fabric VLAN without recreating the
fabric.

Informational Note: Avoid creating fabrics with the same name.

When Netvisor creates the fabric, the switch begins sending multicast messages out on Layer
2 looking for other switches. Netvisor ONE does not propagate the messages to other
networks. This is how Switch B in Figure 4 learns about the fabric.
Once Switch B joins the fabric, the switch downloads the fabric configuration (commands
with scope fabric) and the switch reboots.
If you want to connect to a switch over Layer 3, you must specify the IP address for the
switch in the fabric using the following command:

CLI network-admin@switch > fabric-join switch-ip 192.168.11.1

fabric joined.

Fabric Over Management Interface

Configure fabric communication over either the management interface or the in-band
interface. Because fabric communication over the in-band interface can be disrupted due to
STP, ports flapping, and other factors, fabric communication over management interface
provides a more consistent configuration.
If you create a fabric with the management interface, any nodes joining the fabric inherit this
setting. All nodes in a single fabric all run on the same network type. You cannot run a mixed
configuration of management and in-band interfaces. Fabrics advertised on an incompatible
network are not available for when you issue the fabric-join command. This keeps a
switch from joining an incompatible fabric.
If you configure the fabric on the management interface, all fabric-communication statys on
the management network, except for the following:
Cluster synchronization-related traffic such as VLAG synchronizations and forwarded
STP packets.
Cluster keep-alive packets on the fabric
Fabric keep-alive packets and global-discovery packets because both run on
management and in-band interfaces.
Two new states are added to the state field of fabric-node-show:

CLI network-admin@switch > fabric-node-show ?

[state offline|online|in-band-only-online|mgmt-only-online|
fabric-joined|eula-required|setup-required|fabric-required| fresh-install]

64
Pluribus Networks
www.pluribusnetworks.com
Because there are now two networks for Netvisor ONE to monitor for connectivity, online
denotes reachability for both management and in-band interfaces. in-band-only-online
denotes reachability switch through the in-band network. mgmt-only-online denotes
reachability through the management network, and offline denotes no reachability on
either network.
Netvisor supports monitoring and reporting on both the management and in-band network
connectivity.

Displaying Fabric Information

Display the fabric using the fabric-show command:

CLI network-admin@switch > fabric-show

name id vlan fabric-network control-network tid

---------------- ---------------- ---- -------------- --------------- ----
info-dev a000030:5537b46c 3 in-band in-band 365
ursa-lyon 6000210:566621ee 0 mgmt in-band 5055

You can display information about the fabric using the fabric-info command:

CLI network-admin@switch > fabric-info format all layout vertical

name: pn-EBC4
switch-ip: ::
id: a0000c5:53ab701e
mcast-ip: 239.4.10.111
fid: 327
cid: 0

Display information about fabric statistics using the fabric-stats-show command:

CLI network-admin@switch > fabric-stats-show

switch id servers storage VM vlan vxlan tcp-syn tcp-est tcp-completed

-------- -- ------- ------- -- ---- ----- ------- ------- -------------
pubdev02 0 0 0 0 0 0 14.0K 5 40
pubdev03 0 0 0 0 0 0 3.85K 3 24
tcp-bytes udp-bytes arp
--------- --------- ---
125K 0 0
110M 0 0

Displaying Information about Nodes in the Fabric

Display information about the nodes in the fabric.Take note of the fab-tid value. If the
fab-tid values do not match for each node, you can use the commands
transaction-rollback-to or transaction-rollforward-toto resynchronize the fabric.

Pluribus Networks
www.pluribusnetworks.com
65
About Fabric Transactions
Netvisor uses transactions to synchronize configuration changes across the fabric or cluster.
The originator sends the configuration change request from the client to other nodes in the
fabric. Clients communicate the primary initiations of transactions, and clients consist of a
CLI user, a REST API user, or an OpenStack client.
Netvisor sends transactions over a TCP socket on port 23399 on the fabric network, and
opens a new socket for each transaction and phase of a transaction. Netvisor does not retain
the sockets and closes the socket after each phase of the transaction.
Netvisor uses status updates to distribute information across the fabric. Unlike transactions,
Netvisor does not require the data to be synchronized across the fabric, but usually the data
synchronizes.
Netvisor sends the following states from the local node to other nodes in the fabric:
Node State
Port State
VLAN State
Owned vPort State
Layer 3 Entry State
Netvisor sends the following states to a cluster peer:
All Local vPort State
Spanning Tree State
Configuration File Hash Keys
vRouter state
vRouter Interface State
Bridge Domain State
Tunnel State
A node only sends updates to vPorts, directly connected hosts on the node, and you can
display this information using the command, vport-show. Cluster nodes share a larger data
set in order to act as a single node for redundancy.

More Information on Status Updates

Netvisor sends status updates whenever a state changes on the fabric. For example, if a port
goes down, or you create a a new VLAN, that port or VLAN sends an update. Only the
changed object, for example, VLAN10, sends an update.
Status updates include the entire object, not just the changed field on the object. If sending
data fails, Netvisor ONE continue attempting to resend every 250ms.

More About Transactions

Netvisor ONE uses transactions to synchronize configuration changes in a Netvisor fabric.
Transactions include create, delete, modify, add, and remove commands.
The scope defines the participating nodes in the transaction:
Local - only a local node participates in the transaction.
Cluster - only the two nodes in a cluster participate in the transaction.
Fabric - all nodes in a fabric participate in the transaction.

66
Pluribus Networks
www.pluribusnetworks.com
Rolling Back and Rolling Forward Transactions
The Netvisor ONE transaction log contains both the command and the undo command(s) for
a transaction, used to redo the transaction, roll forward, or undo the transaction, roll back.
Roll back a transaction roll back using the command, transaction-rollback-to. Any scope
can be rolled back. Transactions roll back from the latest, to a specified older transaction, in
order. After successfully rolling back the transaction, the local change completes, and the
transaction removed from the transaction log.
You can roll forward a transaction using the command, transaction-rollforward-to.
Netvisor only supports the scope fabric and cluster, because the local node receives the
transaction from a location on the fabric. Transactions roll forward from the next transaction
in order to the target transaction ID. Netvisor applies the changes locally and updates the
transaction log.

CLI network-admin@switch > fabric-node-show

id: 167772619
name: Leaf2
fab-name: fab1
fab-id: a0001c8:53e2601b
cluster-id: 0:0
fab-mcast-ip: 239.4.10.94
local-mac: 64:0e:94:28:06:f2
mgmt-nic:
mgmt-ip: 192.168.1.14/24
...
in-band-ip: 192.168.254.14/24
...
fab-tid: 9
out-port: 0
version: 2.1.201015836,pn-ONVLnvOS-2.0.2-2000212196
state: online
firmware_upgrade: not-required
device_state: ok
ports: 72
id: 201326827
name: Leaf1
fab-name: fab1
fab-id: a0001c8:53e2601b
cluster-id: 0:0
fab-mcast-ip: 239.4.10.94
local-mac: 64:0e:94:30:03:97
mgmt-nic:
mgmt-ip: 192.168.1.11/24
...
in-band-ip: 192.168.254.11/24
...
fab-tid: 9
out-port: 129
version: 2.1.201015836,pn-ONVLnvOS-2.0.2-2000212196
state: online
firmware_upgrade: not-required
device_state: ok
ports: 72
id: 167772618

Pluribus Networks
www.pluribusnetworks.com
67
name: Spine2
fab-name: fab1
fab-id: a0001c8:53e2601b
cluster-id: 0:0
fab-mcast-ip: 239.4.10.94
local-mac: 64:0e:94:28:06:ee
mgmt-nic:
mgmt-ip: 192.168.1.13/24

An example of a fabric that is out of sync for two nodes in the fabric:

CLI network-admin@switch > fabric-node-show format all layout vertical

id: 100663365
name: CBF-switch
fab-name: pn-CBF4
fab-id: a0000c5:53ab701e
cluster-id: 0:0
fab-mcast-ip: 239.4.10.111
local-mac: 64:0e:94:18:01:03
mgmt-nic:
mgmt-ip: 192.168.1.61/24
...
in-band-ip: 192.168.77.61/24
...
fab-tid: 328
out-port: 128
version: 2.1.201005800,pn-ONVL-2.0.2-2000212196
state: online
firmware_upgrade: not-required
device_state: ok
ports: 68
id: 201326771
name: CBF-Leaf-1
fab-name: corp-CBF4
fab-id: a0000c5:53ab701e
cluster-id: 0:0
fab-mcast-ip: 239.4.10.111
local-mac: 64:0e:94:30:02:4d
mgmt-nic:
mgmt-ip: 192.168.1.53/24
...
in-band-ip: 192.168.77.53/24
...
fab-tid: 329
out-port: 128
version: 2.1.201005800,pn-ONVLnvOS-2.0.2-2000212196
state: online
firmware_upgrade: not-required
device_state: ok
ports: 72
id: 167772357
name: CBF-Spine1
fab-name: pn-CBF4
fab-id: a0000c5:53ab701e
cluster-id: 0:0
fab-mcast-ip: 239.4.10.111

68
Pluribus Networks
www.pluribusnetworks.com
local-mac: 64:0e:94:28:02:de
mgmt-nic:
mgmt-ip: 192.168.1.51/24
...
in-band-ip: 192.168.77.51/24
f you apply a configuration to the fabric, and a node does not respond to it, you evict the
node from the fabric, and then troubleshoot the problem. To evict a node, use the following
command:

CLI network-admin@switch > fabric-node-evict name pleiades25

or

CLI network-admin@switch > fabric-node-evict id b000021:52a1b620

Configuring Ports for Fabric Communication

Netvisor vFlows to match fabric traffic do so based on the packet protocol, TCP or UDP, plus
the Layer 4 destination port. That means any existing TCP or UDP traffic using the same
Layer 4 destination port matches the vFlows.
You configure a port range for TCP or UDP traffic in order to avoid traffic conflicts with
existing TCP or UDP traffic.Netvoisor allows this functionality:
(CLI network-admin@Spine1)>fabric-comm-ports-modify

range-start Specify a port range from 1024 to 65435.

(CLI network-admin@Spine1)>fabric-comm-port-show
switch: techpub-accton-2
range-start: 23300
fabric-port: 23399
notify-port: 23398
proxy-port: 23397
fabric-keepalive-port: 23394
filesystem-replication-port: 23392
cluster-traffic-forwarding-port: 23391
vport-statistics-port: 23390
l2-encap-port: 23389
igmp-encap-port: 23388
icmpv6-encap-port: 23387
arp-encap-port: 23386
cluster-analytics-port: 23385

When you modify the port range, you must modify each node in the fabric which temporarily
interrupts fabric communication until you configure each node with the same port range.
There is no forwarded traffic loss if the interruption is brief. Because application of this
command prevents communication with other nodes, you must log in to each node directly
and separately to apply the command.

Pluribus Networks
www.pluribusnetworks.com
69
Configuring Link Aggregation Control Protocol (LACP)
Link Aggregation Control Protocol (LACP), a IEEE specification 802.3ad, allows you to bundle
several physical ports to form a single logical channel. When you change the number of
active bundled ports on a port channel, traffic patterns reflect the rebalanced state of the
port channel.
LACP supports automatic creation of Gigabit Ethernet port trunks by exchanging LACP
packets between ports. It learns the capabilities of port groups and informs the other ports.
Once LACP identifies correctly matched Ethernet links, it facilitates grouping the links into
Gigabit Ethernet port trunks.
LACP performs the following functions on the switch:
Maintains configuration information to control aggregation.
Exchanges configuration information with other peer devices.
Attaches or detaches ports from the LAG based on the exchanged configuration
information.
Netvisor ONE exchanges LACP packets between ports in these modes:
Active — Places a port into an active negotiating state, and the port initiates
negotiations by sending LACP packets.
Passive — Places a port into a passive negotiating state where the port responds to
LACP packets it receives but does not initiate LACP negotiation. In this mode, the port
channel group attaches the interface to the bundle.
Off — LACP is not enabled on the switch port or trunk
Active and passive modes allow LACP to negotiate between ports to determine if the ports
can form a port channel based on criteria such as port speed and trunking state.
To enable or disable LACP, or change the system priority, use the following command:

CLI network-admin@switch > lacp-modify enable system-priority 35000

Netvisor sets the default system priority value to 32768 with a range from 0 to 65535.

Understanding LACP Priority

You configure LACP port priority i on each port using LACP. You use the default value, 32768,
or configure a specific value from 0 to 65535. LACP uses the port priority with the port
number to form the port identifier. The port priority determines which ports should be in
standby mode when there is a hardware limitation that prevents all compatible ports from
aggregating.
LACP system priority can be configured automatically or using the CLI. LACP uses the system
priority with the device MAC address to form the system ID and also during negotiations with
other systems. Configure a range of LACP system priority from 0 to 65535 with the default
value of 32768.
To create a trunk with LACP, use the following command:

CLI network-admin@switch > trunk-create name trunk23 port 20-36 lacp-mode

active

To modify a trunk with LACP, use the following command:

CLI network-admin@switch > trunk-modify name trunk23 lacp-mode passive

70
Pluribus Networks
www.pluribusnetworks.com
To modify a port configuration and add LACP priority to the port, use the following command:

CLI network-admin@switch > port-config-modify port 33 lacp-priority 34

Active-Standby Link Aggregation on Management Interfaces

You configure dual management ports to form LAG with both ports as active. Netvisor
supports Active-standby, also called active-backup, for LAG on management ports. The
management interfaces form a LAG, and only one of the members becomes active at
anytime. The other link defaults to standby mode, and joins the LAG only if the primary link
goes down. When the primary link comes up, the link joins the LAG and the backup link
unjoins.
Use the switch-setup-modify command option for the parameter, mgmt-lag,
active-standby.

Configuring Trunking for Link Aggregation (LAG)

Informational Note: You must create unique names for each VLAG.

To configure a trunk for aggregating the links connected to ports 1, 2, 3, use the following
steps:
1. Create a trunk called trunk-1 on ports 1, 2, 3, enter the following command:

CLI network-admin@switch > trunk-create name trunk-1 port 1,2,3

2. To verify the configuration, use the trunk-show command:

CLI network-admin@switch > trunk-show

name port speed autoneg jumbo

-------- ----- ------ ------- ------
trunk-1 1-3 10g off off

3. Modify the trunk configuration by removing port 2:

CLI network-admin@switch > trunk-modify name trunk-1 port 1,3

4. Verify the updated trunk configuration.

CLI network-admin@switch > trunk-show

name port speed autoneg jumbo

-------- ----- ------ ------- ------
trunk-1 1,3 10g off off

Notice that the ports have changed from 1-3 to 1,3 indicating that port 2 is no longer a
member of the trunk configuration.
5. Delete the trunk configuration from the switch:

CLI network-admin@switch > trunk-delete name trunk-1

Pluribus Networks
www.pluribusnetworks.com
71
 Verify that the trunk configuration is removed by using the trunk-show command.

High Availability
Netvisor ONE switches automatically perform functions that ease your administrative burden.
With high availability, switches in a fabric automatically detect other switches in the fabric. If
multiple connections exist between two switches, the connections automatically create an
801.3ad Link Aggregation Group (LAG) between the two switches for resiliency and load
balancing. Other features require configuration such as connecting one device to two
switches, or if LAGs are desired between Netvisor ONE switches and other manufacturers’
equipment.
Figure 6:High Availability

Configuring a Cluster
With two Netvisor ONE switches, and require the tw switches to provide networking services
in the event one of the switches fails, the switches must be members of the same fabric, and
you must configure them as a cluster.
You configure one node as a primary node and the other node as a secondary. This reference
is asymmetric, as these assignments do not change unless you explicitly configure them
differently.

72
Pluribus Networks
www.pluribusnetworks.com
When you create a cluster configuration, you specify the nodes as cluster-node-1 and
cluster-node-2. These assignments do not change unless you explicitly change them.
Cluster-node-1 is the primary node and cluster-node-2 is the secondary node. These roles
are used to add asymmetry to some protocols. This reference is asymmetric.
A cluster-link contains the port or ports directly connecting the two cluster nodes together. If
you configure more than one port, this refers to the trunk (LAG) of those ports.
Netvisor ONE reserves VLAN 4094 ass VLAN used for cluster synchronization traffic. It is
added to the in-band interface port and cluster-link automatically when you create the
cluster configuration.
Netvisor ONE detects cluster-links using an extra data set send in LLDP messages. When a
cluster-link is detected, Netvisor automatically adds VLAN 4094 to the cluster link.
Netvisor ONE performs cluster synchronization over the control network of the fabric. For the
in-band interface, synchronization uses the clust4094 vNIC on VLAN 4094 over the
cluster-links. For management, Netvisor performs cluster synchronization on the
management interface.
Cluster synchronization uses keep-alive messages to detect if the peer cluster node state as
online. Cluster synchronization messages contain the following information:
Version
State — online, coming-online, etc.
Sequence number
Flags
Uptime
STP version number
Status version number
The state synchronization designates the online or offline state of the cluster. Additionally,the
cluster exchange version numbers so messages adjust to ensure backward compatibility.
Each cluster node sends synch messages to the other node every 2 seconds. If a node misses
three synchronization messages in a row, the cluster goes offline. When a node comes
online, the node triggers the following behavior:
A resend of all status updates to the peer
A resynchronization of all VLAGs
The transition of STP from independent mode to cluster mode.
To set up a cluster of two switches, pleiades4 and pleiades6, you must verify the switches
as members of the existing fabric:

CLI network-admin@switch > fabric-node-show layout vertical

id: 184549641
name: pleiades4
fab-name: corp-fab
fab-id: b000109:5695af4f
cluster-id: 0:0
local-mac: 3a:7f:b1:43:8a:0f
fabric-network: in-band
control-network: in-band
mgmt-ip: 10.9.19.203/16
mgmt-mac: ec:f4:bb:fe:06:20
mgmt-secondary-macs:
in-band-ip: 192.168.168.203/24

Pluribus Networks
www.pluribusnetworks.com
73
in-band-mac: 3a:7f:b1:43:8a:0f
in-band-vlan: 0
in-band-secondary-macs:
fab-tid: 1
cluster-tid: 0
out-port: 0
version: 2.4.204009451,#47~14.04.1-Ubuntu
state: online
firmware-upgrade: not-required
device-state: ok
ports: 104

To create a cluster configuration, use the following command:

CLI network-admin@switch > cluster-create name cluster1 cluster-node-1

onvlnvOS-switch1 cluster-node-2 onvlnvOS-switch2

To verify the status of the cluster, use the cluster-show command:

CLI network-admin@switch > cluster-show

name state cluster-node-1 cluster-node-2

--------- ------- --------------- --------------
cluster1 online corp-switch1 corp-switch2

To replace a failed cluster node, use the cluster-repeer command. However, you must
evict the failed node from the fabric, and then run the cluster-repeer command on an
active node after replacing the failed node.
To display information about the cluster, use the cluster-info command:

CLI network-admin@switch > cluster-info format all layout vertical

name: cluster-leaf
state: online
cluster-node-1: draco01
cluster-node-2: draco02
tid: 2
mode: master
ports: 69-71,129
remote-ports: 69-71,128
If you want to connect the cluster nodes to an uplink switch, you must configure a VLAG
between the ports on the cluster nodes and the uplink switch. For example, if corp-switch1
has port 53 connected to the uplink switch and corp-switch2 has port 19 connected to the
uplink switch, create a VLAG by executing the vlag-create command on either of the
switches:

CLI network-admin@switch > VLAG-create name VLAG-uplink local-port 53

peer-switch switch1 peer-port 19

This example assumes that you’ve entered the command on switch2.

To verify the configuration, use the following command:

CLI network-admin@switch > vlag-show

name cluster mode switch port peer-switch

74
Pluribus Networks
www.pluribusnetworks.com
-------- ----------- ------------- ---------- ---- ---------
switch-1 cluster-2 active-active spine-1 34 spine-2

peer-port status local-state lacp-mode

-------------- --------- ----------- ---------
129 normal enabled active

Informational Note: Before you can create a VLAG, you must configure the two
switches in a cluster.

More Information on vLAGs

Netvisor ONE uses vLAG synchronization to coordinate Active-Standby and Active-Active
vLAGs using the following rules:
Active-Standby vLAGs
Per vLAG, only one side, ports in a cluster node, can be up at any one time. The other
side remains in standby.
In a competing environment, the side with a longer connection time remains up and
the other side becomes disabled.
Active-Active vLAGs
If and only if both sides of the vLAG are up and the ingress port is the cluster link,
Netvisor ONE adds port egress rules to drop any packets egressing the vLAG port.
Active-Active vLAGs also require Layer 2 entry synchronization.

Modifying LACP Mode on an Existing VLAG Configuration

You can modify the LACP mode on an existing VLAG configuration:
Support LACP Mode changes on configured VLAGs
No restrictions for configuring LACP passive mode on VLAGs
Save VLAG configuration after updating it.
Restore the port configuration after deleting a VLA

CLI network-admin@switch > vlag-modify

name name-string Specify the VLAG name.

failover-move-L2| If you specify the parameter, failover-move-L2,
failover-ignore-L2 Netvisor sends gratuitous ARPs.
lacp-mode Specify the LACP mode as off, passive or active.
off|passive|active
lacp-timeout slow|fast Specify the LACP timeout as slow (30 seconds) or
fast (4 seconds).

Pluribus Networks
www.pluribusnetworks.com
75
 lacp-fallback Specify the LACP fallback mode as individual or
bundle|individual bundled.
lacp-fallback-timeout Specify the LACP fallback timeout in seconds. The
seconds default is 50
seconds

Modifying a Trunk or VLAG Configuration by Changing the LACP Mode

Modify a trunk or VLAG configuration by changing the LACP fallback mode and timeout.
This feature enables ports of a static LACP trunk to operate as individual ports in the absence
of proper LACP negotiation with a network peer. Once any one port member receives a LACP
PDU from the peer, then Netvisor ONE bundles all port members of the trunk to operate as a
trunk.
With this configuration, Netvisor ONE creates the trunk in the switch, but does not add any of
the ports to the trunk. The ports continue to operate individually until the ports hear LACP
PDUs on any of the ports part of the trunk configuration. Once the port sends LACP PDUs
from the peer for trunk, then all ports of the trunk cease to operate individually and Netvisor
adds the ports to the trunk.
If the port does not receive LACP PDUs for the number of seconds configured as the fallback
timeout, Netvisor ONE port LACP checks for LACP negotiation expiration. If LACP negotiation
expires, then ports fall back to individual mode. If the port LACP negotiation has not expired,
Netvisor schedules another fallback timer at a value equal to the fallback timeout.
For example:
LACP fallback timeout is set to 50 seconds and LACP negotiation is set to default 90
seconds.
After 50 seconds, fallback timer is rescheduled because LACP negotiation has not
expired.
After an additional 40 seconds (90 total), LACP negotiation expires and become
inactive.
Another 10 seconds passes (100 seconds total) when the fallback timer expires and
the ports fallback to individual ports

To change the LACP failback mode and timeout, use the new parameters:

CLI network-admin@switch > vlag-modify

lacp-mode off|passive|active Modify the LACP mode.

lacp-timeout slow|fast Modify the LACP timeout.
lacp-fallback bundle|individua Modify the LACP fallback.
lacp-fallback-timeout seconds Modify the LACP fallback timer
between 30 and 60 seconds. The
default value is 50 seconds.
lacp-port-priority integer Configure the LACP port priority
as an integer between 1 and
65535.

76
Pluribus Networks
www.pluribusnetworks.com
Safely Restoring Ports for Cluster Configurations

Informational Note: This feature is only applied during the initial start up of the
cluster slave switch.

Sub-second traffic loss for fail over events is required for a cluster configuration. There are
two types of ports providing redundant data paths: 1) Layer 3 ports over ECMP redundant
routed paths, and 2) virtual LAGS (VLAGs) providing redundant Layer 2 paths. During
failover and recovery port events, it can take measurable time to change the hardware
routing and MAC tables on larger networks. This delay incurs traffic loss on the network. This
delay incurs traffic loss on the network. To reduce delay, this feature allows you to
incrementally restore these ports at start up. By incrementally restoring the ports, the
changes to the hardware are prevented from contending with each other and reduces the
delay between a port up and the hardware updates with the appropriate Layer 3 and Layer 2
information for the port. This process ensures sub-second fail over.
All non-Layer 3 and non-VLAG ports are restored first. This allows the cluster links to activate
and the cluster configuration to synchronize information. Layer 3 and VLAG port restoration
starts after the cluster synchronizes. This is predicated on the cluster becoming active, all
Layer 2 and Layer 3 entries, such as status updates, exchanged, cluster STP status
synchronized, and all router interfaces initialized.
The parameter, maximum-sync-delay, controls the maximum time to wait for synchronization
in the case where the cluster cannot synchronize information. After synchronization is
complete, Layer 3 ports are restored first, since Layer 3 traffic can traverse the cluster link to
the peer VLAG port if needed. Currently the reverse is typically not true.
If VLAG ports are restored first, a Layer 3 adjacency between the two cluster nodes may be
needed but may not exist in some network configurations. After Layer 3 ports are restored,
Netvisor ONE waits a configurable Layer 3 port to VLAG delay to allow time for the routing
protocols to converge and insert the routes. The delay time defaults to 15 seconds.
After the delay, the VLAG ports are restored incrementally. Incrementally restoring ports
allows enough time to move Layer 2 entries from the cluster link to the port. Incrementally
restoring ports also allows the traffic loss to occur in small, 200-300ms per port, rather than
one large time span. This is particularly important for server clusters where temporary small
losses are no issue, but fail or timeout for a large continuous traffic loss. If the node coming
up is the cluster master, then no staggering and no Layer 3 to VLAG wait is applied. And if
the node is the cluster master node, that means the peer is down or coming up, and not
handling traffic. Therefore Netvisor ONE safely restores the ports as soon as possible to start
traffic flowing between the nodes.
The following is the sequence of cluster bringup events:
1. Netvisor ONE re-establishes cluster ports and synchronize Layer 2 protocols, MAC learning
on cluster links.
2. Layer 3 ports establish connectivity to upstream neighbors.
3. VLAN interfaces learn IGP and IP routes.
4. VLAG ports re-establish LACP, Layer 2 protocols and MAC learning.
5. Orphan ports are re-established.

Pluribus Networks
www.pluribusnetworks.com
77
To configure a cluster for restoring Layer 3 ports, use the following commands:

cluster-bringup-modify Modifies the cluster bring up configuration.

Specify one or more of the following options
l3-port-bringup-mode Specify the Layer 3 port bring up mode during
staggered|simultaneous start up.
The recommended value is simultaneous.
l3-port-staggered-interval Specify the interval between Layer 3 ports in
duration: #d#h#m#s Layer 3 staggered mode. This can be in days,
hours, minutes, or seconds.
vlag-port-bringup-mode Specify the VLAG port bring up mode during
staggered|simultaneous start up.
The recommended value is staggered. This
allows enough time for LACP and LLDP protocols
to start and learn or move MAC addresses.
vlag-port-staggered-interval Specify the interval between VLAG ports in VLAG
duration: #d#h#m#s staggered mode.
This can be in days, hours, minutes, or seconds.
maximum-sync-delay duration: Specify the maximum delay to wait for cluster to
#d#h#m#s synchronize before starting Layer 3 or VLAG port
bring up.
This can be in days, hours, minutes, or seconds.
The recommended duration is 60 seconds.
l3-to-vlag-delay duration: #d#h#m#s Specify the delay between the last Layer 3 port
and the first VLAG port bring up.
This can be in days, hours, minutes, or seconds.
The default value is 15 seconds.
l3-to-vlan-interface-delay Specify the delay between the last Layer 3 port
duration: #d#h#m#s and vRouter VLAN interface bringup.
This can be in days, hours, minutes, or seconds.
The default value is 15 seconds.

To display the cluster port restoration configuration, use the cluster-bringup-show

command:

cluster-bringup-show Displays the cluster bring up configuration

information.

78
Pluribus Networks
www.pluribusnetworks.com
Configuring Layer 2 Multipathing for Virtual Chassis Link Aggregation
(VLAG)
You can aggregate links between two switches by configuring Layer 2 multipathing and
virtual chassis Link Aggregation.
A virtual chassis Link Aggregation Group (VLAG) allows links physically connected to two
different switches to appear as a single Ethernet trunk to a third device. The third device can
be a server, switch, or any other networking device. A VLAG creates Layer 2 multipathing
which allows the creation of redundancy, and enabling multiple parallel paths between nodes.
A VLAG requires at least one cross connection between the two switches, also called peers,
where the VLAG links terminate. The specific ports connecting the different switches, do not
require explicit configuration before creating a VLAG.
VLAGs can provide the following benefits:
Allows a single device to use an Ethernet trunk across two access layer (Top of Rack)
switches.
Eliminates Spanning Tree Protocol (STP) blocked ports
Provides a loop-free topology
Provides fast convergence if a link or device fails.
Provides link-level resiliency.
Helps ensure high availability.
Netvisor ONE performs VLAG synchronization to coordinate active-standby and active-active
configurations using the following rules:
Active-Standby VLAGs — For VLAG, only one side, ports on the cluster node, may
be up at anytime. The other side is in standby mode. If there is conflict, the side with
the longest time up remains up and the other side is disabled.
Active-Active VLAGs — If both sides of the VLAG are up, Netvisor ONE adds port
egress rules to drop any packets that egress the VLAG port if the ingress port acts as
the cluster link. This prevents loops.
Netvisor ONE reports the state as up or down and synchronizes the state. For active-standby
VLAGs, Netvisor exchanges port up timestamps to resolve any contest if both ports become
active.
Netvisor ONE performs synchronization from the primary node to the secondary node. If the
secondary node requires synchronization, the secondary node sends a request to the primary
node to perform the synchronization.
Netvisor sends synchronization messages on a per-VLAG basis, and compares the local VLAG
port state with the peer VLAG port state. The port state then determines any port enable or
disable actions for active-standby VLAGs or port egress rule changes for active-active VLAGs.
VLAG synchronization occurs when a trigger happens on the configuration:
A VLAG is created or modified.
A VLAG member port is up or down.
A cluster-link is up or down.
For any port in an active-standby VLAG, Netvisor records the time up of the port, and sends
it as part of the VLAG synchronization message. The time up values are compared on both
nodes to determine the active port.

Pluribus Networks
www.pluribusnetworks.com
79
VLAG Topology Examples
Figure 6:L2 Design - Leaf and Spine with Active-Passive VLAG

80
Pluribus Networks
www.pluribusnetworks.com
Figure 7:L2 Design - Leaf and Spine with Active-Active VLAG

To create a VLAG for aggregating links connected to ports 70 on the local switch and the peer
called, eng-switch-b, you must first create a cluster configuration between the two
switches. Netvisor ONE switches must be members of a cluster configuration before you add
VLAGs to them.

Configuring Active-Active VLAG

Using the sample topology in Figure 8, “” on page 82, use the following steps to configure
Active-Active VLAG:

Informational Note: There must be a physical connection between PN-0 and PN-1
before you can configure VLAG.

Pluribus Networks
www.pluribusnetworks.com
81
Figure 8:Active-Active VLAG over a Trunk with a Switch and Host

Three Netvisor ONE switches in a common fabric with the Spine switch as the RSTP root.
Note that ports 19-22 on PN-0 and PN-1 connect to PN-2 (Spine). Port 26 connects PN-0
to PN-1 for the cluster configuration required for VLAG.
1. On PN-2, use the following command:

CLI network-admin@switch > stp-modify bridge-priority 4096

2. Create the fabric and add the switches:

On PN-2, use the fabric-create command:

CLI network-admin@switch > fabric-create name fab-VLAG

On PN-1, join the fabric:

CLI network-admin@switch > fabric-join name fab-VLAG

On PN-0, join the fabric:

CLI network-admin@switch > fabric-join name fab-VLAG

82
Pluribus Networks
www.pluribusnetworks.com
3. Create VLAN connectivity from the top switch to the bottom:
On PN-2, create the VLAN with scope fabric:

CLI network-admin@switch > vlan-create id 25 scope fabric

On PN-0, add the VLAN and untag the port connected to the host.

CLI network-admin@switch > vlan-port-add vlan-id 25 untagged ports 9

On PN-1, add the VLAN and untag the port connected to the host.

CLI network-admin@switch > vlan-port-add vlan-id 25 untagged ports 9

On PN-0, modify the host STP port to be an edge port.

CLI network-admin@switch > stp-port-modify port 9 edge

On PN-1, modify the host STP port to be an edge port.

CLI network-admin@switch > stp-port-modify port 9 edge

4. Create a cluster configuration between PN-1 and PN-0. This creates the cluster across
port 26.
On PN-0, enter the cluster-create command:

CLI network-admin@switch > cluster-create name VLAG cluster-node-1 PN-0

cluster-node-2 PN-1

5. You must disable ports between PN-2 and PN-0, and then create a static trunk between
them:
On PN-0, modify the ports facing PN-2:

CLI network-admin@switch > port-config-modify port 19,20 disable

6. Then create the trunk on PN-0:

CLI network-admin@switch > trunk-create name pn0-to-pn2 port 19,20 lacp-mode

off

CLI network-admin@switch > trunk-show format all layout vertical

switch: tac-1
trunk-id: 253
name: pn0-to-pn2
ports: none
speed: disable
egress-rate-limit: unlimited
autoneg: off
jumbo: on
enable: off
lacp-mode: off
lacp-priority: 0
lacp-timeout: slow
lacp-fallback: bundle
lacp-fallback-timeout: 50
lacp-individual: none

Pluribus Networks
www.pluribusnetworks.com
83
stp-port-cost: 2000
stp-port-priority: 128
reflect: off
edge-switch: no
pause: no
description:
loopback: off
receive-only: on
unknown-ucast-level: %
unknown-mcast-level: %
broadcast-level: %
lport: 0
rem-rswitch-port-mac: 00:00:00:00:00:00
rswitch-default-vlan: 0
status:
config:
trunk-hw-id: 0
send-port: 4294967295
routing: yes
host-enable: no

From the above output, you find the name of the trunk configuration, pn0-to-pn2. You
need this information to create the VLAG.
Then, on PN-1, repeat the same commands to create a trunk between PN-1 and PN-2.
7. You must disable ports between PN-2 and PN-1, and then create a static trunk between
them:
On PN-1, modify the ports facing PN-2:

CLI network-admin@switch > port-config-modify port 21,22 disable

CLI network-admin@switch > trunk-create name pn1-to-pn2 port 21,22 lacp-mode

off

CLI network-admin@switch > trunk-show format all layout vertical

switch: PN-0
intf: 129
name: pn1-to-pn2
port: 21-22
speed: 10g
autoneg: off
jumbo: off
enable: off
lacp-mode: off
lacp-priority: 32768
lacp-timeout: slow
reflect: off
edge-switch: no
pause: no
description:
loopback: off
mirror-only: off
lport: 0
rswitch-default-vlan: 0
port-mac-address: 06:60:00:02:10:80
status:

84
Pluribus Networks
www.pluribusnetworks.com
config:
send-port: 0
8. Now create the VLAG from the bottom switches going upward and static trunk from the
top down. Keep one side of the VLAG disabled while you configure this step.
On PN-0, use the VLAG-create command:

CLI network-admin@switch > VLAG-create name to-spine port 128 peer-port 129
peer-switch PN-1 lacp-mode off mode active-active

On PN-2, create a trunk with the name trunk-pn:

CLI network-admin@switch > trunk-create name trunk-pn port 19,20,21,22

lacp-mode off

9. Now, enable ports on all switches:

On PN-2, enter the port-config-modify command:

CLI network-admin@switch > port-config-modify port 19,20,21,22 enable

On PN-0, enter the port-config-modify command:

CLI network-admin@switch > port-config-modify port 19,20 enable

On PN-1, enter the port-config-modify command:

CLI network-admin@switch > port-config-modify port 21,22 enable

10. Create the server-facing VLAG:

On PN-0, enter the VLAG-create command:

CLI network-admin@switch > vlag-create name to-spine port 9 peer-port 9

peer-switch PN-1 lacp-mode active mode active-active

Display the VLAG configuration information:

CLI network-admin@switch > vlag-show format all layout vertical

id: a000024:0
name: to-spine
cluster: VLAG
mode: active-active
switch: pubdev02
port: trunk2
peer-switch: pubdev01
peer-port: 129
failover-move-L2: no
status: normal
local-state: enabled,up
lacp-mode: off
lacp-timeout: slow
lacp-key: 26460
lacp-system-id: 110013777969246

Pluribus Networks
www.pluribusnetworks.com
85
Routing over VLAGs
When you use a routing protocol such as OSPF for peering over a VLAG, the cluster egress
filtering rules prevent routed packets from egressing VLAGs, if the packet crosses the cluster
links before egressing the VLAG.
In some suboptimal network designs, for example when you create vrouter-interfaces on
only one side of the cluster due to the use of /30 network addresses, and the cluster
active-active routing feature does not help with the implementation. These suboptimal
network designs may cause Layer 3 traffic to route through the cluster even when you use
VLAG as the point of exit a VLAG locally up on each cluster member. To help in those cases,
Netvisor provides a new parameter to allow packets crossing the cluster to egress out of
VLAGs, if the packet follows a route after crossing the cluster. To enable this feature, use the
following command:

system-settings-modify
routing-over-vlags| Specify if you want to enable or
no-routing-over-vlags disable routing to VLAGs from cluster
links
CLI (network-admin@vs-spine1) > system-settings-show
switch: spine1
routing-over-vlags: off
switch: vanquish2
switch: vanquish4
routing-over-vlags: off
switch: vanquish3
routing-over-vlags: off

CLI (network-admin@spine11) > system-settings-modify routing-over-vlags

CLI (network-admin@spine1) > switch-local system-settings-show

switch: spine1
routing-over-vlags: on

86
Pluribus Networks
www.pluribusnetworks.com
Configuring Virtual Wire Features

Informational Note: Netvisor ONE Virtual Wire only supports on the following
switches: AS5712-54X, AS6712-32X, AS7716-32X, F9272-X, F9232-Q, and F9532-C.
Netvisor does not support Virtual Wire on Dell platforms.

 Overview

 Prerequisites

 Enabling Virtual Wire Mode

 Configuring Ports for Virtual Wire Mode
 Implementing Unidirectional and Bidirectional Virtual Wire Links
 Support for CRC Checks for Virtual Wire Mode
 Support for Many to One Port Associations
 Building a Virtual Wire Fabric
 Example: Configuring a Fabric for Virtual Wire Switches
 Example: Configuring a Fabric for Unidirectional Virtual Wire
 Inline Services for Virtual Wire
 Configuring and Displaying Statistics
 Adding VCF-Insight Analytics for Network Visibility

Overview
Virtual Wire technology uses a software approach to configure cable topologies to
interconnect network devices together. Network devices are physically connected to the
Virtual Wire switch once using Ethernet cables and transceivers that match the device port
media and speed characteristics. The desired cable topology is then obtained by a remote
software configuration of the Virtual Wire switch and consists of a set of Virtual Wire links.
Virtual Wire topology configurations can be dynamically created, saved and re-applied
without any manual intervention on the physical infrastructure.
Netvisor Virtual Wire implementation uses transparent low-latency Ethernet forwarding
between physical ports over a non-blocking any-port to any-port switching architecture.
Virtual Wire transparently cross bridges any standard or proprietary Ethernet protocol of any
size, including these types of traffic:
 IPv6, Q-in-Q, VN-TAG

 Ethernet control plane traffic such as BPDU, LACP and LLDP protocol packets
 Proprietary or experimental Ethernet fabric
 Undersized or invalid frames
Network devices interconnected through a Virtual Wire link behave as directly connected
devices with a single physical cable. For example, as shown in Figure 1, if the port of Device
A goes down, the Virtual Wire switch automatically shuts down the port facing Device B.

Pluribus Networks
www.pluribusnetworks.com
87
Figure 1: Virtual Wire Topology

In addition, a Virtual Wire switch can act as an intelligent media converter, enabling Ethernet
communication between devices with different port speed and media type. In the example
shown in Figure 2, you create a Virtual Wire link between an optical cable connecting device
A and a copper cable on device B.
Figure 2: Virtual Wire Topology with Optical and Copper Cables

Prerequisites
Please refer to Adding License Keys to Netvisor in this guide for how to install a license on the
switch.
Virtual Wire functionality uses any supported Pluribus Network transceiver at
1Gbs/10Gbs/40Gbs. For a list of supported transceivers please refer to the product
datasheet.
All commands described in this chapter require a management fabric. Information on how to
create or join a management fabric is available in section Introduction to Netvisor Fabric.

88
Pluribus Networks
www.pluribusnetworks.com
To add the Virtual Wire feature to an existing Pluribus Networks switch in your network, you
must use the switch-config-reset command to erase the current configuration. Then, after
reconfiguring the initial setup, you must upgrade to the latest version of Netvisor that
supports Virtual Wire mode. And then, install the license key for Virtual Wire.

Informational Note: Before using the REST APIs described in this

document, be sure to set up Swagger as your REST API reference.Refer to
the instructions in the Configuring REST API Clients chapter. To use the
REST API feature, you must enable the Web service on the switch using the
command, admin-service-modify web on.

Enabling Virtual Wire Mode

To setup the switch in Virtual Wire mode, you must install the required license key:

CLI network-admin@Leaf1 > software-license-install key bear,stage,grey,book

The following command instructs the switch to operate in Virtual Wire mode and it is used to
enable global Virtual Wire functionality on a switch:

CLI network-admin@Leaf1 > switch-mode-modify switch-mode virtual-wire

Informational Note: When Virtual Wire mode is enabled, regular Layer 2

switching is inhibited on the switch. This includes Layer 2 learning and
forwarding as well as the processing and forwarding of BPDUs.

You now select the Virtual Wire mode, using the switch-mode-modify command:

CLI network-admin@switch > switch-mode-modify switch-mode virtual-wire

To display the switch mode, use the switch-mode-show command:

CLI network-admin@Leaf1 > switch-mode-show

switch: pluribus
switch-mode: virtual-wire

Configuring Ports for Virtual Wire Mode

By default, Netvisor configures all 10Gb switch ports for 10Gb Ethernet speed.
In 10Gb Ethernet mode, Netvisor requires SFP+ or QSFP+ transceivers to connect to hosts or
other switches. If you change the port speed to 1 Gigabit Ethernet, you need SFP
transceivers to plug into the ports. You must also enable jumbo frames for the port.
To modify the ports to 1Gb speed and enable auto-negotiation, use the following syntax:

Pluribus Networks
www.pluribusnetworks.com
89
CLI network-admin@switch > port-config-modify port 1-8 speed 1g autoneg jumbo

Informational Note: Jumbo frames are disabled by default, but Virtual

Wire links support any frame size.

REST API - PUT http://<switch-ip>/vRestport/port-configs/1-8/

{
"speed": "1g",
"autoneg": "autoneg",
}

To display port configuration information, use the port-config-show command. To see all
output, add the parameters format all layout vertical. Using the vertical layout displays
the information in a more readable format:

CLI network-admin@Leaf1 > port-config-show port 1 format all layout vertical

switch: vw-switch01
intf: 1
port: 1
speed: 1g
egress-rate-limit: unlimited
autoneg: on
jumbo: on
enable: on
lacp-priority: 32768
lacp-individual: none
stp-port-cost: 2000
stp-port-priority: 128
reflect: off
edge-switch: no
pause: no
description:
loopback: default
mirror-only: off
lport: 1
rem-rswitch-port-mac: 00:00:00:00:00:00
rswitch-default-vlan: 0
port-mac-address: 06:a0:00:02:40:1e
send-port: 0
routing: yes
host-enable: yes

REST API - GET http://<switch-ip>/vRest/port-configs/1

To display the port status, use the port-show command. Your output looks similar to the
following:

90
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@Leaf1 > port-show format all layout vertical

switch: pubdev02
port: 47
ip: 192.168.42.30
mac: 64:0e:94:28:03:56
hostname: pubdev03
status: up,PN-fabric,LLDP
lport: 47
rport: 47
config: fd,10g
trunk: trunk2
switch: pubdev02
port: 48
ip: 192.168.42.30
mac: 64:0e:94:28:03:56
hostname: pubdev03
status: up,PN-fabric,LLDP
lport: 48
rport: 48
config: fd,10g
trunk: trunk2

REST API - GET http://<switch-ip>/vRest/ports

To display all details about ports, use the port-phy-show command:

CLI network-admin@Leaf1 > port-phy-show format all layout vertical

switch: pubdev02
port: 25
state: up
autoneg: none
speed: 10000
eth-mode: 10Gbase-cr
max-frame: 1540
link-quality: great (59/41)
learning: off
def-vlan: 1
dfe-mode: continuous
dfe-coarse: complete
dfe-fine: complete
switch: pubdev02
port: 26
state: up
autoneg: none
speed: 10000
eth-mode: 10Gbase-cr
max-frame: 1540
link-quality: good (57/38)
learning: off
def-vlan: 1
dfe-mode: continuous
dfe-coarse: complete
dfe-fine: complete

REST API - GET http://<switch-ip>/vRest/port-phys

Pluribus Networks
www.pluribusnetworks.com
91
 Informational Note: The columns def-vlan, max-frame, and learning
display default fixed values because Virtual Wires diesables regular
switching on the ports.

Informational Note: Link-quality information is only available when a

you install a 10Gbps transceiver in a port.

To display the transceivers connected to the ports, use the port-xcvr-show command:

CLI network-admin@Leaf1 > port-xcvr-show port 1-4

switch port vendor-name part-number serial-number supported

-------- ---- ---------------- ------------ ------------- ---------
pubdev02 1 PluribusNetworks SFP10-CU0P5M Y05B200393 Yes
pubdev02 2 PluribusNetworks SFP10-CU0P5M Y05B200747 Yes
pubdev02 3 PluribusNetworks SFP10-CU0P5M Y05B200413 Yes
pubdev02 4 PluribusNetworks SFP10-CU0P5M Y05B200804 Yes

REST API - GET http://<switch-ip>/vRest/port-xcvrs

Informational Note: Each port has a LED indicator light that displays
status information about the port. If the LED indicates solid green, Netvisor
enables the port is enabled. If the LED indicates green and blinking rapidly
then the port is at 80% of the throughput capacity.

Implementing Unidirectional and Bidirectional Virtual Wire Links

In this section you configure a single bidirectional Virtual Wire link using the
port-association-create command with the option virtual-wire to create a single
bidirectional Virtual Wire link. Netvisor implements in two ways that are functionally
equivalent:
 Configuring Virtual Wire direction individually - or

 Configuring a Virtual Wire link using the bidir parameter

1. To configure a unidirectional Virtual Wire link from device A to device B, enter the following
command:

CLI network-admin@Leaf1 > port-association-create name A-to-B virtual-wire

master-ports 10 slave-ports 20

REST API - POST http://<switch-ip>/vRest/port-associations

{
“name”: “A-to-B”,

92
Pluribus Networks
www.pluribusnetworks.com
“virtual-wire”:”true”,
“master-ports”: “10”,
“slave-ports”: “20”
}

Informational Note: Please note that the parameter “mode(?) must be

set to “true”. This is the case when the switch is running in Virtual Wire
mode and you are configuring Virtual Wire features.

1. To configure a unidirectional Virtual Wire link from device B to device A, enter the following
command:
port-association-create name B-to-A virtual-wire master-ports 20 slave-ports 10

REST API - POST http://<switch-ip>/vRest/port-associations

{
“name”: “B-to-A”,
“virtual-wire”:”true”,
“master-ports”: “20”,
“slave-ports”: “10”
}
1. To configure a bidirectional Virtual Wire link from device A to device B, enter the following
command:

CLI network-admin@Leaf1 > port-association-create name A-to-B bidir

virtual-wire master-ports 10 slave-ports 20

REST API - POST http://<switch-ip>/vRest/port-associations

{
“name”: “A-to-B”,
“bidir”,
“virtual-wire”:”true”,
“master-ports”: “20”,
“slave-ports”: “10”
}
To display existing Virtual Wire links, use the port-association-show command:

CLI network-admin@Leaf1 > port-association-show

switch name master-ports slave-ports policy virtual-wire bidir

--------- ------ ----------- ----------- ----------- ------------ -----
vw-switch A-to-B 10 20 all-masters true true

REST API - GET http://<switch-ip>/vRest/port-associations

To delete an existing Virtual Wire link, use the port-association-delete command with
the name string parameter:

Pluribus Networks
www.pluribusnetworks.com
93
CLI network-admin@Leaf1 > port-association-delete name A-to-B

REST API - DELETE http://<switch-ip>/vRest/port-associations/A-to-B

Support for CRC Checks for Virtual Wire Mode

A switch running in Virtual Wire Mode currently interprets the CRC header of the packets
passing through. This achieves perfect transparency of the switch. However it does place
limitations on the types of vFlows created on the switch, as any vFlow that modifies the
packet renders the CRC on that packet invalid without updating it.
Netvisor implements the CRC regeneration as a configurable option per port, so you decide
on a per-port basis whether the switch performs, or not perform CRC regeneration.

Figure 1: Example Virtual Wire Mode Topology

On the virtual-wire switch, to convert traffic on port 43 tagged with VLAN 101 as tagged with
VLAN 102 so Host1 and Host2 communicates as if the two hosts are on the same VLAN, then
you configure the following two vFlows:

CLI network-admin@Leaf1 > vflow-create name vlan_map_101_102 scope local table

L1-Virtual-Wire-1-0 vlan 101 in-port 43 precedence 15 action setvlan
action-value 102 action-to-ports-value 39

CLI network-admin@Leaf1 > vflow-create name vlan_map_102_101 scope local table

L1-Virtual-Wire-1-0 vlan 102 in-port 39 precedence 15 action setvlan
action-value 101 action-to-ports-value 43

However the packets with a different VLAN now have an incorrect CRC value unless you
update the CRC when egressing the port.
Use the following command:

CLI network-admin@Leaf1 > port-config-modify port 39,43 crc-check-enable

After this configuration, Netvisor updates any packets egressing from ports 39 and 43 with
the CRC check.

Informational Note: The parameter, crc-check-enable is only be available on

switches in Virtual Wire mode. Furthermore, when you change the switch mode to
Virtual Wire mode, Netvisor configures all ports as crc-check-disable by default.

94
Pluribus Networks
www.pluribusnetworks.com
Support for Many to One Port Associations
Currently, when you configure port associations for Virtual Wire mode, you specify a single
port for a master and a single port for a slave. This is a limitation for use cases when the
traffic needs to be spread out across several ports.
Netvisor now provides the ability to specify multiple ports for master and slave parameters
for port-associations in virtual-wire mode.

CLI network-admin@Leaf1 > port-association-create name PA_1 master-ports 1

slave-ports 2,3 virtual-wire

CLI network-admin@Leaf1 > port-association-create name PA_2 master-ports 2

slave-ports 1,3 virtual-wire

Add the parameter, monitor-ports, to allow for ports not tracked by the port-association.

CLI network-admin@Leaf1 > CLI> port-association-create name PA_1 master-ports 1

slave-ports 2 monitor-ports 3 virtual-wire

CLI network-admin@Leaf1 > CLI> port-association-create name PA_2 master-ports 2

slave-ports 1 monitor-ports 3 virtual-wire

These commands create the same set of port-associations except that when ports 1 or 2
goes down, port 3 is not affected.

Packet Load Balancing over One to Many Links

When you deploy Virtual Wire as legacy packet broker, moving packets from production to an
analyzer tool, it requires load balancing feature because you monitor 10Gb links with 1Gb
tools.
Netvisor load balances the traffic by distributing the traffic load to different tool ports or
appliances in order to scale the monitoring.
This also provides redundancy in the monitoring technology.
When the member port goes down, Netvisor switches traffic on the port to remaining
member ports and evenly distributed.
To configure load balancing, use the following steps:
1. First configure a trunk on the desired ports. In this case, ports 15 and 16 are configured as
a trunk:

CLI network-admin@Leaf1 > trunk-create name lb_trunk ports 15,16

Created trunk lb_trunk, id 128

2. Create the port association on the switch:

CLI network-admin@Leaf1 > port-association-create name pa1 master-ports 1

slave-ports 128 virtual-wire bidir

3. Display the configuration:

Pluribus Networks
www.pluribusnetworks.com
95
CLI network-admin@Leaf1 > port-association-show

switch name master-ports slave-ports policy virtual-wire bidir

----------- ---- ------------ ----------- ----------- ------------ -----
vw-switch-1 pa1 1 128 all-masters true true

CLI network-admin@Leaf1 > port-show port 1,16,16

switch port vnet hostname status config trunk

----------- ---- ---- -------- ------ --------- --------
vw-switch-1 1 40g,jumbo
vw-switch-1 16 trunk 10g,jumbo lb_trunk

CLI network-admin@Leaf1 > vflow-show

switch name scope type in-port ether-type dst-ip

proto
----------- ----------------------- ----- ------ ------- ---------- ---------
-----
vw-switch-1 Internal-Keepalive local system ipv4 239.4.9.7
udp
vw-switch-1 VIRT_WIRE_MAS_SLV local system 1

vw-switch-1 VIRT_WIRE_SLV_MAS local system 128

tcp-flags flow-class precedence action action-to-ports-value enable

--------- ---------- ---------- ----------- --------------------- ------
control 14 to-cpu enable
14 to-port 128 enable
14 to-port 1 enable

table-name
-------------------

L1-Virtual-Wire-1-0
L1-Virtual-Wire-1-0

Building a Virtual Wire Fabric

You interconnect multiple Virtual Wire switches to form a single Virtual Wire fabric. A Virtual
Wire fabric acts as a highly scalable and distributed patch panel that you dynamically and
remotely provision to implement single dedicated wire speed links between any two device
ports in the network.
When you add all of the switches in the Virtual Wire fabric to the same Management Fabric,
you provision and control as a single logical Virtual Wire switch.

Informational Note: A Management Fabric for more than four Virtual

Wire switches requires a software license already included on E68-M and
E28-Q switches.

96
Pluribus Networks
www.pluribusnetworks.com
The most efficient design for a Virtual Wire fabric uses the classic leaf-spine architecture, or
Clos, a nonblocking, multistage switching topology, as in Figure 4
Figure 4: Leaf and Spine Topology for Virtual Wire Fabric.

Informational Note: In Clos architecture, Netvisor does not limit the

number of Virtual Wire links between device ports physically connected to
the same leaf. Instead, the number of Virtual Wire links between device
ports connected to different leafs depend on the oversubscription ratio
between leaf and spine.

With this approach, select the desired oversubscription ratio and build a modular and scalable
architecture to scale up to thousands of device ports. For example, using E68-M switches as
building blocks, a possible leaf switch configuration uses 44 x 10 Gigabit Ethernet ports to
connect to device ports and 24 x 10 Gigabit Ethernet ports to connect to the spine layer,
resulting in a 1.8:1 oversubscription ratio. Based on the desired maximum number of device
ports, you select from different scale options:
Figure 5: 17 Leafs and 6 Spines

 17 leaf 6 spine at 1.8:1 oversubscription ratio for a total of 748 device 10 Gbps/1Gbps
ports

Pluribus Networks
www.pluribusnetworks.com
97
Figure 6: 34 Leafs and 12 Spines

 34 leaf 12 spine at 1.8:1 oversubscription ratio for a total of 1496 device 10Gbps/1Gbps
ports
Figure 7: 68 Leafs and 24 Spines

 68 leaf 24 spine at 1.8:1 oversubscription ratio for a total of 2992 device 10Gbps/1Gbps
ports

Example: Configuring a Fabric for Virtual Wire Switches

The configuration example in this section refers to a Virtual Wire fabric composed by one
spine and two leaf switches as in Figure 8. Two devices, device-A and device-B, have
respectively two ports and one port that are physically connected to the Virtual Wire switch
Leaf-1. A third device, device-C, is physically connected to the Virtual Wire switch Leaf-2. The
desired logical setup consists in a bidirectional service chain topology where device-A is
inserted in-line between device-B and device-C.

98
Pluribus Networks
www.pluribusnetworks.com
Figure 8: Bidirectional Traffic over a Virtual Wire Connection

To create a bidirectional virtual link from device-A to device-C, use these steps:
1. Configure a port association for device-A to device-C using port 1 and port 45 on Leaf-1.

network-admin@Leaf-1>port-association-create name link-AC virtual-wire

bidir master-ports 1 slave-ports 45

2. Configure a port association on Spine-1 between ports 1 and 2:

network-admin@Spine-1>port-association-create name link-AC virtual-wire bidir
master-ports 1 slave-ports 2
3. Configure a port association on Leaf-2 between ports 45 and 1:

network-admin@Leaf-2>port-association-create name link-AC virtual-wire

bidir master-ports 45 slave-ports 1

4. Configure a port association from device-A to device-B:

network-admin@Leaf-1>port-association-create name link-AB virtual-wire

bidir master-ports 2 slave-ports 3

Example: Configuring a Fabric for Unidirectional Virtual Wire

Unidirectional Virtual Wire links can be used for testing link fault signaling features like Cisco
Unidirectional Link Detection (UDLD). In the example in Figure 5, Virtual Wire separates the
traffic directions and individually controls creating a unidirectional Virtual Wire links in a
Virtual Wire fabric composed by one spine and two leaf switches. In the resulting logical
topology, device-B and device-C are directly interconnected in one direction. In the opposite
direction insert device-A, a traffic impairment tool, in-line. Use Device-A to introduce errors
on the wire or to emulate unidirectional fiber cut events.

Pluribus Networks
www.pluribusnetworks.com
99
Figure 5: Unidirectional Traffic over a Virtual Wire Connection

To configure the Virtual Wire switch for unidirectional traffic, use the following steps:
1. Configure a port association on Leaf-1, ports 1 and 45.

network-admin@Leaf-1>port-association-create name link-AC virtual-wire

master-ports 1 slave-ports 45

2. Configure a port association on Spine-1, ports 1 and 3:

network-admin@Spine-1>port-association-create name link-AC virtual-wire

master-ports 1 slave-ports 3

3. Configure a port association on Leaf-2, ports 45 and 1:

network-admin@Leaf-2>port-association-create name link-AC virtual-wire

master-ports 45 slave-ports 1

This configuration connects device-A to device-C over a unidirectional virtual wire link.
To connect device-C to device-B over a unidirectional virtual link, use the following steps:
1. Configure a port association on Leaf-1 for ports 3 and 46:

network-admin@Leaf-1>port-association-create name link-CB virtual-wire

master-ports 3 slave-ports 46

2. Configure a port association on Spine-1 for ports 2 and 4:

network-admin@Spine-1>port-association-create name link-CB virtual-wire

master-ports 2 slave-ports 4

100
Pluribus Networks
www.pluribusnetworks.com
3. Configure a port association on Leaf-2 for ports 46 and 1:

network-admin@Leaf-1>port-association-create name link-CB virtual-wire

master-ports 46 slave-ports 1

This configuration connects over a unidirectional virtual wire link.

To configure a virtual wire connection between device-B and device-A:
1. Configure a port association on Leaf-1 for ports 3 and 2:

network-admin@Leaf-1>port-association-create name link-BA virtual-wire

master-ports 3 slave-ports 2

Inline Services for Virtual Wire

The Inline Service feature manages service chains for Layer 1 Virtual Wire switches. The
term, Inline Services, refers to services attached to a Layer 1 Virtual Wire switch such as
Next-Generation Firewall (NGFW), Intrusion Detection System (IDS), Intrusion Prevention
System (IPS), and Distributed Denial of Service attack (DDoS) Prevention.
When an Inline Service fails, a policy determines if Netvisor allows traffic to bypass the Inline
Services or if Netvisor blocks the traffic until the Inline Services recovers.
For security services such as NGFW, IDS, IPS, and DDoS, important for any network
deployment, Inline Services provides continuous monitoring of the network for improved
security. Inline security services can fail due to power failure, maintenance or other reasons.
An Inline Service failure has the potential to affect the flow of traffic in the network,
potentially bringing the network down. This requires continues monitoring of services on
network for better security.
To safeguard against such failures, the Inline Service feature provides a way to steer traffic
around the failed Inline Service so as not to impact traffic. During a failure, Inline Services
does not protect the service.
Netvisor detects Inline Service recover and failure by the port link states, UP and DOWN,
between the Layer 1 Virtual Wire switch and the Inline Service.
However a device connected to the switch can fail without the port sending an UP or Down
link state. In such cases, Netvisor relies on a heartbeat, or a probe in a form of a pre-defined
packet, sent to an attached device.

Pluribus Networks
www.pluribusnetworks.com
101
Figure 1: Example of Inline Services
You configure the order of the Inline Services using the port-association-service-*
commands.
If you configure an inline service with the parameter, fail-open, Netvisor sends traffic and
skips any Inline Services failing on the network.
For example, if you configure Inline Services with the chain 1->2->3->4->5, and the Inline
Service 3 fails, the new chain is 1->2->4->5.
If you configure an Inline Service with the parameter, fail-close, and any Inline Service
fails, Netvisor blocks network traffic. For example, if you configure the chain
1->2->3->4->5, and any Inline Service such as 2, 3, or 4 fails, no network traffic flows
through the chain, and network traffic flow stops.
Configuring Heartbeat Service
Netvisor generates a packet from the CPU to send to the receive port of an Inline Service and
the Netvisor vFlow configured for snooping is not port-specific, as Netvisor accepts the
response from either the receive port or the transmit port. Configure the heartbeat as an
additional parameter for a specific Inline Service.

102
Pluribus Networks
www.pluribusnetworks.com
For example, to create a heartbeat detection service named FW-Probe, use the following
syntax:
(CLI network-admin@Spine1)>service-heartbeat-create name FW_probe interval 5s
retry 3 vlan-id 10 src-mac 64:6e:11:1c:11:11 dst-mac 01:1b:11:01:01:01 type
normal payload 54 63 82 ff 01 46 12 ce a2 d4 00 00 00 00 00 00 00 00

In this example, you define the frequency of the heartbeats as well as the number of missed
probes before Netvisor detects the service with this heartbeat is down.
To add the Heartbeat Service to Inline Services, FW-1 and FW-2, use the following syntax:
(CLI network-admin@Spine1)>inline-service-create name FW1 tx-port 11 rx-port
11 heartbeat FW_probe
(CLI network-admin@Spine1)>inline-service-create name FW2 tx-port 9 rx-port 10
heartbeat FW_probe

Netvisor counts the missed heartbeats separately for FW-1 and FW-2.

Configuring the Payload

Specify the payload as a packet including Ethertype of the packet, but excluding the CRC at
the end. For example, an ARP packet uses this format:
Payload(including CRC):

0: ffff ffff ffff 0011 0100 0001 0806 0001 ................

16: 0800 0604 0001 0011 0100 0001 0101 0101 ................
32: 0000 0000 0000 0101 0102 0000 0000 0000 ................
48: 0000 0000 0000 0000 0000 0000 2160 cc6b ............!`.k

A heartbeat service, HB_4 for this ARP packet has the following syntax:
(CLI network-admin@Spine1)>service-heartbeat-create name HB4_arp interval 1s
retry 10 vlan 1 src-mac 00:11:01:00:00:01 dst-mac ff:ff:ff:ff:ff:ff payload
"0806 0001 0800 0604 0001 0011 0100 0001 0101 0101 0000 0000 0000 0101 0102
0000 0000 0000 0000 0000 0000 0000"

When you create the Heartbeat Service, Netvisor installs a specific vFlow in the vFlow table.
Netvisor verifies the functionality of the Inline Service using two methods: 1) a normal
heartbeat, and 2) a passthrough heartbeat. When you configure the parameter, type, you
specify the type of heartbeat for the service as normal, a request-response heartbeat
indicating the service responds to the heartbeat. If you specify pass-through as the
heartbeat, Netvisor sends the packet and returns it the switch through the service.

Configuring Inline Services with a Heartbeat Service

To configure the example topology displayed inFigure 1 Example of Inline Servicesuse the
following steps:
1. Configure the North-South port association, use the following syntax:
(CLI network-admin@Spine1)>port-association-create name NorthToSouth
master-ports 1 slave-ports 8 virtual-wire no-bidir

Pluribus Networks
www.pluribusnetworks.com
103
2. Define and configure the Heartbeat Service parameters:

(CLI network-admin@Spine1)>service-heartbeat-create name FW_probe interval 5s

retry 3 vlan-id 10 src-mac 64:6e:11:1c:11:11 dst-mac 01:1b:11:01:01:01 type
passthrough payload 54 63 82 ff 01 46 12 ce a2 d4 00 00 00 00 00 00 00 00

3. Configure the Inline Services chain:

(CLI network-admin@Spine1)>port-association-service-add port-association-name

NorthToSouth inline-service IPS order 2 policy-action fail-open
(CLI network-admin@Spine1)>port-association-service-add port-association-name
NorthToSouth inline-service DDoS order 3 policy-action fail-open
(CLI network-admin@Spine1)>port-association-service-add port-association-name
NorthToSouth inline-service NGWF order 4 policy-action fail-closed

Netvisor uses new commands to configure Heartbeat Services:

(CLI network-admin@Spine1)>service-heartbeat-create

name name-string Specify a name for the Heartbeat Service.

interval duration: #d#h#m#s Specify the interval between heartbeat
packets.
retry retry-number Specify the number of times to retry
sending a packet.
vlan vlan-id5 Specify a VLAN ID.
src-mac mac-address Specify the source port MAC address.
dst-mac mac-address Specify the destination MAC address.
type normal|pass-through Specify the type of heartbeat response as
normal or passthrough. A normal
response indicates that the Inline Service
sends the response. A passthrough
response indicates that Netvisor sends the
response and returns it to the Inline
Service.
payload payload-string Specify the payload for the heartbeat
packet.

(CLI network-admin@Spine1)>service-heartbeat-delete

name name-string Specify a name for the Heartbeat Service.

(CLI network-admin@Spine1)>service-heartbeat-modify

name name-string Specify a name for the Heartbeat Service.

interval duration: #d#h#m#s Specify the interval between heartbeat
packets.
retry retry-number Specify the number of times to retry
sending a packet.

(CLI network-admin@Spine1)>service-heartbeat-show

104
Pluribus Networks
www.pluribusnetworks.com
name name-string Displays the name for the Heartbeat
Service.
interval duration: #d#h#m#s Displays the interval between heartbeat
packets.
retry retry-number Displays the number of times to retry
sending a packet.
vlan vlan-id5 Displays a VLAN ID.
src-mac mac-address Displays the source port MAC address.
dst-mac mac-address Displays the destination MAC address.
type normal|pass-through Displays the type of heartbeat response as
normal or passthrough. A normal
response indicates that the Inline Service
sends the response. A passthrough
response indicates that Netvisor sends the
response and returns it to the Inline
Service.
payload payload-string Displays the payload for the heartbeat
packet.

Configuring Service Chains

Configure A service chain i using port-association-service-* commands. Manage the
services in the chain using inline-service-* commands.
Configure Inline Services using the following commands:
(CLI network-admin@Spine1)>port-association-service-add

port-association-name Specify the name of the port association to apply the

name-string service.
switch name-string Specify the switch name where the service is located.
inline-service Specify the name of the Inline Service.
inline-service-name
order number Specify a number to designate the order of the
service. This is a value between 1 and 65535
policy-action Specify a policy action when the service fails on the
fail-open|fail-closed network.
(CLI network-admin@Spine1)>port-association-service-modify

port-association-name Specify the name of the port association to apply the

name-string service.
switch name-string Specify the switch name where the service is located.
inline-service Specify the name of the Inline Service.
inline-service-name
order number Specify a number to designate the order of the
service. This is a value between 1 and 65535
policy-action Specify a policy action when the service fails on the
fail-open|fail-closed network.

(CLI network-admin@Spine1)>port-association-service-remove

Pluribus Networks
www.pluribusnetworks.com
105
port-association-name Specify the name of the port association to apply the
name-string service.
switch name-string Specify the switch name where the service is located.
inline-service Specify the name of the Inline Service.
inline-service-name

(CLI network-admin@Spine1)>port-association-service-show

port-association-name Displays the name of the port association to apply

name-string the service.
switch name-string Displays the switch name where the service is
located.
inline-service Displays the name of the Inline Service.
inline-service-name
order number Displays a number to designate the order of the
service. This is a value between 1 and 65535
policy-action Displays a policy action when the service fails on the
fail-open|fail-closed network.

(CLI network-admin@Spine1)>inline-service-create

name name-string Specify a name for the Inline Service.

tx-port port-list Specify the transmit port for the Inline Service.
rx-port port-list Specify the receive port for the Inline Service.

(CLI network-admin@Spine1)>inline-service-delete

name name-string Specify a name for the Inline Service.

(CLI network-admin@Spine1)>inline-service-show

name name-string Specify a name for the Inline Service.

tx-port port-list Specify the transmit port for the Inline Service.
rx-port port-list Specify the receive port for the Inline Service.

Configuring and Displaying Statistics

Display standard statistics consisting of flow-based information collected and tracked
continuously by the switch. To modify statistics logging, use the stats-log-modify
command and disable or enable statistical logging as well as change the interval, in seconds,
between statistical events.
To show connection-level statistics, traffic flows between a pair of hosts for an application
service, including current connections and all connections since the creation of the fabric,
enter the following CLI command at the prompt:

106
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@Leaf1 > connection-stats-show

switch: pubdev02
count: 0
mac: 64:0e:94:28:00:8e
vlan: 3
ip: 192.168.42.10
port: 25
iconns: 6
oconns: 0
ibytes: 224K
obytes: 10.5K
total-bytes: 235K
first-seen: 02-26,17:19:52
last-seen: 02-26,17:19:57
last-seen-ago: 17d14h6m5s
switch: pubdev02
count: 0
mac: 64:0e:94:28:03:56
vlan: 3
ip: 192.168.42.30
port: 128
iconns: 0
oconns: 3946878
ibytes: 4.50M
obytes: 13.5M
total-bytes: 18.0M
first-seen: 01-06,09:23:07
last-seen: 08:25:20
last-seen-ago: 42s

REST API - GET http://<switch-ip>/vRest/connection-stats

From the information displayed in the output, you can see statistics for each switch, VLANs,
client and server IP addresses, as well as the services on each connection. Netvisor also
displays latency and other information.
The latency (us) column displays the running latency measurement for the TCP connection in
microseconds. It indicates end-to-end Round-Trip-Time (RTT) between TCP/IP session client
and server and includes the protocol stack processing for the connected hosts and all
intermediary network hops.
To display connection latency, use the connection-latency-show command:

Pluribus Networks
www.pluribusnetworks.com
107
CLI network-admin@Leaf1 > connection-latency-show

switch min max num-conns percent avg-dur obytes ibytes total-bytes

-------- ------ ------ --------- ------- ------- ------ ------ -----------
switch-v 0.00ns 20.0us 67.5K 76% 17.12m 32.9K 18.0K 51.0K
switch-v 20.0us 40.0us 2.74K 3% 1.64h 8.77M 123M 132M
switch-v 40.0us 60.0us 10.4K 11% 1.40h 22.0M 403M 425M
switch-v 60.0us 80.0us 1.85K 2% 1.10h 8.16M 127M 135M
switch-v 80.0us 100us 901 1% 1.02h 3.39M 53.5M 56.9M
switch-v 100us 120us 1.35K 1% 1.23h 5.49M 126M 132M
switch-v 120us 140us 801 0% 1.06h 5.67M 39.2M 44.9M
switch-v 140us 160us 545 0% 1.19h 1.88M 29.4M 31.3M
switch-v 160us 180us 1.08K 1% 1.21h 5.04M 82.8M 87.8M
switch-v 180us 200us 583 0% 56.77m 5.15M 72.7M 77.8M
switch-v 200us 729 0% 48.51m 2.57G 184M 2.75G

REST API - GET http://<switch-ip>/vRest/connection-latencies

Adding VCF-Insight Analytics for Network Visibility

Virtualization-Centric Fabric Insight Analytics (VCF™ Insight Analytics) is an application
developed by Pluribus Network that enables the network administrator to extract the
analytical value from the telemetry data reported by the network switches powered by
Pluribus Networks Netvisor® network operating system.
VCF Insight Analytics connects the switches as part of the Netvisor® fabric to gain visibility
into the network, and extract all the telemetry data made visible by the Pluribus Network
Operating System.
Once data is collected, VCF Insight Analytics relies upon a modern search engine database
infrastructure to store, aggregate, filter, correlate and visualize vast amounts of data in
real-time as well as with a powerful “time machine” functionality.
As shown in Figure 1, VCF IA can be deployed in a Virtual Wire fabric topology, by connecting
the VCF IA server(s) to the fabric management network. Using Netvisor® API, VCF IA
establishes a connection to any Virtual Wire switch in fabric to gain visibility into the entire
fabric network and extract all the device layer telemetry data made visible by the distributed
Pluribus Network Operating System.
Once data is collected, VCF™ IA relies upon a modern search engine database infrastructure
to store, aggregate, filter, correlate and visualize vast amount of data in real time.

108
Pluribus Networks
www.pluribusnetworks.com
Figure 1: Overview of VCF-IA Topology

To add VCF to your network, please visit

http://www.pluribusnetworks.com/products/vcf-insight-analytics/

Pluribus Networks
www.pluribusnetworks.com
109
Configuring Layer 2 Features
 Configuring Tagged and Untagged VLANs
 Configuring Rapid Spanning Tree Protocol (RSTP)
 Multiple Spanning Tree Protocol (MSTP)
 About VXLANs
 Configuring VXLANs and Tunnels
 VXLAN Routing In and Out of Tunnels
 VXLAN Port Termination
 Virtual Link Extension with Cluster Configurations
 Port Replication for Virtual Link Extensions
 Support for Configuring Keep-Alive Time for Virtual Link Extension (VLE)
 About Port Hairpinning
 Topic Feedback

Configuring Tagged and Untagged VLANs

Create untagged VLANs for connecting the switch to devices without support for IEEE 802.1Q
VLAN tags. You configure ports to map untagged packets to a VLAN.

Reserved VLANs and VLAN 0 and 1

Netvisor uses a 12-bit field in the header of each packet as a VLAN identifier. Therefore, the
maximum number of VLANs you can define is 4091. Netvisor OS switches reserve VLANs 0,
1, 4093, 4094, and 4095 for internal use.

Informational Note: Due to Rapid Spanning Tree Protocol (RSTP)

support, the F-series switch supports only 256 VLANs. The E-series
switch supports 512 VLANs.

VLAN 0 is not a standard VLAN in Netvisor OS. VLAN 0 represents all untagged or non-VLAN
traffic. Creating an interface on VLAN 0 does not add VLAN 0 to the VNET. Instead, it is a way
to bridge the VNET to the rest of the untagged network. If there is untagged traffic destined
for the VNET, then add VLAN 0 to the VNET. For example, if the default router for the switch
is untagged at IP address 12.15.1.1/24 and you want VNET, vnet1, to access the default
router, add VLAN 0 to the VNET using the following commands:

CLI network-admin@switch > vnet-manager-interface-add vnet-manager-name

vnet1-mgr vlan 0 ip 12.15.1.1/24

110
Pluribus Networks
www.pluribusnetworks.com
 Informational Note: For switches with ONVL, the only available VNET
is a global VNET created when a fabric is created for the first time. Use
tab complete in the CLI to display the VNET and continue the
configuration.

The VNET administrator can only add interfaces on VLAN 0 and VLANs assigned to the VNET.
The network administrator can add interfaces on any VLAN. If you want to bridge a VNET to
another VNET, the network administrator must add the interfaces.
Netvisor uses VLAN 1 as the default untagged traffic VLAN. Untagged traffic can be mapped
to any VLAN, but by default, Netvisor maps to VLAN 1.

Warning: If you create a VLAN with scope fabric and untag all ports,
you can cause problems with the fabric communication.

Informational Note:The untagged VLAN feature is not the same as the

default VLAN using the IEEE 802.1Q tag 1.

1. To create a VLAN on the current switch, with the identifier 595, use the following
command:

CLI network-admin@Leaf1 > vlan-create id 595 name VLAN595 scope local

By default, Netvisor trunks all ports on the new VLAN. If you want to specify trunked ports
, use the optional parameter, ports, with a comma separated list of ports, or specify a range
of ports.
In some cases, you may not want the VLAN created on all ports. Specify none to apply the
VLAN to internal ports only.

CLI network-admin@Leaf1 > vlan-create id 35 scope fabric ports none

CLI network-admin@Leaf1 > vlan-show

switch: pubdev01
id: 35
nvid: a000030:23
scope: fabric
name: vlan-35
active: yes
stats: yes
vrg: 0:0
ports: 65-72,255
untagged-ports: none
active-edge-ports: none
switch: pubdev02

Pluribus Networks
www.pluribusnetworks.com
111
 To map ports on different switches into the scope fabric VLAN, use the following
command:

CLI network-admin@Leaf1 > vlan-port-add switch switch-name ports

To create a VLAN for a VNET, use the vnet-create command and include the VLANs that
map to the VNET.
To modify a VLAN name, use the vlan-modify command to modify VLAN 25 description
from blue to red:

CLI network-admin@Leaf1 > vlan-modify id 25 description blue

To modify the port list, use the vlan-port-add and the vlan-port-remove commands. If
you want to remove a VLAN with the scope, fabric, you need to specify the switch name.
2. To display the VLANs configured on the switch, use the vlan-show command.

CLI network-admin@Leaf1 > vlan-show format all layout vertical

switch: pubdev01
id: 1
nvid: a000030:1
scope: local
name: default-1
active: yes
stats: yes
vrg: 0:0
ports: 1-72,128,255
untagged-ports: 1-72,128,255
active-edge-ports: 31,45-46,66,128
active-edge-ports: 65,128-129
switch: pubdev02
id: 1
nvid: a000024:1
scope: local
name: default-1
active: yes
stats: yes
vrg: 0:0
ports: 1-72,128-129,255
untagged-ports: 1-72,128-129,255

3. To configure ports 17 and 18 to accept untagged packets and map them to VLAN 595, use
the following command:

CLI network-admin@Leaf1 > vlan-port-add vlan-id 595 ports 17,18 untagged

Displaying VLAN Statistics

Display network traffic statistics per VLAN using the vlan-stats-show command. This may
be useful when troubleshooting network issues.

112
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@Leaf1 > vlan-stats-show format all layout vertical

switch: pubdev03
time: 10:51:02
vlan: 1
vnet:
ibytes: 36.2T
ipkts: 89.0G
idrops-bytes: 119M
idrops-pkts: 313K
obytes: 0
opkts: 0
odrops-bytes: 0
odrops-pkts: 0
switch: pubdev03
time: 10:51:02
vlan: 35
vnet:
ibytes: 10.8K
ipkts: 154
idrops-bytes: 0
idrops-pkts: 0
obytes: 0
opkts: 0
odrops-bytes: 0
odrops-pkts: 0
switch: pubdev02
time: 10:51:02
vlan: 1
vnet:
ibytes: 34.9T
ipkts: 84.6G
idrops-bytes: 3.03M
idrops-pkts: 5.69K
obytes: 0
opkts: 0
odrops-bytes: 0
odrops-pkts: 0

The output displays the following information:

switch — switch name
time — when the output generated
VLAN ID — ID assigned to the VLAN
vnet — the VNET assigned to the VLAN
incoming and outgoing bytes — in K (Kilobytes), M (Megabytes), or G (Gigabytes
incoming and outgoing packets — number of packets incoming and outgoing
incoming and outgoing dropped bytes — in K (Kilobytes), M (Megabytes), or G
(Gigabytes
incoming and outgoing dropped packets — number of dropped packets incoming
and outgoing

Pluribus Networks
www.pluribusnetworks.com
113
Configuring Rapid Spanning Tree Protocol (RSTP)
Rapid Spanning Tree Protocol (RSTP), a standard inter-switch protocol, ensures an ad hoc
network topology loop-free at Layer 2, on a per-VLAN basis. If your network connections
form loops and you disable STP, packets re-circulate between the switches, causing a
degradation of network performance. STP does not allow for Layer 2 multipathing and can
result in sub-optimal utilization of available network links. Therefore, a fabric of switches
does not run RSTP within the boundaries of the fabric. Pluribus Networks recommends the
use of RSTP for ad hoc networks that interoperate in a heterogeneous, multi-vendor switch
environment.
To build a loop-free topology, switches (“bridges”) determine the root bridge and compute
the port roles, root, designated, or blocked. To do this, the bridges use special data frames
called Bridge Protocol Data Units (BPDUs) to exchange information about bridge IDs and root
path costs. BPDUs exchange regularly, typically at two second intervals, and enable switches
to keep track of network topology changes and to start and stop forwarding on ports as
required. Hosts should not send BPDUs to the switch ports and to avoid malfunctioning or
malicious hosts from doing so, the switch can filter or block BPDUs. If you enable BPDU
filtering on a port, BPDUs received on that port drop but other network traffic forwards as
usual. If you enable BPDU blocking on a port, BPDUs received on that port drop and the port
shuts down.
nvOS switches support the Per VLAN Spanning Tree (PVST) variation of STP, and if a PVST
BPDU is detected on a port, PVST is used on that port. This enhances third party switch
support.
Rapid Spanning Tree Protocol supports modifying an RSTP port and configuring the port as an
edge port.

Informational Note: RSTP is enabled on the switch by default.

Before you begin, view the status of STP on the switch by using the following command:

CLI network-admin@Leaf1 > stp-show

switch: tac-1
enable: yes
stp-mode: rstp
bpdus-bridge-ports: yes
bridge-id: 3a:7f:b1:43:8a:0f
bridge-priority: 32768
hello-time: 2
forwarding-delay: 15
max-age: 20
cluster-mode: master

1. To disable STP, use the following command:

CLI network-admin@Leaf1 > stp-modify disable

114
Pluribus Networks
www.pluribusnetworks.com
2. To display the STP state, use the following command:

CLI network-admin@Leaf1 > stp-state-show

switch: Leaf01
vlan: 1
ports: none
instance-id: 1
name: stg-default
bridge-id: 66:0e:94:65:e1:ef
bridge-priority: 8193
root-id: 64:0e:94:c0:06:4b
root-priority: 4097
root-port: 128
hello-time: 2
forwarding-delay: 15
max-age: 20
disabled: none
learning: none
forwarding: 25-28,128-129
discarding: none
edge: 25-28
designated: 25-28,129
alternate: none
backup: none

To display information about STP on ports, use the stp-port-show command:

CLI network-admin@Leaf1 > stp-port-show

switch port block filter edge bpdu-guard root-guard priority cost

-------- ---- ----- ------ ---- ---------- ---------- -------- ----
draco01 1 off off no no no 128 500
draco01 2 off off no no no 128 2000
draco01 3 off off no no no 128 2000
draco01 4 off off no no no 128 2000
draco01 5 off off no no no 128 500
draco01 6 off off no no no 128 500
draco01 7 off off no no no 128 2000
draco01 8 off off no no no 128 2000
draco01 9 off off no no no 128 2000
draco01 10 off off no no no 128 500

3. To filter BPDUs on port 17, use the following command:

CLI network-admin@Leaf1 > stp-port-modify port 17 filter

4. To block BPDUs on port 17 and shut down the port if BPDUs are received on the port, use
the following command:

CLI network-admin@Leaf1 > stp-port-modify port 17 block

5. To stop blocking BPDUs on port 17, use the following command:

CLI network-admin@Leaf1 > stp-port-modify port 17 no-block

Pluribus Networks
www.pluribusnetworks.com
115
6. Disable STP on a port or a group of ports. If the host devices connect to the switch ports
and not downstream switches, then disable STP and the enabled port becomes much faster
when the switch restarts.
7. To enable RSTP on port 35, use the following command:

CLI network-admin@Leaf1 > stp-port-modify port 35 edge

8. To enable STP, use the following command:

CLI network-admin@Leaf1 > stp-modify enable

Managing STP BPDU After Disabling LLDP

Netvisor ONE optimizes STP BPDUs by not sending BPDUs on any ports but not switch ports.
Netvisor ONE uses this as the default setting for the parameter, bpdus-bridge-port. eBy not
configuring LLDP, Netvisor One does not detect host ports and does not send BPDU packets.
As a result, both ports remain in a Forwarding state.
When you add the parameter, bpdus-all-ports, to allow sending BPDUs on ports even if
Netvisor does not detect ports, unless you configure the port as an edge port. On a switch
with a port connected to itself with this configuration one of the ports goes into a Discarding
state.

Fast Failover for STP and Cluster

Previously, cluster STP operation did not support fast failover because Netvisor does not
share STP state machine state between the two nodes. As a result, when the master fails,
the slave recomputes the STP state from scratch. When the cluster comes online, the cluster
recomputes the STP state from scratch. This causes topology changes which cause traffic loss
until the STP converges. By default, Netvisor supports fast failure.
Netvisor supports show commands for this feature:

stp-state-show

stp-port-state-show

116
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@Leaf1 > stp-state-show

switch: Leaf-1
vlan: 1
ports: none
instance-id: 1
name: stg-default
bridge-id: 66:0e:94:d5:b0:cc
bridge-priority: 32769
root-id: 66:0e:94:35:c2:ce
root-priority: 32769
root-port: 128
hello-time: 2
forwarding-delay: 15
max-age: 20
disabled: none
learning: none
forwarding: none
discarding: none
edge: none
designated: none
alternate: none
backup: none

STP parameters such as bridge-priority, port cost values configured before upgrading to
Netvisor OS 2.4.0 are set to default values after upgrade to Netvisor OS 2.4.0. You must
reconfigure STP after upgrading the software.

Active-Active VLAG Forwarding with Loopback Recirculation

The loopback recirculation for active-active VLAG implements a dual next-hop cluster with
internal recirculation for routed packets entering the switch from cluster links. This requires
provisioning a set of physical ports in loopback mode without requiring external cabling. In
particular, the Pluribus E28-Q model offers up to unexposed 12x10GE ports on the front
panel and can be configured in loopback mode for a forwarding capacity of 120 Gbps.
The loopback function prevents packets from dropping when Neighbor A sends traffic to a
cluster switch without ownership of the next-hop.
For example, Netvisor sends packets with destination 00:00:00:10:10:01 (next-hop
10.10.10.1) to cluster switch 2.
And Netvisor sends packets with destination 00:00:00:10:10:02 (next-hop 10.10.10.2) to
cluster switch 1.
Netvisor installs a vFlow rule on both cluster switches and redirects traffic ingressing from
the cluster link with the destination MAC address as the local next-hop to the internal
loopback port. When Netvisor re-circulates the packet, it routes normally and not subject to a
port egress filtering rule. When configured, 50% of the traffic traverses the cluster links.

Pluribus Networks
www.pluribusnetworks.com
117
Configuring Active-Active VLAG Forwarding with Loopback Recirculation
1. 1) For each vrouterto redirect traffic to the loopback - first find the MAC address of the
interface:

CLI network-admin@switch > vrouter-interface-show vrouter-name UP1

vrouter-name UP1
nic: eth11.200
ip: 200.200.200.1/24
assignment: static
mac: 66:0e:94:10:29:e1
vlan: 200
vxlan: 0
if: data
vm-nic-type: data
exclusive: no
nic-config: enable
nic-state: up

2. Configure a loopback port or trunk:

Single Port:

CLI network-admin@switch > port-config-modify port 89 loopback

Trunk:

CLI network-admin@switch > port-config-modify port 89-100 loopback

CLI network-admin@switch > trunk-create name loopB_trunkUP ports 89-100

loopback

3. Then create the vFlow:

Single Port:

CLI network-admin@switch > vflow-create name UP1_vflow scope local in-port 129
dst-mac 66:0e:94:10:29:e1 action set-dmac-to-port action-value 89
action-set-mac-value 66:0e:94:10:29:e1

Trunk:

CLI network-admin@switch > vflow-create name UP1_vflow scope local in-port 129
dst-mac 66:0e:94:10:29:e1 action set-dmac-to-port action-value 130
action-set-mac-value 66:0e:94:10:29:e1

For the action-value 130, this is the trunk-id assigned in Step 2 when the trunk is
created.
For a cluster pair, you must configure each spine using the above configuration steps.

118
Pluribus Networks
www.pluribusnetworks.com
Multiple Spanning Tree Protocol (MSTP)
Multiple Spanning Tree Protocol as defined in IEEE802.1s or IEEE802.1Q-2005 provides the
ability to manage multiple VLANs from a single Multiple Spanning Tree (MST) instance. MST
allows the formation of MST regions that can run multiple MST instances (MSTIs). Multiple
regions and other STP bridges interconnect using one single common spanning tree (CST).
MSTP regions define a collection of switches with the same VLANs on all of the switches. Each
MST region must a root bridge. The root bridge may not reside outside of the region.Netvisor
supports a single MST for a single region. This enables multiple MST instances within a single
region.
The following commands support the configuration of MST instances on a local switch:

CLI network-admin@Leaf1 > mst-config-create

instance-id id Specify the ID as a number between 0

and 63 for MST configuration.
vlans vlan-list Specify the list of VLANs associated with
the MST configuration
bridge-priority bridge-priority-number Specify the bridge priority number for
MST.

Additional commands for MST include the following:

 mst-config-delete

 mst-config-modify

 mst-config-show

To configure MST, use the following commands:

CLI network-admin@Leaf1 > mst-config-create instance-id 1-63 vlans vlan-list

bridge-priority 4096

Netvisor defines the bridge priority as a value from 0 to 65536, with a default value of 0. The
value increments by 4096 each time. For example, the values can be 0, 4096, 8192, up to
65536.

Pluribus Networks
www.pluribusnetworks.com
119
About Port Hairpinning
Port hairpinning allows Layer 2 bridged traffic to exit out of the same switch-port where the
packet arrived. The feature supports hosting containers with Single Root I/O Virtualization
(SR-IOV) network interfaces and classifies traffic sent to the applications.
This feature also allows the first-hop switch to enforce policies and security rules in
hardware, through vflows, and may be used where a Netvisor OS-enabled switch
micro-segments traffic, such as whitelists
You can use this feature when modifying a port configuration and when creating or modifying
a trunk configuration with link aggregation.

Informational Note: If you configure this feature on a port not connected to a

server, it may cause network issues.

The following types of traffic to bridge back:

Layer 2 Unicast traffic
Layer 2 Broadcast, Unknown Unicast, Multicast (BUM) traffic
CPU originated packets
To enable this feature, use the following command:

CLI network-admin@Leaf1 > port-config-modify port port-list reflect

To disable :

CLI network-admin@Leaf1 > port-config-modify port port-list no-reflect

Command Options
Configure the port with the following options for the port-config-modify command:

CLI network-admin@Leaf1 > port-config-modify

port-config-modify modifies a port configuration

reflect|noreflect enables or disables physical port reflection

CLI network-admin@switch > port-config-show

port-config-show displays information about port

configurations
reflect|noreflect indicates if physical port reflection is
enabled or not

Configure the trunk with the following hairpinning options for the trunk-create,
trunk-modify, and trunk-show commands:

120
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@Leaf1 > trunk-create

trunk-create create a trunk configuration for link

aggregation
reflect|noreflect enables or disables physical port reflection

CLI network-admin@Leaf1 > trunk-modify

trunk-modify modify a trunk configuration for link

aggregation
reflect|noreflect indicates if physical port reflection is
enabled or not

CLI network-admin@Leaf1 > trunk-show

trunk-show display trunk configuration

reflect|noreflect indicates if physical port reflection is
enabled or not

Topic Feedback
Was this topic useful to you? Please provide feedback to improve the content.

Pluribus Networks
www.pluribusnetworks.com
121
Configuring VXLANs
Configuring VXLANs
Configuring VXLANs and Tunnels
Configuring a VXLAN with Netvisor ONE
VXLAN Head End Replication Counters
Creating Tunnels
Egress ECMP Load Distribution for VXLAN Traffic from the VTEP Switch
VXLAN Routing In and Out of Tunnels
VXLAN Port Termination
Virtual Link Extension with Cluster Configurations
 Example Topology for Virtual Link Extension and Cluster Configuration
Virtual Link Static Bidirectional Association
PortReplication for Virtual Link Extensions
Support for Configuring Keep-Alive Time for Virtual Link Extension (VLE)
Support for Virtual Link Extension (VLE) Analytics

About VXLANs
Netvisor provides traditional network segmentation using Virtual Local Area Networks
(VLANs) and standardized under the IEEE 802.1Q group. VLANs provide logical segmentation
of the network at Layer 2 or broadcast domains. Due to less than optimal use of available
network links with VLANs, rigid requirements exist for the placement of devices in the
network and the scalability limited to a maximum of 4096 VLANs. Using VLANs becomes a
limiting factor when building large multi-tenant data centers.
Virtual Extensible LANs (VXLAN) design provides the same Ethernet Layer 2 network services
as VLANs but with greater extensibility and flexibility. When compared to VLANs, VXLANs
offer the following benefits:
Flexible placement of multi-tenant segments through the data center, because the
feature provides a solution to extend Layer 2 segments over the underlying shared
network infrastructure and tenant workload load balances across physical pods in the
data center.
Increased scalability to address more Layer 2 segments as VXLANs use a 24-bit
segment ID known as the VXLAN Identifier (VNID) which enables up to 16 million
segments to coexist in the same administrative domain.
Improved utilization of available network paths in the underlying infrastructure since
VXLAN packets transfer through the underlying network based on the Layer3 header.
This takes advantage of Layer 3 routing, equal-cost multipath (ECMP) routing, and
link aggregation protocols to use all available paths.
As a Layer 2 overlay scheme over a Layer 3 network, VXLANs uses MAC Address-in-User
Datagram Protocol (MAC-in-UDP) encapsulation to provide a means to extend Layer 2
segments across the data center network. VXLAN supports a flexible, large-scale
multi-tenant environment over a shared common physical infrastructure. VXLANs use IP plus
UDP as a transport protocol over the physical data center network.

Pluribus Networks
www.pluribusnetworks.com
123
Netvisor supports VXLANs on non-redundant and redundant spine-leaf topology. VXLAN
configuration at high level involves 5 major steps in addition to VLAN, trunk, vLAG, and
vRouter configuration as needed.

Configuring VXLANs
1. Configure underlay vRouter interfaces:
a. Add vRouter and add vRouter interfaces for each VTEP.

CLI network-admin@switch > vrouter-create name <vr-name> vnet <vnet-name>

router-type hardware hw-vrrp-id <id>

CLI network-admin@switch > vrouter-interface-add vrouter-name <vr-name> ip

<network/netmask> vlan <y> if data mtu <mtu>

b. Configure VIP for redundant VTEPs.

CLI network-admin@switch > vrouter-interface-add vrouter-name <vr-name> ip

<network/netmask> vlan <y> if data vrrp-id <id> vrrp-primary <ethz.y> mtu <mtu>

2. Optionally, add ports to vxlan-loopback-trunk:

CLI network-admin@switch > trunk-modify name vxlan-loopback-trunk ports <list

of ports>

3. Configure tunnels:
On non-redundant switches, configure a tunnel with scope local and on redundant switch,
create a tunnel using scope cluster.

CLI network-admin@switch > tunnel-create name <tunnel-name> local-ip <ip1>

remote-ip <ip2> scope local vrouter-name <vr-name>

CLI network-admin@switch > tunnel-create name <tunnel-name> local-ip <ip1>

remote-ip <ip2> scope cluster vrouter-name <vr-name> peer-vrouter-name
<peer-vr-name>

4. Configure overlay:
Create mapping between VXLAN VNID and VLANs on respective switches

CLI network-admin@switch > vlan-create id <vlan-id> scope <scope> vxlan <vnid>

5. Add VNIDs to tunnels:

The mapping allows configured VLAN VNIDs to be carried over VXLAN tunnel.

CLI network-admin@switch > tunnel-vxlan-add name <tunnel-name> vxlan <vnid>

In order to carry Layer 2 broadcast, unicast, and multicast (BUM) traffic over VXLAN tunnels
on Netvisor OS switches, you must configure one physical port to recirculate the packet and
do head-end replication. Based on the hardware architecture of the switch, configure a front
panel port. Depending on the amount of BUM traffic, you can use either a 10G port or a 40G
port.
For monitoring VXLAN specific states and statistics, use the following commands:

124
Pluribus Networks
www.pluribusnetworks.com
 vlan-show — displays the VXLAN ID associated with the VLAN ID.

 tunnel-show — displays the configured tunnel and the state

 trunk-show — displays the port used for BUM traffic recirculation

 ports-stats-show — displays statistics for each port

 tunnel-stats-show — displays statistics for each tunnel

 vxlan-stats-show — displays statistics for each VXLAN ID

Configuring VXLANs and Tunnels

Informational Note: VXLAN encapsulated packets are recirculated in

using hardware features and not software.

In today’s virtualized environments, networks place an increasing demand on MAC address

tables of switches that connect to servers. Instead of learning one MAC address per server
link, the switch now learns the MAC addresses of individual VMs, and if the MAC address
table overflows, the switch may stop learning new MAC addresses until idle entries age out.
Virtual Extensible LAN (VXLAN), essentially a Layer 2 overlay scheme over a Layer 3
network, and the term for each overlay names a VXLAN segment. Only VMs within the same
VXLAN segment communicate with each other. Netvisor identifies each VXLAN segment by a
24 bit segment ID called the VXLAN Network Identifier (VNI).
VXLANs increase the scalability of a network up to 16 million logical networks and is used to
contain broadcast, multicast, and unknown unicast traffic.
Because of this encapsulation, VXLAN could also be called a tunneling scheme to overlay
Layer 2 networks over top of Layer 3 networks. However, the tunnel does not terminate on
the switch, and the switch sits in the middle of the tunnel and identifies packets as L3
tunneled packets. Netvisor then forwards the packets using L2 or L3 forwarding.
Pluribus Networks supports two scenarios for VXLAN:
1. The tunnel does not terminate on the switch and does not support VTEPs. Though the
switch does not participate in the creation of a tunnel, Netvisor ONE still performs the
following tasks.
a. Analytics Collection — All TCP control packets are captured as well as ARP packets
traversing the tunnel. These packets are used to build connection statistics and provide
visibility as to which VXLAN nodes are on specific ports.
b. ARP Optimization — An ARP request is captured and if a Layer 2 entry exists in the
switch Layer 2 table, Netvisor OS sends a response back to the sender of the ARP request
over the tunnel. Otherwise, the ARP request is re-injected into the tunnel without any
modification to continue crossing the tunnel.
2. The tunnels terminate at a switch and the switch performs the role of a VTEP. In this sce-
nario, the responsible switch encapsulates packets arriving from non-VXLAN nodes on a
Layer 2 network and transmis them over the tunnel. Similarly, the switch decapsulates the

Pluribus Networks
www.pluribusnetworks.com
125
 packets arriving through the tunnel and forwards the inner packet over the L2 network. The
switch also collects statistics and optimizes ARP requests as in the first scenario.

Informational Note: Netvisor supports a one to one mapping of VXLAN to VLAN.

Netvisor does not support multicast traffic. VXLAN uses the scope local on all
switches, and must be in the same subnet.

Configuring a VXLAN with Netvisor ONE

The first scenario requires no additional configuration. The second scenario requires the
following steps, in order:
1. Create a hardware vRouter.
2. Add interfaces to the vRouter, one per tunnel. The tunnel endpoint IP address must be
routable.
3. Create one or more tunnels.
4. Create the VXLAN with the VNI, and add the tunnels created in the previous steps.
To create a VXLAN, vx-seg1, with the VNID 25, scope fabric, and turn off deep inspection,
use the following syntax:

CLI network-admin@switch > vxlan-create name vx-seg1 vnid 25 scope fabric

deep-inspection no

To delete a VXLAN, use the vxlan-delete command.

To display information about VXLANs, use the vxlan-show command.
If you added a port to the VXLAN configuration, use the vxlan-port-remove command.

Configuration Example
The following example assumes that one VTEP resides on the generic switch and the other
VTEP resides on a Pluribus Networks switch. Also, the nodes connect on a L3 IP network, and
the tunnel forms between the generic switch and the Pluribus Networks switch.
Figure 5:VTEP Generic Switch-VTEP Pluribus Networks Switch

The example also includes VLAN 10 and port 47 on Host2 as well as the VNET fab-global.

126
Pluribus Networks
www.pluribusnetworks.com
1. Create the vRouter using the vrouter-create command:

CLI (server-switch)>vrouter-create name vx-vrouter vnet fab-global router-type

hardware

2. Add the vRouter interface:

CLI (server-switch)>vrouter-interface-add vrouter-name vx-vrouter ip 192.168.0.1

netmask 255.255.255.0 vlan 10

3. Create the tunnel:

CLI (server-switch)>tunnel-create name vx-tunnel scope local local-ip 192.168.0.1

remote-ip 192.168.5.1 next-hop 192.168.0.2 next-hop-mac 00:01:02:03:04:05 router-if
vx-router.eth0

4. Create the VXLAN:

CLI (server-switch)>vxlan-create vnid 14593470 scope local name vxlan1 vlan 10

If VLAN 10 does not exist, then the vxlan-create command creates the VLAN on the
switch, but you may need to add local ports to the VLAN.
5. Add port 47 to the VXLAN:

CLI (server-switch)>vxlan-port-add vxlan-name vxlan1 ports 47

The configuration associates all packets from port 47 on VLAN 10 with the VXLAN ID,
14593470.
6. Add the tunnel to the VXLAN:

CLI (server-switch)>vxlan-tunnel-add vxlan-name vxlan1 tunnel-name vx-tunnel

To display the configuration, use the vxlan-show command.
You cannot configure different VLANs for the tunnel and the local hosts, and you cannot
associate different VLANs on different ports for the same VXLAN.

VXLAN Head End Replication Counters

This feature introduces support for collecting statistics (stats) for head end replication (HER)
packets to tunnels, on VXLAN VLANs and enhances the tunnel stats output to display counter
for head replicated packets. When Broadcast/UnknownUnicast/UnknownMcast traffic floods
on VXLAN VLANs, the traffic head end replicates to tunnels part of the VXLAN. Currently,
Netvisor OS uses vxlan-loopback-trunk port stats counters for HER packets statistics.
Although all tunnels use the same vxlan-loopback-trunk ports, this provides a very
cumulative view of the system. This feature provides additional counters for HER packets per
tunnel.
Netvisor attaches statistical objects and flexible counters to multicast VPs created for head
end replication for each tunnel. Packet (pkt) and byte counts for HER statistics displays per
tunnel. Two new fields added to tunnel-stats-show output as HER-pkts, HER-bytes and
provides flood packet counters per tunnel. Netvisor updates tunnels local to each node and
therefore no impact to High Availability (HA).

CLI (network-admin@Spine1) > tunnel-stats-show show-diff-interval 1

format all

Pluribus Networks
www.pluribusnetworks.com
127
 ...

Creating Tunnels
You create tunnels to encapsulate protocols on the network. You can create tunnels for
IP-in-IP, VXLAN, and NVGRE network traffic. However, Netvisor supports tunnels for the local
scope only and does not use any discovery mechanism.
IP-in-IP protocol encapsulates an IP header with an outer IP header for tunneling. The outer
IP header source and destination identifies the endpoints of a tunnel. The inner IP header
source and destination identify the original sender and recipient of the datagram.
In addition to the IP header and the VXLAN header, the VTEP also inserts a UDP header.
During ECMP, the switch includes this UDP header to perform the hash function. The VTEP
calculates the source port by performing the hash of the inner Ethernet frame's header.
Netvisor supports the Destination UDP port on the VXLAN port .
The outer IP header contains the Source IP address of the VTEP performing the
encapsulation. The remote VTEP IP address or the IP Multicast group address provides the
destination IP address.
Network Virtualization with Generic Routing Encapsulation (NVGRE) uses GRE to tunnel Layer
2 packets over Layer 3 networks. NVGRE seems similar to VXLAN but it doesn’t rely on IP
multicast for address learning.
To create a tunnel for IP-in-IP traffic, local IP address 192.168.100.35, and the router,
tunnel-network, use the following syntax:

CLI network-admin@switch > tunnel-create scope local name ipinip type ip-in-ip
local-ip 192.168.100.35 router-if vrouter-hw-if eth0.0

To remove a tunnel, use the tunnel-delete command.

To modify a tunnel, use the tunnel-modify command.

128
Pluribus Networks
www.pluribusnetworks.com
Logging Changes to Tunnel States
This feature enables you to log tunnel state changes so you view tunnel state historical data
for debugging purposes. The following state changes are logged in tunnel history:
Creating tunnels
Deleting tunnels
Tunnel hardware state changes including:

Virtual Router ID (VRID) associated with the tunnel vrouter

the router interface associated with the tunnel local-ip
the tunnel next hop add or remove
tunnel next hop egress ports
Equal-Cost Multi-Path routing (ECMP) group updates

Tunnel History Commands

To update tunnel history settings, use the tunnel-history-settings-modify command:

CLI network-admin@switch > tunnel-history-settings-modify enable|disable

disk-space disk-space-number log-file-count 1..20

To display tunnel history settings, use the tunnel-history-settings-show command:

CLI network-admin@switch > tunnel-history-settings-show

To display historical tunnel state history, us the tunnel-history-show command:

CLI network-admin@switch > tunnel-history-show

Egress ECMP Load Distribution for VXLAN Traffic from the VTEP Switch
Equal-cost multi-path routing (ECMP) defies a routing strategy where next-hop packet
forwarding to a single destination can occur over multiple best paths. Tunnel next hops
update based on underlay routes information. RIB/FIB information leverages to program
next hops for a tunnel remote endpoint. If multiple next hops exist for a tunnel remote
endpoint, an ECMP group uses the list of next hops and the tunnel programmed accordingly.

Pluribus Networks
www.pluribusnetworks.com
129
For example, a tunnel leaf1toleaf 2 with a remote IP address 32.4.11.1, Netvisor displays 2
next hops, 192.178.0.6 and 192.178.0.2. Netvisor hashes the traffic going over tunnel
leaf1toleaf 2 using these two next hop links.
CLI (network-admin@lleaf11) > tunnel-show
scope: cluster
name: leaf1toleaf2
type: vxlan
vrouter-name: leafpst1
peer-vrouter-name: leafpst2
local-ip: 22.3.11.1
remote-ip: 32.4.11.1
router-if: eth12.11
next-hop: 192.178.0.6
next-hop-mac: 66:0e:94:8c:d4:0f
nexthop-vlan: 4091
remote-switch: 0
active: yes
state: ok
error:
route-info: 32.4.11.0/24
scope:
name:
type:
vrouter-name:
peer-vrouter-name:
local-ip:
remote-ip:
router-if:
next-hop: 192.178.0.2
next-hop-mac: 66:0e:94:5b:90:2b
nexthop-vlan:
remote-switch: 4092
active: 0
state: ok
error:
route-info: 32.4.11.0/24
scope: cluster
name: leaf1toleaf2-2nd
type: vxlan
vrouter-name: leafpst1
peer-vrouter-name: leafpst2
local-ip: 22.3.12.1
remote-ip: 32.4.12.1
router-if: eth9.12
next-hop: 192.178.0.6
next-hop-mac: 66:0e:94:8c:d4:0f
nexthop-vlan: 4091
remote-switch: 0
active: yes
state: ok
error:
route-info: 32.4.11.0/24
scope:
name:
type:
vrouter-name:

130
Pluribus Networks
www.pluribusnetworks.com
peer-vrouter-name:
local-ip:
remote-ip:
router-if:
next-hop: 192.178.0.2
next-hop-mac: 66:0e:94:5b:90:2b
nexthop-vlan:
remote-switch: 4092
active: 0
state: ok
error:
route-info: 32.4.11.0/24

CLI (network-admin@leaf-pst-1) > vrouter-rib-routes-show ip 32.4.11.0

vrid ip prelen number-of-nexthops nexthop flags vlan intf_ip
intf_id
---- --------- ------ ------------------ ----------- ---------- ----
----------- -------
0 32.4.11.0 24 2 192.178.0.6 ECMP,in-hw 4091 192.178.0.5
1
0 32.4.11.0 24 2 192.178.0.2 ECMP,in-hw 4092 192.178.0.1 0

CLI (network-admin@leaf-pst-1) > vrouter-rib-routes-show ip 32.4.12.0

vrid ip prelen number-of-nexthops nexthop flags vlan intf_ip
intf_id
---- --------- ------ ------------------ ----------- ---------- ----
----------- -------
0 32.4.12.0 24 2 192.178.0.6 ECMP,in-hw 4091 192.178.0.5
1
0 32.4.12.0 24 2 192.178.0.2 ECMP,in-hw 4092 192.178.0.1
0

VXLAN Routing In and Out of Tunnels

Informational Note: The VXLAN tunnel loopback infrastructure, identified by the

trunk object named "vxlan-loopback-trunk", used for bridging multicast or broadcast
traffic in the extended VLAN and for routing traffic before VXLAN encapsulation or
after VXLAN decapsulation. Non-routed unicast traffic bridges and encapsulated or
decapsulated and bridged without using the VXLAN tunnel loopback.

This feature provides support for centralized routing for VXLAN VLANs. For hosts on different
VXLAN VLANs to communicate with each other, SVIs on VXLAN VLAN configured on one
cluster pair in the fabric. Any VXLAN VLAN packets be routed between two hosts sent to a
centralized overlay vrouter and then VXLAN encapsulated or decapsulated depending on
source or destination host location.

Pluribus Networks
www.pluribusnetworks.com
131
Because the E68-M and E28Q cannot perform VXLAN routing in and out of tunnels in a single
instance, loopback support exists. Netvisor leverages vxlan-loopback-trunk to support
recirculation of the packets. Be sure to add ports to vxlan-loopback-trunk so that VXLAN
routing in and out of tunnels works correctly. After VXLAN decapsulation, if packets route,
the inner DMAC uses either the vRouter MAC address or VRRP MAC address. The packet
needs to recirculate after decapsulation as part of the routing operation. To accomplish this,
Layer 2 entries for route RMAC address or VRRP MAC address on VXLAN VLAN program to
point to vxlan-loopback-trunk ports in hardware. The show output for the command,
l2-table-show, updates with a vxlan-loopback flag to indicate the hardware state.

CLI network-admin@switch > l2-table-show vlan 200

mac: 00:0e:94:b9:ae:b0
vlan: 200
vxlan 10000
ip: 2.2.2.2
ports: 69
state: active,static,vxlan-loopback,router
hostname: Spine1
peer-intf: host-1
peer-state:
peer-owner-state:
status:
migrate:
mac: 00:0e:94:b9:ae:b0
vlan: 200
vxlan 10000
ip: 2.2.2.2
ports: 69
state: active,static,vxlan-loopback,router
hostname: Spine1
peer-intf: host-1
peer-state: active,vrrp,vxlan-loopback active,vrrp
peer-owner-state:
status:
migrate:

CLI network-admin@switch > l2-table-show vlan 100

mac: 00:0e:94:b9:ae:b0
vlan: 100
vxlan 20000
ip: 1.1.1.1
ports: 69
state: active,static,vxlan-loopback,router
hostname: Spine1
status:
migrate:

132
Pluribus Networks
www.pluribusnetworks.com
Also for Layer3 entries behind VXLAN tunnels, routing and encapsulation operations requires
two passes . To obtain the Layer 3 entry, the hardware points to vxlan-loopback-trunk. The
show output of the l3-table-show displays the hardware state with a vxlan-loopback flag.

CLI (network-admin@Spine1) > l3-table-show ip 2.2.2.3 format all

mac: 00:12:c0:88:07:75
ip: 2.2.2.3
vlan: 200
public-vlan: 200
vxlan: 10000
rt-if: eth5.200
state: active,vxlan-loopback
egress-id: 100030
create-time: 16:46:20
last-seen: 17:25:09
hit: 22
tunnel: Spine1_Spine4

VXLAN Port Termination

When configuring overlay VLANs on a port, Netvisor does not allow VXLAN termination on a
port even if the VXLAN termination criteria matches. Netvisor enforces the configuration for
ports facing bare metal servers or single root input/output virtualization (SRIOV) hosts. With
underlay VLANs configured on a port, Netvisor OS allows VXLAN termination on a port which
may have HWvtep or SWVtep configured for the port.
Prior to Version 2.5.3, when you configured overlay VLANs on a port, VXLAN encapsulated
packets received on a port do not terminate on a VXLAN tunnel. In versions later than 2.5.3,
Netvisor no longer enforces the configuration.
One sample use case has both overlay and underlay VLANs on a port. In this case, Netvisor
disables the VXLAN termination on the port since the port has overlay VLAN and therefore,
any VXLAN encapsulated traffic received on this port no longer terminates even if the
destination employs a local HWvtep.
To support this sample use case, Netvisor OS provides a port-config-modify parameter to
enable or disable VXLAN termination on the port.

CLI network-admin@switch > CLI (network-admin@Spine1)>port-config-modify port

35 vxlan-termination

Enables tunnel termination of VXLAN encapsulated packets received on the port when VXLAN
tunnel termination criteria is met.

CLI network-admin@switch > CLI (network-admin@Spine1)>port-config-modify port

35 no-vxlan-termination

Disable vxlan-termination on a port when VXLAN encapsulated packets are received on port.
This enforces the security to prevent any malicious host from generating VXLAN
encapsulated packets that would otherwise be subject to VXLAN tunnel termination.
Managed ports added to a VNET with vlan-type private, relies on VXLAN functionality and
therefore always carry overlay VLANs only. Therefore when you configure a port as a
managed port, VXLAN termination is disabled by default.

Pluribus Networks
www.pluribusnetworks.com
133
Default Settings
1. VNETs with vlan-type private relies on VXLAN functionality. The vlan-type private
are VXLAN overlay VLANs. Hence when a port is configured to be a managed port with
vlan-type private, vxlan-termination is disabled by default.
2. Shared/underlay ports have vxlan-termination on by default and can use the port-con-
fig-modify command to enable or disable vxlan-termination as is deemed to enforce port
level security.
VXLAN termination is disabled on VXLAN loopback trunk ports.

Virtual Link Extension with Cluster Configurations

Limitations for this release are as follows:
Only supported on the Accton and Pluribus Freedom platforms.
Only create VLE tunnels between a port or trunk and a VXLAN tunnel.
VTEPs configured for VLE used as endpoints for carrying multiple VLE VLANs.
This feature only allows creation of VTEPs on switches in cluster mode and does not
support any active or backup failover between VTEPs.
VTEPs cannot use the VRRP virtual IP address as the local IP address.
Cluster ports always allow VXLAN termination.
LACP packets on VLE VLANs are not sent to CPU.
Netvisor supports Virtual Link Extension (VLE) on switches part of a cluster configuration iby
creating a dedicated VXLAN tunnel end points (VTEPs). Configure the VTEPs using one of the
physical or primary IP addresses on the switch. The physical or primary IP address can be
from a new Layer 3 interface dedicated for VLE configuration or from reusing the existing
physical or primary IP addresses used to build the cluster VIP and used for VXLAN tunnel
redundancy in a cluster environment. These dedicated tunnels and VTEPs are stateless with
no dependency on each other.

134
Pluribus Networks
www.pluribusnetworks.com
Figure 3: Example Topology for Virtual Link Extension and Cluster Configuration

In the example topology, Host1 connects to both cluster nodes, PN-SW1 and PN-SW2. No
VLAG on PN-SW1 and PN-SW2 connected to Host1. Host2 has 2 links connected to
PN-SW3, a standalone switch. PN-SW3 does not configure trunking on the ports connected
to Host2. Configure both Host1 and Host2 with LACP on links connecting to switches to
High Availability (HA) functionality.
Create a new VLAN Layer 3 interface on the local vRouter used as a VTEP source IP. Create
the VLAN as local only and dedicated for this usage.
In this example configuration, you must configure one virtual link extension for each point to
point connectivity.
1. Configure VLE VLANs for each virtual link extension and add the ports:
On PN-SW1

CLI network-admin@switch > vlan-create id 400 vxlan 400 vxlan-mode transparent

scope local

CLI network-admin@switch > vlan-port-add vlan-id 400 ports 11

On PN-SW2

CLI network-admin@switch > vlan-create id 401 vxlan 401 vxlan-mode transparent

scope local

CLI network-admin@switch > vlan-port-add vlan-id 401 ports 11

Pluribus Networks
www.pluribusnetworks.com
135
On PN-SW3

CLI network-admin@switch > vlan-create id 400 vxlan 400 vxlan-mode transparent

scope local

CLI network-admin@switch > vlan-create id 401 vxlan 401 vxlan-mode transparent

scope local

CLI network-admin@switch > vlan-port-add vlan-id 400 ports 11

CLI network-admin@switch > vlan-port-add vlan-id 401 ports 12

Create VXLAN tunnels using the Primary IP address. Note that 10.10.10.1 and
10.10.10.2 are primary IP addresses on PN-SW1 and PN-SW2 and 20.20.20.3 is
primary IP on PN-SW3.
On PN-SW1

CLI network-admin@switch > tunnel-create scope local name VTEP1 vrouter-name

vr-s1 local-ip 10.10.10.1 remote-ip 20.20.20.3

On PN-SW2

CLI network-admin@switch > tunnel-create scope local name VTEP2 vrouter-name

vr-s2 local-ip 10.10.10.2 remote-ip 20.20.20.3

On PN-SW3

CLI network-admin@switch > tunnel-create scope local name VTEP3 vrouter-name

vr-s3 local-ip 20.20.20.3 remote-ip 10.10.10.1

CLI network-admin@switch > tunnel-create scope local name VTEP4 vrouter-name

vr-s3 local-ip 20.20.20.3 remote-ip 10.10.10.2

2. Add VLE VLANs and VXLANs to VXLAN tunnels.

On PN-SW1

CLI network-admin@switch > tunnel-vxlan-add name VTEP1 vxlan 400

On PN-SW2

CLI network-admin@switch > tunnel-vxlan-add name VTEP2 vxlan 401

On PN-SW3

CLI network-admin@switch > tunnel-vxlan-add name VTEP3 vxlan 400

CLI network-admin@switch > tunnel-vxlan-add name VTEP4 vxlan 401

VLE1 is created on PN-SW1 with port 11 and VTEP1, 10.10.10.3 to 50.50.50.5, over
VXLAN 400.
VLE1 is created on PN-SW1 with port 11 and VTEP3 tunnel, 50.50.50.5 to
10.10.10.3, over VXLAN 400.
Port 11 and VTEP2, 10.10.10.4 to 51.51.51.5, over VXLAN 401.

136
Pluribus Networks
www.pluribusnetworks.com
 VLE2is created on PN-SW2 with pVLE2 is created on PN-SW3 with port 12 and
VTEP4, 51.51.51.5 to 10.10.10.4, over VXLAN 401.

Virtual Link Static Bidirectional Association

Virtual Link Extension (VLE) static bidirectional association is between a physical port or
trunk and a VXLAN tunnel for a selected VNID. Every virtual link extension is defined using a
VLAN or VXLAN VNID and adding physical ports or trunk and tunnels to this VLAN or VXLAN
VNID. One endpoint of virtual link extension must be a physical port or trunk and other end
point must be a virtual tunnel end point (VTEP). A VTEP can carry multiple VLE VLANs or
VXLANs.
Netvisor creates VLE VLANs using a special VXLAN mode called “transparent” which
prevents flooding and learning on VLE VLANs in the switch. Netvisor sends packets
directly between a physical port or trunk and the VLE part of VLE regardless of traffic
types. There is no BPDU processing for VLE VLAN.
Enhancements to VLE VLANs now provide the capability of adding and removing ports,
as well as assigning or modifying trunk ports. Note that when you delete a trunk
carrying a VLE VLAN, Netvisor removes the VLE VLAN from the trunk members to
avoid any hardware inconsistencies.

Virtual Link Static Bidirectional Command

Use the vlan-create vxlan-mode standard|transparent command to configure this
feature.

Port Replication for Virtual Link Extensions

This feature provides a mechanism for link state tracking between two ports of two switches
in the same fabric for Virtual Link Extension (VLE).
When you configure a VLE between two physical ports of two switches, the VLE remains up
as long as both physical ports are in the link up state. When you configure VLE tracking on a
trunk port, the VLE stays up as long as at least one port in the trunk remains up and the
remote port remains up. When the last trunk member goes link down, Netvisor brings down
the VLE. Note that when you configure VLE tracking on a trunk port, you cannot configure
tracking on individual trunk members.
VLE tracking helps achieve VLE high availability on Netvisor OS nodes and avoids the need on
the client side to run LACP for link up/down detection.
Use these commands to create, modify, delete, and show link state tracking:
 vle-create

 vle-modify

 vle-delete

 vle-show

To create virtual link extension tracking, use the vle-create command. You can execute this
command from any fabric node to create a virtual link extension between any two switches in
the fabric.

Pluribus Networks
www.pluribusnetworks.com
137
CLI network-admin@switch > vle-create name name-string node1 fabric-node name
node-2 fabric-node name node-1-port node-1-port-number node-2-port
node-2-port-number [tracking|no-tracking]

vle-create Create virtual link extension tracking

name name-string Specify the VLE name.
node-1 fabric-node name Specify VLE node 1 name.
node-2 fabric-node name Specify VLE node 2 name.
node-1-port node-1-port-number Specify VLE node-1 port.
node-2-port node-2-port-number Specify VLE node-2 port.
[tracking|no tracking] Enable or disable tracking between
VLE ports

To enable or disable tracking between existing VLE ports, use the vle-modify command:

CLI network-admin@switch > vle-modify name name-string tracking|no tracking

vle-modify Modify virtual link extension tracking

name name-string Modify the VLE name
tracking|no tracking Enable or disable tracking between
VLE ports

To delete a virtual link extension, use the vle-delete command:

CLI network-admin@switch > vle-delete name name-string

To view a virtual link extension status, use the vle-show command:

CLI network-admin@switch > vle-show

name node-1 node-2 node-1-port node-2-port status tracking

---------- -------- -------- ----------- ----------- ------ --------
Test1 mynode1 mynode2 11 11 up yes

Support for Configuring Keep-Alive Time for Virtual Link Extension (VLE)
As part of VLE tracking, each node hosting VLEs sends fabric fast keepalives every one
second to every other node that hosts the endpoint of the local VLEs. A local node times out
a remote node if a keepalive is not received from the remote node in 3 seconds which is the
default VLE tracking timeout.
If a remote node times out, Netvisor brings down the local VLE ports, terminating on the
remote node. In some deployments, Netvisor determines a three second times out.
Note that the periodic fast keepalive send frequency remains one second. Only the timeout
value adjusts to the configured value.
(CLI network-admin@Spine1)>system-settings-modify vle-tracking-timeout seconds

Configure a value from 3 to 30 seconds.

(CLI network-admin@Spine1)>fabric-node-show keepalive-timeout high resolution
time: #ns

Configure the fabric keepalive timeout in nanoseconds.

138
Pluribus Networks
www.pluribusnetworks.com
Support for Virtual Link Extension (VLE) Analytics
Currently, Netvisor OS does not copy VLE traffic control frames to the CPU on the switch.
Netvisor does not remove the inner tag, if present. Netvisor achieves this by installing a
system vFlow, Virtual-Link-Extend, with highest priority 15 with no action specified so that
Netvisor does not terminate LLDP or other control frames and send to CPU.
To support VLE analytics, Netvisor installs a few additional system vFlows with the same
priority as the existing Virtual-Link-Extend vFlow to copy TCP-SYN/FIN/RST packets to CPU.
This ensures that any VLE-SYN/FIN/RST packets target System-VLE-x flows and not
Virtual-Link-Extend flow.
(CLI network-admin@Spine1)>vflow-show format
name,scope,type,proto,tcp-flags,precedence,action,enable

name scope type proto tcp-flags precedence action enable

---------------------- ----- ------ ----- --------- ---------- -----------
------
System-VLE-S local system tcp syn 15 copy-to-cpu enable
System-VLE-F local system tcp fin 15 copy-to-cpu enable
System-VLE-R local system tcp rst 15 copy-to-cpu enable
Virtual-Link-Extend local system 15 none enable

(CLI network-admin@Spine1)>connection-show
vnet vlan vxlan src-ip dst-ip dst-port cur-state syn-resends
syn-ack-resends
---- ---- ----- ---------- ---------- -------- --------- -----------
100 100 20.20.20.1 20.20.20.2 http fin 0 0

latency obytes ibytes total-bytes age

------- ------ ------ ----------- --------
74.8us 149 311 460 2h11m21s

Pluribus Networks
www.pluribusnetworks.com
139
Configuring Layer 3 Features
IPv6 Neighbor Discovery Process Support and Optimization
Displaying Hardware Routes History
Configuring MTU Parameters for vRouter Interfaces
Support for IPv4 and IPv6 on a vRouter Interface
IPv6 Support for vRouter Loopback Addresses
Configuring Prefix Lists for BGP and OSPF
Configuring Packet Relay for DHCP Servers
Configuring Hardware Routing for a vRouter
Support for Displaying Quagga Routing and Debug Information for vRouters
Configuring BGP on a vRouter
 Configuring BGP for Two VLANs
Support for BGP SNMP MIBs
Support for AS and AS Prepending and BGP
Bidirectional Forwarding Detection Support for IPv6 BGP Neighbor and IPv6 Static
Routes
Support for Border Gateway Protocol (BGP) Communities
Configuring Open Shortest Path First (OSPF)
Display Default Timers for OSPF Configurations
BFD Support for OSPF Fault Direction
Support for Route Maps for OSPF Routes
Support for OSPF SNMP MIBs
Adding Default Route Information Settings for OSPF Routing
Adding Metric and Metric Type for Route Maps
Configuring Routing Information Protocol (RIP)
Configuring Static Routes
Support for Bidirectional Forwarding Detection on Static Routes
Adding IPv6 Link-Local Addresses for Static Routing
Configuring Multicast Listener Discovery (MLD)
Configuring an IGMP Querier IP Address
Multicast Listener Discovery (MLD) Snooping per VLAN
Creating MLD Static Sources and Static Groups
Configuring Virtual Router Redundancy Protocol
Using an L3 Network to Establish the Netvisor Fabric
Support for Policy-based Routing
Cluster Active-Active Routing Support for IPv6 Addresses
Support for PIM Source Specific Multicast (PIM-SSM) Forwarding
Virtual Routing and Forwarding (VRF) Support

140
Pluribus Networks
www.pluribusnetworks.com
Configuring vRouter Services
A Virtual Router (vRouter) remains an important part of fabric functionality. For example, for
a VLAN to communicate with other VLANs, or networks external to the fabric, the VLAN may
need a vRouter that spans the internal and the external network. vRouter commands can
only be executed at the fabric level by the fabric administrator.

Informational Note: For switches with ONVL, the only available VNET is a global
VNET created when a fabric is created for the first time. Use tab complete in the
CLI to display the VNET and continue the configuration. However, some white box
switches support multiple VNETs. Please refer to the Release Notes for the
supported platforms.

Routing protocols essentially work the same way on virtual routers as physical routers.

IPv6 Hardware Routing

CurrentlyNetvisor supports IPv6 natively only, which means IPv6 cannot co-exist with
IPv4 on the same interface and no encapsulation.
Netvisor currently supports 4000 IPv6 routes in hardware.
For IPv6 routes with a mask greater than 64, Netvisor supports only 64 routes.
Netvisor now supports OSPFv3 and BGP-Multiprotocol with IPv6 unicast and static routing as
Control Plane protocols. You configure vRouter interfaces with globally scoped IPv6
addresses.
CLI (network-admin@Spine-1)vrouter-interface-add vrouter-name vrouter-1 ip
2200:1111:2222:3333::1/64 vlan 100

CLI network-admin@switch > vrouter-interface-add vrouter-name vr0 ip 2200::1/64

vlan 200

CLI network-admin@switch > vrouter-interface-add vrouter-name vr0 ip 2211::1/64

vlan 211

CLI network-admin@switch > vrouter-ospf6-add vrouter-name vr0 nic eth0.200

ospf6-area 0.0.0.0

CLI network-admin@switch > vrouter-ospf6-add vrouter-name vr0 nic eth0.211

ospf6-area 0.0.0.0

BGP-Multiprotocol with IPv6 Unicast Peers Distributing IPv6 Prefixes

CLI (network-admin@Spine1)>vlan-create id 200 scope fabric ports 9
CLI (network-admin@Spine1)>vlan-create id 211 scope fabric
CLI (network-admin@Spine1)>vnet-create name vnet0 scope fabric vlans 200
CLI (network-admin@Spine1)>vrouter-create name vr0 vnet vnet0 router-type
hardware router-id 1.1.1.1 hw-vrrp-id 1
CLI (network-admin@Spine1)>vrouter-interface-add vrouter-name vr0 ip
2200::1/16 vlan 200

Pluribus Networks
www.pluribusnetworks.com
141
CLI (network-admin@Spine1)>vrouter-interface-add vrouter-name vr0 ip
2211::1/24 vlan 211
CLI (network-admin@Spine1)>vrouter-modify name vr0 bgp-as 200
no-bgp-ipv4-unicast
CLI (network-admin@Spine1)>vrouter-modify name vr0 bgp-redistribute
static,connected
CLI (network-admin@Spine1)>vrouter-bgp-add vrouter-name vr0 neighbor 2211::2
remote-as 222 multi-protocol ipv6-unicast
CLI (network-admin@Spine1)>vrouter-bgp-add vrouter-name vr0 neighbor 2211::3
remote-as 233 multi-protocol ipv6-unicast
CLI (network-admin@Spine1)>vrouter-bgp-network-add vrouter-name vr0 network
2200::/64
CLI (network-admin@Spine1)>vrouter-bgp-network-add vrouter-name vr0 network
2211::/64

IPv6 and Prefix Lists

CLI (network-admin@Spine1)>vrouter-prefix-list-add vrouter-name vr0 name
block200 action deny prefix 2200::/16 seq 95
CLI (network-admin@Spine1)>vrouter-bgp-add vrouter-name vr0 neighbor 2211::2
remote-as 222 multi-protocol ipv6-unicast prefix-list-out block200
Static Route
CLI (network-admin@Spine1)>vrouter-static-route-add vrouter-name vr0 network
7777::/64 gateway-ip 2200::9 distance 20

IPv6 Neighbor Discovery Process Support and Optimization

The IPv6 Neighbor Discovery Process (NDP) uses ICMPv6 messages and solicited-node
multicast addresses to determine the link-layer address of a neighbor on the same network
(local link), verify the reachability of a neighbor, and keep track of neighboring routers. NDP
provides the same functionality as ARP in an IPv4 network. NDP has additional features such
as autoconfiguration of IPv6 addresses and duplicate address detection (DAD).
In an IPv6 Layer 3 network, a Netvisor vRouter can be configured as a First Hop Router and
send Router Advertisements to announce the presence, host configuration parameters,
routes, and on-link prefixes. In a Layer 2 network, Netvisor can enable NDP optimization to
prevent flooding of neighbor solicitation messages.
Supported NDP Messages
Router Solicitation (ICMPv6 type 133)
Router Advertisement (ICMPv6 type 134)
Neighbor Solicitation (ICMPv6 type 135)
Neighbor Advertisement (ICMPv6 type 136)
Redirect (ICMPv6 type 137)
Netvisor sends Neighbor Solicitation messages (ICMPv6 Type 135) on the local link by nodes
attempting to discover the link-layer addresses of other nodes on the local link. Netvisor
sends the Neighbor Solicitation message to the solicited-node multicast address. The source
address in the neighbor solicitation message is the IPv6 address of the node sending the
Neighbor Solicitation message. The Neighbor Solicitation message also includes the link-layer
address of the source node.

142
Pluribus Networks
www.pluribusnetworks.com
After receiving a Neighbor Solicitation message, the destination node replies by sending a
Neighbor Advertisement message (ICPMv6 Type 136) on the local link. The source address in
the Neighbor Advertisement message reflects the IPv6 address of the node sending the
Neighbor Advertisement message. The destination address reflects the IPv6 address of the
node sending the Neighbor Solicitation message. The data portion of the Neighbor
Advertisement message includes the link-layer address of the node sending the Neighbor
Advertisement message.
After the source node receives the Neighbor Advertisement, the source node and destination
node communicate.
Netvisor uses Neighbor Solicitation messages to verify the reachability of a neighbor after
identifying the link-layer address of a neighbor. When a node requires verification of the
reachability of a neighbor, the destination address in a Neighbor Solicitation message
includes the unicast address of the neighbor.
Netvisor sends Neighbor Advertisement messages when a change occurs in the link-layer
address of a node on a local link. When there is such a change, the destination address for
the Neighbor Advertisement includes the all-nodes multicast address.
Netvisor periodically sendsRouter Advertisement messages (ICMPv6 Type 134) to each IPv6
configured interface of security appliance. Netvisor also sends the Router Advertisement
messages to the all-nodes multicast address.
Router Advertisement messages typically include the following information:
One or more IPv6 prefix the nodes use on the local link to automatically configure the
IPv6 addresses.
Lifetime information for each prefix included in the advertisement.
Sets of flags that indicate the type of autoconfiguration (stateless or stateful) that can
be completed.
Default router information (whether the router sending the advertisement should be
used as a default router and, if so, the amount of time (in seconds) the router should
be used as a default router).
Additional information for hosts, such as the hop limit and MTU a host should use in
origination packets.
The amount of time between neighbor solicitation message retransmissions on a given
link.
The amount of time a node considers a neighbor reachable.
Netvisor sends Router Advertisements t in response to Router Solicitation messages (ICMPv6
Type 133). Hosts sends Router Solicitation messages at system startup so that the host can
immediately auto-configure without waiting for the next scheduled router advertisement
message. Router Solicitation messages usually sent by hosts at system startup, and the host
does not have a configured unicast address, the source address in Router Solicitation
messages includes the unspecified IPv6 address (0:0:0:0:0:0:0:0). If the host has a
configured unicast address, the source address in the message uses the unicast address of
the interface sending the Router Solicitation message. The destination address in Router
Solicitation messages uses the all-routers multicast address with a scope of the link. When
sending a Router Advertisement in response to a Router Solicitation message, the destination
address in the Router Advertisement message uses the unicast address of the source of the
Router Solicitation message.
Configure the following settings for router advertisement messages:

Pluribus Networks
www.pluribusnetworks.com
143
 The time interval between periodic Router Advertisement messages. Netvisor uses the
default time interval of 200 seconds with a range of 3 to 1800 seconds or 500 to
1800000 milliseconds if you specify milliseconds.
The router lifetime value, which indicates the amount of time IPv6 nodes should
consider the switch to be the default router. Valid values range from 0 to 9000
seconds. Netvisor has a default value of 1800 seconds. Entering 0 indicates that the
switch is not considered a default router on the selected interface.
The IPv6 network prefixes in use on the link. In order for stateless auto-configuration
to work properly, the advertised prefix length in Router Advertisement messages
must always be 64 bits.
Whether or not an interface transmits Router Advertisement messages. By default,
Netvisor automatically sends Router Advertisement messages in response to Router
Solicitation messages. If you suppress the Router Advertisement messages, the
switch appear as a regular IPv6 neighbor on the link and not as an IPv6 router.
Unless otherwise noted, the interface has specific the Router Advertisement message
settings.
To configure NDP, use the vrouter-interface-config-add command:

CLI network-admin@Leaf1 > vrouter-interface-config-add

nd-suppress-ra|no-nd-suppress-ra Control the transmission of IPv6 Router

Advertisements
v6-ra-prefix ip-address IPv6 prefix to include in Router
Advertisement
prefix-netmask netmask IPv6 prefix netmask
autoconf|no-autoconf given prefix can be used for IPv6 autoconf
ra-interval mseconds Time interval between IPv6 router
advertisements
ra-lifetime seconds Time for which router is considered as
default router

144
Pluribus Networks
www.pluribusnetworks.com
Displaying Hardware Routes History
You can display the history of hardware routes in the RIB table. This becomes useful when
troubleshooting hardware routing on the network.
CLI (network-admin@spine1)>vrouter-rib-history-show time 15:30

time caller reason ip prelen nexthop flags vlan

intf_ip intf_id
-------- --------- ------ ----------- ------ ----------- ------------------
---- ----------- ---
15:29:41 router-if add 10.16.111.0 24 10.16.111.1 in-hw,local-subnet
111 10.16.111.1
15:29:43 router-if add 10.16.100.0 24 10.16.100.1 in-hw,local-subnet 100
10.16.100.1 1

You can also modify the settings for collecting the history:
CLI (network-admin@spine1)>vrouter-rib-history-settings-modify

Configuring MTU Parameters for vRouter Interfaces

Support for IPv4 and IPv6 addresses requires configuring the MTU parameter differently for
each type of IP address. By default, Netvisor sets the MTU at 1500 for the VNICs. For IPv4
addresses, the Maximum Transmission Unit (MTU) the range is to 68 to 9216, but the range
for IPv6 addresses is 1280-9216. However, Netvisor displays the MTU range as 68-9216
which is supported by IPv4 addresses. But if you configure an IPv6 interface with an MTU
value outside of the 1280-9216 for IPv6 addresses, Netvisor returns an error.
In the case of interfaces configured with IPv4 and IPv6 addresses (dual-stack), Netvisor
limits the MTU range to 1280-9216. The following special rules apply to dual-stack
interfaces:
If you configure the MTU option for a vRouter interface with an IPv6 address using the
command, vrouter-interface-create, Netvisor performs a range check to ensure
the MTU value is within 1280-9216.
If you configure the MTU option for a vRouter interface using the
vrouter-interface-modify command, Netvisor performs a check for IPv6
addresses, and then performs a range check for 1280-9216.
If you add an IPv6 address with the command, vrouter-interface-ip-add, Netvisor
performs a range check to ensure the MTU of the VNIC interface is within 1280-9216.
If you configure a hardware vRouter, Netvisor propagates the MTU value to the switch
interface, so the MTU values are in the switch ASIC.

Support for IPv4 and IPv6 on a vRouter Interface

You can now add IPv4 and IPv6 addresses to a vRouter interface. If you enable or disable the
interface, both IPv4 and IPv6 are affected. Routing protocols can be enabled or configured
independently using the IP address.
When you add a vRouter interface, you can configure it with two IP addresses, IPv4 and
IPv6:

Pluribus Networks
www.pluribusnetworks.com
145
CLI network-admin@Leaf1 > vrouter-interface-add vrouter-name name-string vlan
vlan-id ip ip-address netmask netmask assigment none|dhcp ip2 ip-address
netmask2 netmask assigment none|dhcp|dchpv6

You can display the IP addresses:

CLI network-admin@Leaf1 > vrouter-interface-show vrouter-name name-string vlan

vlan-id ip ip-address netmask netmask assignment none|dhcp|dchpv6 ip2
ip-address netmask2 netmask assignment none|dhcp|dchpv6

To migrate the interface from a single stack to a dual stack, use the following commands:

CLI network-admin@Leaf1 > vrouter-interface-ip-add

vrouter-name name-string Specify the name of the vRouter.

nic nic-string Specify the virtual NIC assigned to the interface.
ip ip-address Specify the IP address assigned to the interface.
netmask netmask Specify the netmask of the IP address.
Specify any of the following options
vnet vnet-name Specify the VNET assigned to the vRouter.
l2-net l2-net name Specify the Layer 2 interface.

CLI network-admin@Leaf1 > vrouter-interface-ip-remove

vrouter-name name-string Specify the name of the vRouter.

nic nic-string Specify the virtual NIC assigned to the interface.
ip ip-address Specify the IP address assigned to the interface.

CLI network-admin@Leaf1 > vrouter-interface-ip-show

vrouter-name name-string Specify the name of the vRouter.

nic nic-string Specify the virtual NIC assigned to the interface.
ip ip-address Specify the IP address assigned to the interface.
netmask netmask Specify the netmask of the IP address.
Specify any of the following options

146
Pluribus Networks
www.pluribusnetworks.com
 vnet vnet-name Specify the VNET assigned to the vRouter.
l2-net l2-net name Specify the Layer 2 interface.

IPv6 Support for vRouter Loopback Addresses

Netvisor supports IPv4 and IPv6 addresses on the same loopback interface.
When you configure a loopback interface, you can optionally associate a vRouter interface to
the configuration. If you do not associate a vRouter interface, Netvisor selects the first
non-VRRP vRouter interface. Netvisor uses the loopback interface for a vRouter as a host
route in the Forwarding Information Database (FIB) for packets received from any port and
routes them to the correct vRouter.
If no configured vRouter interface exists, Netvisor determines the loopback interface
unreachable on the network. When you add the first vRouter interface, Netvisor determines
the loopback interface reachable.
If you remove the associated vRouter interface, Netvisor selects the next available vRouter
interface.
Netvisor does not support multiple vRouter loopback interfaces for IPv6 addresses.
To add an IPv4 address to a loopback interface for vRouter, vr1, use the following syntax:

CLI network-admin@Leaf1 > vrouter-loopback-interface-add vrouter-name vr1 ip

99.1.1.1

To add an IPv6 vRouter loopback interface to vRouter, vr1, use the following syntax:

CLI network-admin@Leaf1 > vrouter-loopback-interface-add vrouter-name vr1 ip

2999:1000::1

To display the vRouter loopback interface, use the vrouter-loopback-interface-show

command:

CLI network-admin@Leaf1 > vrouter-loopback-interface-show

vrouter-name index ip router-if

174 ------------ ----- ------------ ---------
175 vr1 1 99.1.1.1 eth0.100
176 vr1 2 2999:1000::1 eth0.100

CLI network-admin@Leaf1 > vrouter-routes-show

vrouter-name network type interface next-hop

distance
------------ ---------------- --------- --------- -------------------------
--------
vr1 10.1.1.0/24 connected eth0.100
vr1 10.1.1.0/24 connected eth0.100
vr1 20.1.1.0/24 ospf eth0.121 12.1.1.2 110
vr1 88.1.1.1/32 ospf eth0.121 12.1.1.2 110
vr1 99.1.1.1/32 connected lo
vr1 2100:100::/64 connected eth0.100

Pluribus Networks
www.pluribusnetworks.com
147
vr1 2121:100::/64 connected eth0.121
vr1 2200:100::/64 ospf6 eth0.121 fe80::640e:94ff:fe4d:d0c8 110
vr1 2888:1000::1/128 ospf6 eth0.121 fe80::640e:94ff:fe4d:d0c8 110
vr1 2999:1000::1/128 connected lo
vr1 fe80::/64 connected eth0.121

metric
------

20
20

10
10

Configuring Prefix Lists for BGP and OSPF

Prefix lists allow you to permit or deny host IP addresses from route distribution in BGP and
OSPF configurations. To configure prefix lists for BGP, this example assumes you have a
vRouter configured for BGP, vrouter-bgp, and you want to deny the IP address, 172.26.0.0
with the netmask 255.255.0.0, sequence number 5, and minimum prefix length 17 bits:

CLI network-admin@Leaf1 > vrouter-prefix-list-add vrouter-name vrouter-bgp

name deny-bits action deny prefix 172.26.0.0 netmask 255.255.0.0 seq 5
min-prefix-len 17

This prefix list rejects any subnets of 172.26.0.0/16 with prefixes 17 bits or longer. For
example, the subnets 172.26.16.9/30 and 172.26.101.0/24 are rejected from route
distribution.
The sequence number allows you to insert or remove new lines in a prefix list as well as at
the beginning or end. Pluribus Networks recommends you increment the sequence numbers
by 10 so you can easily add or subtract lists from the configuration. See also:
 Configuring Open Shortest Path First (OSPF)
 Configuring BGP on a vRouter

148
Pluribus Networks
www.pluribusnetworks.com
Configuring Packet Relay for DHCP Servers
You configure a vRouter to relay DHCP requests from local clients to a centralized DHCP
server. Because the initial DHCP request arrives from a client that typically does not have an
IP address, the client must find the DHCP server using a Layer 2 broadcast.
The DHCP server needs information before the server can allocate an IP address to the client.
It must know the subnet and the MAC address of the client. The DHCP server needs the
subnet information to ensure that the IP address that the client receives can work on the
client’s subnet. The MAC address is necessary so that the DHCP server can find any
information that is unique to the client.
When you configure the vRouter as a DHCP proxy, the vRouter converts the local broadcast
packet from the client to a unicast packet and forwards it to the server.
Because the DHCP client does not have an IP address when it sends the DHCP request
packet, the client uses the IP address, 0.0.0.0, as the source IP address and the general
broadcast address 255.255.255.255 for the destination.
The vRouter replaces the source address with the IP address assigned to the interface
receiving the the request, and replaces the destination IP address with the address you
specify in the vRouter packet-relay command.
To configure packet-relay for a DHCP server with the IP address 172.16.21.34 and vRouter
interface eth11.100, use the following syntax:

CLI network-admin@switch > vrouter-packet-relay add vrouter-name vrouter-dhcp

forward-proto dhcp forward-ip 172.16.21.34 nic eth11.100

Once you add the configuration, you cannot modify it. If you make a mistake or want to add
a new configuration, you must use the vrouter-packet-relay-remove command.

Configuring Hardware Routing for a vRouter

Create interfaces on hardware routers and map them to virtual NICs in the vRouter zone.
Currently, only one hardware vRouter is supported with Netvisor OS.
Netvisor supports the following protocols:
OSPF — OSPF does not use a TCP/IP transport protocol such as UDP or TCP, but is
encapsulated in the IP datagram with protocol number 89. OSPF uses multicast
addressing for route flooding on a broadcast domain. For nonbroadcast network,
special provisions in the configuration facilitate neighbor discovery. OSPF reserves the
multicast addresses 224.0.0.5/6 for IPv4 or FF02::5/6 for IPv6.
BGP — BGP uses TCP and port number 179
RIP — uses the following parameters:

RIPv1 — IPv4 uses UDP and port 520, and advertise address - broadcasting
RIPv2 — IPv4 uses UDP and port 520, and advertise address - 224.0.0.9
RIPng — IPv6 uses UDP and port 521, and advertise address - FF02::9
PIM — IPv4 uses protocol 103 with multicast address 224.0.0.13
To create a hardware router on a vRouter, hwtest, use the following command:

CLI network-admin@Leaf1 > vrouter-create hwtest router-type hardware

Use the same commands as software routing to add protocols and interfaces.

Pluribus Networks
www.pluribusnetworks.com
149
Support for Displaying Quagga Routing and Debug Information for vRouters
This feature provides a new vRouter command to display the output of any Quagga routing
and debug information from the Netvisor OS command line interface(CLI).

CLI (network-admin@Spine1) > help vrouter-vtysh-cmd

vrouter-vtysh-cmd display output of a Quagga show command
name name-string name of service config
cmd cmd-string any Quagga show/debug/undebug/no debug command
Show Output Examples
CLI (network-admin@Spine1) > vrouter-vtysh-cmd name vr1 cmd "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
> - selected route, * - FIB route

C>* 100.1.1.0/24 is directly connected, eth1.100

C>* 127.0.0.0/8 is directly connected, lo0
CLI (network-admin@Spine1) > vrouter-vtysh-cmd name vr1 cmd "show
running-config"
Building configuration...

Current configuration:
!
hostname vr1
log file zebra.log
!
password zebra
enable password zebra
!
interface eth1.100
ipv6 nd suppress-ra
multicast
no link-detect
!
interface lo0
no link-detect
!
ip forwarding
ipv6 forwarding
!
line vty
!
end

For this feature, Netvisor only allows show commands as legal values in the CLI.

CLI network-admin@Leaf1 > vrouter-vtysh-cmd name vr1 cmd "config t"

vrouter-vtysh-cmd: Illegal value for "cmd"

vrouter-vtysh-cmd: Legal values: commands starting with show, in quotes

150
Pluribus Networks
www.pluribusnetworks.com
Viewing Quagga Logs
Quagga logs files, such as Zebra, OSPF, OSPF6, BGP, BFD and RIP can be viewed directly
from the console using the Netvisor OS CLI. Because Quagga log files accumulate on the
switch, you may want to clear a particular log file so they do not create space issues.

New Commands
Use this command to view Quagga logs files directly from your console:

CLI network-admin@Leaf1 > vrouter-log-show vrouter-name vrouter-name protocol

vrouter-log-show Displays vrouter protocol logs

one or more of the following options:
vrouter-name vrouter-name Specify name of the vRouter.
protocol Specify the name of the protocol
zebra|ospf|ospf6|bgp|bfd|rip files to view.

Use this command to clear Quagga files from a specific protocol log file:

CLI network-admin@Leaf1 > vrouter-log-clear

vrouter-log-clear Clears vrouter protocol logs from a

protocol log file
one or more of the following options:
vrouter-name vrouter name Specify the name of the vRouter
service.
protocol Specify the name of the log file to
zebra|ospf|ospf6|bgp|bfd|rip clear.

Support for Hardware vRouter Migration

In earlier versions of Netvisor OS, you could not migrate a hardware vRouter from one switch
to another switch, and you had to reconfigure the hardware vRouter on the new switch.
You can migrate the hardware vRouter using the vrouter-migrate command. For example,
to migrate the hardware vRouter, hw-vrouter1 to a new location, Spine3,and storage pool,
disk3, use the following syntax:

CLI network-admin@switch > vrouter-migrate name hw-vrouter1 location Spine3

storage-pool disk3

Pluribus Networks
www.pluribusnetworks.com
151
Configuring BGP on a vRouter
Border Gateway Protocol (BGP) is a path-vector protocol and the most commonly used
routing protocol on the Internet. BGP advertises the paths required to reach a certain
destination. BGP resides on top of TCP, and simpler to configure than Open Shortest Path
First (OSPF). In Figure 1 Configuring BGP for Two VLANs, you want network traffic from the
source host to reach the destination host. But when you configure different VLANs, the
source host traffic becomes unaware of the route between the source host and the
destination host. However, a VLAN spans VLAN 33 and VLAN 55. You solve this problem by
configuring BGP in the same Autonomous System (AS) 100 sending traffic over VLAN 35.
This allows the source host to learn the route to the destination host.
Using a loopback address for peering is useful when multiple paths exist between the BGP
peers which would otherwise tear down the BGP session if the physical interface used for
establishing the session goes down. It also allows the vRouters running BGP with multiple
links between them to load balance over the available paths.
Figure 1: Configuring BGP for Two VLANs

This example assumes you have two VLANs, VLAN33 and VLAN55. Also, you have added
ports to the configuration.
Begin by configuring vRouter1, a software vRouter, on VLAN 33 with the BGP information:

CLI network-admin@Leaf1 > vrouter-create name vrouter1 fabricname-global

router-type hardware bgp-as 100 bgp-redist-connected-metric none

Additional BGP parameters include the following:

bgp-redist-static-metric — redistribute static BGP route metric number
bgp-redist-connected-metric — redistribute connected BGP route metric
bgp-redist-rip-metric — redistribute BGP into RIP process metric
bgp-redist-ospf-metric — redistribute BGP into OSPF process metric
bgp-cluster-id — the ID assigned to the BGP cluster.
bgp-ibgp-multipath — allow the BGP vRouter to select multiple paths for load
sharing

152
Pluribus Networks
www.pluribusnetworks.com
 bgp-max-paths — maximum number of BGP paths
bgp-bestpath-as-path — allow BGP to use the best path for traffic forwarding
bgp-dampening|no-bgp-dampening — suppress flapping routes so they are not
advertised.
bgp-graceful-restart|no-bgp-graceful-restart — mechanism for BGP that helps
minimize the negative effects on routing caused by BGP restart
bgp-stalepath-time — how long a router waits before deleting stale routes after an
end of record (EOR) message is received from the restarting router.
Add the IP addresses and VLANs:

CLI network-admin@Leaf1 > vrouter-interface-add vrouter-name vrouter1 ip

10.16.35.33/24 vlan 35

CLI network-admin@Leaf1 > vrouter-interface-add vrouter-name vrouter1 ip

10.16.33.1/24 vlan 33

Add the BGP information:

CLI network-admin@Leaf1 > vrouter-bgp-add vrouter-name vrouter1 neighbor

10.16.35.55 remote-as 100

CLI network-admin@Leaf1 > vrouter-bgp-add vrouter-name vrouter1 network

10.16.33.0/24

Display the interface information for vrouter1:

CLI network-admin@Leaf1 > vrouter-interface-show format all layout vertical

vrouter-name: vrouter1
nic: eth1.33
ip: 10.9.100.100/16
assignment: static
mac: 66:0e:94:30:c6:92
vlan: 33
vxlan: 0
if: data
alias-on:
exclusive: no
nic-config: enable
nic-state: up
secondary-macs:
vrouter-name: vrouter1
nic: eth2.33
ip: 192.168.42.11/24
assignment: static
mac: 66:0e:94:30:25:5e
vlan: 33
vxlan: 0
if: data
alias-on:
exclusive: no
nic-config: enable
nic-state: up
secondary-macs:

Pluribus Networks
www.pluribusnetworks.com
153
To filter IP hosts, add prefix lists to the BGP configuration. See Configuring Prefix Lists for
BGP and OSPF.
Then, configure vRouter2 on VLAN 55:

CLI network-admin@Leaf1 > vrouter-create name vrouter2 fabricname-global

router-type hardware bgp-as 100 bgp-redist-connected-metric none

Add the IP addresses and VLANs:

CLI network-admin@Leaf1 > vrouter-interface-add vrouter-name vrouter2 ip

10.16.35.55/24 vlan 35

CLI network-admin@Leaf1 > vrouter-interface-add vrouter-name vrouter2 ip

10.16.55.1/24 vlan 55

Then add the BGP information:

CLI network-admin@Leaf1 > vrouter-bgp-add vrouter-name vrouter2 neighbor

10.16.35.33 remote-as 100

CLI network-admin@Leaf1 > vrouter-bgp-add vrouter-name vrouter2 network

10.16.55.0/24

And finally, add the loopback address:

CLI network-admin@Leaf1 > vrouter-loopback-interface-add vrouter-name vrouter1

index 5 ip 1.1.1.1

Netvisor uses the index value as a number to uniquely identifies the vRouter in the AS.
Display the vRouter BGP configuration:

CLI network-admin@Leaf1 > vrouter-bgp-show format all layout vertical

vrouter-name: vrouter33
ip: 10.16.35.55
neighbor: 10.16.35.55
remote-as: 100
next-hop-self: no
route-reflector-client: no
override-capability: no
soft-reconfig-inbound: no
max-prefix-warn-only: no
vrouter-name: vrouter33
ip: 10.16.33.0
network: 10.16.33.0/24
vrouter-name: vrouter55
ip: 10.16.35.33
neighbor: 10.16.35.33
remote-as: 100
next-hop-self: no
route-reflector-client: no
override-capability: no
soft-reconfig-inbound: no
max-prefix-warn-only: no
vrouter-name: vrouter55
ip: 10.16.55.0

154
Pluribus Networks
www.pluribusnetworks.com
network: 10.16.55.0/24

To reset BGP neighbors, use the vrouter-bgp-neighbor-reset command.

To display BGP neighbors, use the vrouter-bgp-neighbor-show command.

CLI network-admin@switch > vrouter-bgp-neighbor-show

vrouter-name: vrouter1
neighbor: 10.9.100.201
ver: 4
remote-as: 100
msg_rcvd: 11
msg_sent: 19
tblver: 0
inQ: 0
outQ: 0
up/down: 00:54:04
state/pfxrcd: Connect
vrouter-name: vrouter2
neighbor: 10.9.100.101
ver: 4
remote-as: 100
msg_rcvd: 12
msg_sent: 18
tblver: 0
inQ: 0
outQ: 0
up/down: 00:53:37
state/pfxrcd: Connect

Additional BGP Parameters

There are additional BGP parameters that you can use to optimize your BGP network. Add
any of the following parameters:
ebgp-multihop — a value for external BGP to accept or attempt BGP connections to
external peers, not directly connected, on the network. This is a value between 1 and
255.
update-source vrouter — the source IP address of BGP packets sent by the router.
This parameter is required if you want BGP to perform peering over a loopback
interface.
prefix-list-in — specify a list of incoming prefixes for route redistribution.
prefix-list-out — specify a list of outgoing prefixes for route redistribution.
override-capability — override the result of capability negotiation with the local
configuration. This parameter allows you to ignore a remote peer’s capability value.
soft-reconfig-inbound — defines the route refresh capability by allowing the local
device to reset inbound routing tables dynamically by exchanging route refresh
requests to supporting peers.
max-prefix — allows you to specify the maximum number of IP prefixes to filter.
max-prefix-warn — add a parameter to warn when the maximum number of prefixes
is reached.

Pluribus Networks
www.pluribusnetworks.com
155
Support for BGP SNMP MIBs
You now enable or disable SNMP MIBs for BGP configurations by using the command,
vrouter-create, or vrouter-modify:

CLI network-admin@switch > vrouter-create name-string bgp-snmp|no-bgp-snmp

bgp-snmp-notification|no-bgp-snmp-notification

Support for AS and AS Prepending and BGP

You can prepend one or more autonomous system (AS) numbers at the beginning of an AS
path. The AS numbers are added at the beginning of the path after the actual AS number
from which the route originates has been added to the path. Prepending an AS path makes a
shorter AS path look longer and therefore less preferable to BGP.
The BGP best path algorithm determines how the best path to an autonomous system (AS) is
selected. The AS path length determines the best path when all of the following conditions
are met:
There are multiple potential routes to an AS.
BGP has the lowest preference value, sometimes referred to as the administrative
distance, of the available routes
The local preferences of the available routes are equal.
When these conditions are met, Netvisor uses the AS path length as the tie breaker in the
best path algorithm. When two or more routes exist to reach a particular prefix, BGP prefers
the route with the shortest AS Path length.
If you have multi-homing to one or more service providers, you may prefer for incoming
traffic take a particular path to reach your network. Perhaps you have two connections, but
one costs less than the other. Or you might have one fast connection and another, much
slower connection that you only want to use as a backup if your primary connection is down.
AS path prepending is an easy method that you can use to influence inbound routing to your
AS.
Netvisor OS has new parameters for the vrouter-route-map-* commands:

CLI network-admin@Leaf1 > vrouter-route-map-add

as-path-prepend integer Specify a value between 1 and

4,294,967,295.
as-path-prepend-last-as integer Specify a value between 1 and 10.
as-path-exclude integer Specify a value between 1 and
4,294,967,295.

CLI network-admin@Leaf1 > vrouter-route-map-show

as-path-prepend integer Displays a value between 1 and

4,294,967,295.
as-path-prepend-last-as integer Displays a value between 1 and 10.
as-path-exclude integer Displays a value between 1 and
4,294,967,295.

156
Pluribus Networks
www.pluribusnetworks.com
Bidirectional Forwarding Detection Support for IPv6 BGP Neighbor and IPv6
Static Routes
This feature adds bidirectional forwarding detection for IPv6 BGP neighbor and provides
support for IPv6 BGP NBR reachability detection by using BFD protocol. When a BFD session
goes from UP to DOWN, BFD informs BGP to bring the neighbor (NBR) down, until BFD
returns to an UP state.
You create the BFD session by adding the bfd parameter to the vRouter configuration using
the bfd parameter for the command, vrouter-bgp-modify. IPv6 BFD sessions for BGP NBRs
are hosted in Netvisor. The BFD session starts when you add the bfd parameter the BGP
vRouter configuration.

CLI network-admin@Leaf1 > vrouter-bgp-neighbor-show

vrouter-name neighbor ver remote-as msg_rcvd msg_sent up/down

state/pfxrcd
------------ ----------- --- --------- -------- -------- ------ ---------
-----
vr10 2002:100::2 4 51000 189 193 00:00:49 0

multi-protocol
--------------
ipv6-unicast

Netvisor supports IPv6 static route reachability detection using BFD protocol. Add IPv6 BFD
sessions by specifying two end point IPv6 addresses, a source IPv6 address and a destination
IPv6 address. The source IPv6 address must be a known local IPv 6 address. When a BFD
session is up, Netvisor assesses all the IPv6 static routes configured with a gateway or a BFD
destination IPv6 address matching the destination IPv6 address of the BFD session. When
Netvisor finds a match, this static route installs in Routing Information Database (RIB) and
Forwarding Information Database (FIB).

CLI network-admin@Leaf1 > vrouter-static-bfd-show

vrouter-name src-ip dst-ip type

------------ ------------------------- ------------------------- ----------
vr10 2006:100::2 2002:100::2 multi-hop

Support for Border Gateway Protocol (BGP) Communities

A BGP community consists of a group of prefixes sharing some common property and can be
configured with the BGP community attribute. The BGP Community attribute comprises an
optional transitive attribute of variable length. The attribute consists of a set of four octet
values specifying a community. Netvisor encodes the community attribute values with an
Autonomous System (AS) number in the first two octets, with the remaining two octets
defined by the AS. A prefix may have more than one community attribute. A BGP speaker
sees multiple community attributes in a prefix can act based on one, some or all the
attributes. You have the option to add or modify a community attribute before the router
passes the attribute on to other peers.
The local preference attribute indicates to the AS the preferred path in order to reach a
certain network. When multiple paths to the same destination occur, Netvisor uses the path
with the higher preference (the default value of the local preference attribute is 100).
Common Community Attributes

Pluribus Networks
www.pluribusnetworks.com
157
 Standard (well known) — These community attributes are 4 octets long, with well
known values
Internet (0) — advertise these routes to all neighbors.
no-export (0xFFFFFF01) — do not advertise to outside a BGP confederation
boundary.
no-advertise (0xFFFFFF02) — do not advertise to other BGP peers .
local-AS (0xFFFFFF03) — do not advertise to external BGP peers.
Standard - generic (AS:value) — These community attributes are also 4 octet long,
but values can be really generic. The first 16-bit number is normally the AS number of
the network that sets the community or looks for it, and the second number is one
that conveys the intended information, for example: 65001:100.
For example to set the community attribute, no-export, to all route prefixes matching prefix
subnet100, use the following syntax:

CLI network-admin@Leaf1 > vrouter-route-map-add vrouter-name vr1 name rmap1

seq 10 action permit match-prefix subnet100 community-attribute no-export

To set the community attribute, 65002:200 to all route prefixes matching prefix subnet100,
use the following syntax:

CLI network-admin@Leaf1 > vrouter-route-map-add vrouter-name vr1 name peer vr2

action permit seq 20 match-prefix subnet99 community-attribute-generic
65002:200

Community Lists
BGP community list consists of a user defined BGP communities attribute list. The BGP
community list can be used for matching or manipulating BGP communities attribute in
updates. Netvisor uses community list on the receive side of the BGP updates to match
existing in the received updates. Community lists can be used in route-map with
match-community keyword to apply any policy on the receive side.
Standard — Standard community list defines attribute which matches standard
communities as defined above (well known or generic).
To set the community list permitting the community value 300 for AS 65002, use the
following syntax:

CLI network-admin@Leaf1 > vrouter-community-list-add vrouter-name vr2 style

standard name clist300 action permit community-attribute 65002:300

Expanded — Expanded means the string entered for the community-attribute can be
a regular expression instead of AS:NN format or some well-known attributes.
To set an expanded community list denying updates with community values 1 through 99
in autonomous System 50000, use the following syntax:

CLI network-admin@Leaf1 > vrouter-community-list-add vrouter-name vr1 style

expanded name c199 action deny community-attribute 50000:[0-9][0-9]

158
Pluribus Networks
www.pluribusnetworks.com
The Netvisor commands for vrouter-route-maps-* support additional parameters for BGP
communities:

CLI network-admin@Leaf1 > vrouter-route-map-add

match-community Specify the community string to match. (BGP only)

match-community-string
exact-match|no-exact-match Specify if the community string is an exact match or
not. (BGP only)
community-attribute-generic Specify a generic community attribute such as
community-attribute-generic- AA:NN. (BGP only)
string
additive|no-additive Specify if a given community is appended to existing
communities value.
comm-list-del vrouter Specify if you want to remove community values
community-list name from BGP community attributes.

New commands support creating BGP Communities:

CLI network-admin@Leaf1 > vrouter-community-list-add

vrouter-name name-string Specify a vRouter to add the community list.

Add the following community list parameters:
style standard|expanded Specify the style of the community list.
name name-string Specify a name for the community list.
action permit|deny Specify the action for the community list.
community-attribute Specify the community attribute.
community-attribute-string

CLI network-admin@Leaf1 > vrouter-community-list-remove

vrouter-name name-string Specify a vRouter to remove the community list.

Add the following community list parameters:
style standard|expanded Specify the style of the community list.
name name-string Specify a name for the community list.
action permit|deny Specify the action for the community list.
community-attribute Specify the community attribute.
community-attribute-string

CLI network-admin@Leaf1 > vrouter-community-list-show

Pluribus Networks
www.pluribusnetworks.com
159
 vrouter-name name-string Displays the vRouter name.
Add the following community list parameters:
style standard|expanded Displays the style of the community list.
name name-string Displays a name for the community list.
action permit|deny Displays the action for the community list.
community-attribute Displays the community attribute.
community-attribute-string

Configuring Open Shortest Path First (OSPF)

Open Shortest Path First (OSPF) consists of a robust link-state interior gateway protocol
(IGP). It uses the concept of Areas which allows further segmentation on the network.
OSPF uses link-state information to make routing decisions, and make route calculations
using the shortest path first (SPF) algorithm. Each vRouter configured for OSPF floods
link-state advertisements throughout the area containing information about the interfaces
attached to the router and routing metrics.
You can add more configuration options, such as hello intervals, for OSPF using the
vrouter-interface-config commands. In addition, you can add stub or not-so-stubby
areas to the OSPF configuration.
You can also manually change the OSPF cost for the configuration. Cost uses the metric in
OSPF to judge the feasibility of a path. If you specify 0 as the cost, the vRouter automatically
calculates the cost based on the bandwidth of the interface.

Informational Note: For switches with ONVL, the only available VNET
in Netvisor consists of a global VNET created when you create a fabric for
the first time. Use tab complete in the CLI to display the VNET and
continue the configuration.

In this example, you configure OSPF for two vRouters with an area of 5. The network has the
following configuration:
 VLAN 35 with IP addresses 10.16.35.0/24

 VLAN 45 with IP addresses 10.16.55.0/24

Figure 2: OSPF

1. First, create the vRouter for Router1.

160
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > vrouter-create name vrouter1 fabricname-global
router-type hardware

2. Add vRouter interfaces to the vRouter:

CLI network-admin@switch > vrouter-interface-add vrouter-name vrouter1 ip

10.0.3.1 netmask 24 vlan 35 if data nic-enable

CLI network-admin@switch > vrouter-interface-add vrouter-name vrouter1 ip

172.37.0.17 netmask 16 vlan 55 if data nic-enable

3. Add the subnets, 10.0.3.0/24 and 172.37.0.0/16, to VLAN33 with the area 0:

CLI network-admin@switch > vrouter-ospf-add vrouter-name vrouter1 network

10.0.3.0/24 ospf-area 0

4. Add the second IP address with the area 0.

CLI network-admin@switch > vrouter-ospf-add vrouter-name vrouter1 network

172.37.0.0/16 ospf-area 0

5. Add interfaces for OSPF hello intervals of 30 seconds:

CLI network-admin@switch > vrouter-interface-config-add name router1 nic

eth0.35 ospf-hello-interval 30 ospf-cost 0

CLI network-admin@switch > vrouter-interface-config-add name router1 nic

eth0.55 ospf-hello-interval 30 ospf-cost 0

If you specify 0 as the cost value, the vRouter calculates the OSPF cost automatically based
on the bandwidth of the interface.
When you modify the OSPF hello interval, the ospf-dead-interval is automatically reset to
4 times the hello interval.

6. Display the configuration by using the vrouter-ospf-show command:

CLI network-admin@switch > vrouter-ospf-show layout vertical

vrouter-name: vrouter1
network: 10.0.3.0
netmask: 24
ospf-area: 0
vrouter-name: vrouter1
network: 172.37.0.0
netmask: 16
ospf-area: 0
stub-area: 11
stub-type: stub
ospf-hello-interval: 30
metric: 34

The metric value can reflect the cost of routes advertised as OSPF routes. It may also reflect
the cost of routes advertised with other protocols.

Pluribus Networks
www.pluribusnetworks.com
161
Display Default Timers for OSPF Configurations
Netvisor allows you to display default timers for OSPF configurations. To add a timer or
modify an existing timer, use the following commands:

CLI network-admin@Leaf1 > vrouter-interface-config-add

ospf-retransmit-interval Specify the OSPF retransmit interval from 3

seconds to 65535 seconds. The default is 5 seconds.
(IPv4 orIPv6)

CLI network-admin@Leaf1 > vrouter-interface-config-modify

ospf-retransmit-interval Specify the OSPF retransmit interval from 3

seconds to 65535 seconds. The default is 5 seconds.
(IPv4 orIPv6)

A new command displays the OSPF configuration for an interface:

CLI network-admin@Leaf1 > vrouter-ospf-interface-show

i
vrouter-name name-string Displays the name of the vRouter.
Specify any of the following optional OSPF parameters:
nic vrouter interface nic Displays the name of the vNIC.
nic-state down|up Displays the vNIC state.
l3-port l3-port-number Displays the Layer 3 port numbers.
ip ip-address Displays the IPv4 address of the interface.
netmask netmask Displays the netmask of the IPv6 address.
broadcast ip-address Displays the broadcast IP address.
area ip-address Displays the area ID for the interface in IPv4
format.
mtu mtu-number Displays MTU for the interface.
mtu-mismatch-detection| Displays if MTU mismatch detection is
no-mtu-mismatch-detection configured.
router-id ip-address Displays the router ID as an IP address.
network-type Displays the OSPF network type.
point-to-point|broadcast|loopback
state Displays OSPF interface state.
down|loopback|waiting|point-to-poin
t|dr-other|backup|dr
dr-id ip-address Displays the designated router ID.
dr-ip ip-address Displays the designated router IP address.

162
Pluribus Networks
www.pluribusnetworks.com
bdr-id ip-address Displays the backup designated router ID.
bdr-ip ip-address Displays the designated router IP address.
priority priority-number Displays the priority.
cost cost-number Displays the cost.
hello hello-number(s) Displays the hello-interval in seconds.
dead dead-number(s) Displays the dead time in seconds.
retransmit retransmit-number(s) Displays the retransmit interval time in seconds.
hello-due hello-due-string Displays the hello due in.
neighbor neighbor-number Displays the neighbor count.
adjacent adjacent-number Displays the adjacent number count.

CLI network-admin@Leaf1 > vrouter-ospf6-interface-show

i
vrouter-name name-string Displays the name of the vRouter.
Specify any of the following optional OSPF parameters:
nic vrouter interface nic Displays the name of the vNIC.
nic-state down|up Displays the vNIC state.
l3-port l3-port-number Displays the Layer 3 port numbers.
link-local ip-address Displays the IPv6 link-local IP address.
ip6 ip-address Displays the IPv6 address of the interface.
netmask-ip6 netmask Displays the netmask of the IP address.
area ip-address Displays the area ID for the interface in IPv4
format.
mtu mtu-number Displays MTU for the interface.
mtu-mismatch-detection| Displays if MTU mismatch detection is
no-mtu-mismatch-detection configured.
state Displays OSPF interface state.
down|loopback|waiting|point-to-
point|dr-other|backup|dr
dr-id ip-address Displays the designated router ID.
bdr-id ip-address Displays the backup designated router ID.
priority priority-number Displays the priority.
cost cost-number Displays the cost.
hello hello-number(s) Displays the hello-interval in seconds.
dead dead-number(s) Displays the dead time in seconds.
retransmit retransmit-number(s) Displays the retransmit interval time in seconds.
if-scoped-lsa if-scoped-lsa-number Displays the number of interface LSAs scoped for
the area.

Pluribus Networks
www.pluribusnetworks.com
163
ls-update ls-update-number Displays the number of pending LSAs for
LSUpdate.
ls-ack ls-ack-number Displays the number of pending LSAs for LSAck.

Adding Areas and Prefix Lists to OSPF

You can configure OSPF areas as a stub area, stub-no-summary area, or a not so stubby area
(NSSA). Stub areas see detailed routing information from other areas, but only summary
information about networks outside of the AS. Stub-no-summary areas summarize external
routes and routes from other areas. Routers in this type of area only see routing information
local to their area. Not so stubby areas (NSSA) connects to the external network by
introducing a Link State Advertisement (LSA) used within the area to carry external routes
originating with boundary routers connected to this area.
To add a stub area to vRouter, vrouter-ospf, with area 100, use the following command:

CLI network-admin@switch > vrouter-ospf-area-add vrouter-name vrouter-ospf

area 100 stub-type stub

In addition, you can add prefix lists to filter host IP addresses. To add prefix lists to OSPF
areas, see Configuring Prefix Lists for BGP and OSPF.

BFD Support for OSPF Fault Direction

Bidirectional Forwarding Detection (BFD) can be used for OSPF fault detection. This feature
provides fast failure detection when an intermediate device resides between two
non-adjacent OSPF neighbors. BFD provides fast failure detection between two nodes and
notifies any protocol (OSPF) of this event to make it converge much faster than with default
timers in protocols. Because OSPF hello timers may not be fast enough for detecting
neighbor loss, you can use BFD to establish a BFD connection with the OSPF neighbor and
bring an OSPF neighbor down as soon as BFD detects an issue.
Note that you use BFD for down detection only. This means that regular OSPF neighbor
discovery and state machine transitions are not affected by enabling BFD.
You can enable BFD on all OSPF interfaces at the global level or on a specific interface. By
default, Netvisor disables BFD globally and on all interfaces, with individual interface
configuration taking precedence over the global setting.

Informational Note: Netvisor does no support BFD for OSPFv6.

New Command Options

Setting the vrouter global OSPF level:
To enable OSPF BFD per vRouter on global OSPF level:

164
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > vrouter-modify name vrouter-name

vrouter-modify modify a vrouter

one or more of the following
options:
enables or disables BFD protocol for
ospf-bfd-all-if|no-ospf-bfd-all- fault detection on all OSPF
if interface. Default is disabled.

Setting the Interface (nic) OSPF level:

To enable OSPF BFD per interface If there is no interface configuration :

CLI network-admin@switch > vrouter-interface-config-add vrouter-name nic .

vrouter-interface-config-add Add an interface

configuration to a vRouter
VNIC name.
one or more of the following options:
nic vrouter interface nic Specify the name of the
VNIC.
[ospf-bfd] default|enable|disable Enable BFD protocol support
for OSPF fault detection.

To enable OSPF BFD per interface if a interface configuration already exists:

CLI network-admin@switch > vrouter-interface-config-modify vrouter-name name

nic nic ospf-bfd enable|disable|default

vrouter-interface Modify an interface

configuration to a vRouter

One or more of the following options: Specify the name of the

VNIC.
ospf-bfd enable|disable|default Enables or disables the BFD
protocol for fault detection on
all OSPF interface. Default is
disabled.

Displaying the OSPF BFD Configuration State

To display the configuration state of OSPF BFD, use these show commands:

CLI network-admin@switch > vrouter-show

CLI network-admin@switch > vrouter-interface-show

Informational Note: There are no changes to the commands:

vrouter-ospf-neighbor-show and vrouter-bfd-neighbor-show

Pluribus Networks
www.pluribusnetworks.com
165
Support for Route Maps for OSPF Routes
Configure route maps to associate a redistribute metric or metric-type for OSPFv3 and
currently, you cannot configure metrics for OSPF. In this release, you can now define a route
map to prevent OSPF routes from getting added to the routing table. This filtering happens at
the moment when OSPF installs the route in the routing table.
Before you configure route maps, configure a list of prefixes using the following command:

CLI network-admin@Leaf1 > vrouter-prefix-list-add

Use the following set of commands to configure route maps for OSPF:

CLI network-admin@Leaf1 > vrouter-route-map-add

vrouter-name name-string Specify the name of the vRouter service.

name name-string Specify the name for the route map.
seq integer Specify the sequence number as an integer between
1 and 4294967295.
action permit|deny Specify the route map action as permit or deny.
match-prefix vrouter prefix-list Specify the name of the prefix list used for the route
name-string map.
community-attribute Specify the community attribute for the route map.
unset|none|no-export|no-advertise
|
internet|local-AS
local-pref integer Specify the local preference as an integer between -1
and 4294967295.
metric none Specify the metric as none.
metric-type 1|2 Specify the metric type as 1 or 2.

CLI network-admin@Leaf1 > vrouter-route-map-remove

vrouter-name name-string Specify the name of the vRouter service.

name name-string Specify the name for the route map.
seq integer Specify the sequence number as an integer between
1 and 4294967295.

CLI network-admin@Leaf1 > vrouter-route-map-show

vrouter-name name-string Displays the name of the vRouter service.

name name-string Displays the name for the route map.
seq integer Displays the sequence number as an integer
between 1 and 4294967295.
action permit|deny Displays the route map action as permit or deny.
match-prefix vrouter prefix-list Displays the name of the prefix list used for the
name-string route map.

166
Pluribus Networks
www.pluribusnetworks.com
community-attribute Displays the community attribute for the route map.
unset|none|no-export|no-advertis
e|
internet|local-AS
local-pref integer Displays the local preference as an integer between
-1 and 4294967295.
metric none Displays the metric as none.
metric-type 1|2 Displays the metric type as 1 or 2.

Support for OSPF SNMP MIBs

You now enable or disable SNMP MIBs for OSPF configurations by using the command,
vrouter-create, or vrouter-modify:

CLI network-admin@switch > vrouter-create name-string ospf-snmp|no-ospf-snmp

ospf-snmp-notification|no-ospf-snmp-notification

Adding Default Route Information Settings for OSPF Routing

An OSPF vRouter, by default, does not generate a default route into the OSPF domain. In
order for OSPF to generate a default route, you must explicitly configure the vRouter with
this information.
There are two ways to inject a default route into a normal area.
1. If the ASBR already has the default route in the routing table, you can advertise the
existing 0.0.0.0/0 into the OSPF domain with the ospf-default-information parameter.
2. If the ASBR doesn't have a default route, you can add the keyword always to the
ospf-default-information command.

The OSPF vRouter advertises a default route into the OSPF domain, even if a route to 0.0.0.0
is configured. Another benefit of adding always keyword is that it can add stability to the
internetwork. For example, if the ASBR is learning a default route from another routing
domain such as RIP and this route is flapping, then without specifying always keyword, each
time the route flaps, the ASBR sends a new Type 5 LSA into the OSPF domain causing
instability inside the OSPF domain. With the always keyword, the ASBR always advertises
the default route inside the OSPF domain, and the flapping of the default route from the RIP
domain does not cause any instability inside the OSPF domain.
When you configure the metric type, you can use the parameters described for the vrouter-
commands for configuring a route map.
These parameters control default route generation for IPv4 and IPv6 default routes.

CLI network-admin@Leaf1 > vrouter-create

Pluribus Networks
www.pluribusnetworks.com
167
 ospf-default-information Specify if you want to use the
none|originate|always default route information for
OSPF.
• none — no default route is
generated.
• originate — the default
route is generated only if a
default route is present in the
routing table.
• always — the default route is
generated even if no default
route is present in the routing
table.
ospf-default-info-originate-metric none Specify the metric for the
default route.
ospf-default-info-originate-metric-type 1|2 Specify the metric type as 1 or
2.

CLI network-admin@Leaf1 > vrouter-modify

ospf-default-information Specify if you want to use the

none|originate|always default route information for
OSPF.
• none — no default route is
generated.
• originate — the default
route is generated only if a
default route is present in the
routing table.
• always — the default route is
generated even if no default
route is present in the routing
table.
ospf-default-info-originate-metric none Specify the metric for the
default route.
ospf-default-info-originate-metric-type 1|2 Specify the metric type as 1 or
2.
ospf-default-info-originate-route-map Specify the OSPF default
vrouter route-map name information route map.

CLI network-admin@Leaf1 > vrouter-show

168
Pluribus Networks
www.pluribusnetworks.com
 ospf-default-information Displays the default route
none|originate|always information for OSPF.
• none — no default route is
generated.
• originate — the default
route is generated only if a
default route is present in the
routing table.
• always — the default route is
generated even if no default
route is present in the routing
table.
ospf-default-info-originate-metric none Displays the metric for the
default route.
ospf-default-info-originate-metric-type 1|2 Displays the metric type as 1 or
2.
ospf-default-info-originate-route-map Displays the OSPF default
vrouter route-map name information route map.

Adding Metric and Metric Type for Route Maps

Configure route maps to associate a redistribute metric and metric-type for OSPFv3 and you
cannot configure metrics using the current OSPF commands. Use route maps method to
configure metric and metric-type on routers. Configuration of metrics and metric-type for
OSPFv3 requires route maps.

CLI network-admin@Leaf1 > vrouter-route-map-add

metric none Specify a metric for the route map.

metric-type none|1|2 Specify the metric type.

The new parameters also apply to the vrouter-route-map-modify and

vrouter-route-map-show commands.
You can also specify metrics and route maps for vRouters configured as OSPF and BGP
vRouters.

CLI network-admin@Leaf1 > vrouter-modify

bgp-redist-static-route-map Specify the route map for BGP

vrouter route-map name redistribution of static routes.
bgp-redist-connected-route-map Specify the route map for BGP
vrouter route-map name redistribution of connected routes.
bgp-redist-ospf-route-map Specify the route map for BGP
vrouter route-map name redistribution of OSPF routes.

Pluribus Networks
www.pluribusnetworks.com
169
 ospf-redist-static-route-map Specify the route map for OSPF
vrouter route-map name redistribution of static routes.
ospf-redist-connected-route-map Specify the route map for OSPF
vrouter route-map name redistribution of connected routes.
ospf-redist-bgp-route-map Specify the route map for OSPF
vrouter route-map name redistribution of BGP routes.

Configuring Routing Information Protocol (RIP)

Routing Information Protocol (RIP) remains the oldest routing protocol and provides
networking information to routers. Routers need to know the available networks and the
distance required to reach it.
RIP consists of a distance vector protocol, and uses hop counts to determine distance and
destination. Every 30 seconds, RIP sends routing information to UDP port 50. If you
designate the router as default gateway, the router advertises by sending 0.0.0.0 with a
metric of 1.
Figure 3:I RIPTopology

1. Create vRouter1 on VLAN33:

CLI network-admin@switch > vrouter-create name vrouter1 fabricname-global

router-type hardware

You can also specify how Netvisor distributes RIP routes using the parameter,
rip-redistribute
static|connected|ospf|bgp.
2. Add network 10.16.33.0/24 to vrouter1:

170
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > vrouter-rip-add vrouter-name vrouter1 network
10.16.33.0/24 metric 2

3. Add network 10.16.35.0/24 to vrouter1:

CLI network-admin@switch > vrouter-rip-add vrouter-name vrouter1 network

10.16.55.0/24 metric 2

4. To view the configuration, use the vrouter-rip-show command. This displays all RIP
routes configured using the vrouter-rip-add command.
To view RIP routes not configured using the vrouter-rip-add command, use the
vrouter-rip-routes-show command.

Configuring Static Routes

vRouters forward packets using either routing information from route tables manually
configured or routing information calculated using dynamic routing algorithms.
Static routes define explicit paths between two vRouters and do not automatically updated.
When network changes occur, you have to reconfigure static routes. Static routes use less
bandwidth than dynamic routes.
Figure 5: Configuring a Static Route

In this example, you configure a static route on vRouter1 for the network, 172.16.10.10/24
with a gateway IP address, 172.16.20.1:

CLI network-admin@switch > vrouter-static-route-add vrouter-name vrouter1

network 172.16.10.10/24 gateway-ip 172.16.20.1

Pluribus Networks
www.pluribusnetworks.com
171
Support for Bidirectional Forwarding Detection on Static Routes
This version of Netvisor OS provides support for static route reachability detection using
Bidirectional Forwarding Detection (BFD) protocol. Netvisor installs a static route entry in the
Routing Information Database (RIB) only if BFD can communicate with the gateway. After
installation, BFD periodically monitors reachability and removes the route if connectivity is
lost.
When you configure a vRouter as a gateway, Netvisor OS sends out BFD hellos periodically to
specified neighbors. You configure static BFD sessions for this purpose.
Currently when you create a static route, Netvisor installs a route entry in RIB regardless of
the reachability of the gateway. By making static routes conditional over BFD, neighborship
formation alleviates this problem.
Netvisor OS now provides new parameters to configure static BFD sessions between a
Netvisor OS vRouter and any remote routers supporting BFD. The vRouter sends out periodic
BFD packets to these neighbors, so that the routers can determine if the Netvisor vRouter,
acting as a gateway, is alive or not.
If the BFD destination IP address matches with a static route gateway IP address, Netvisor
considers the static route BFD enabled. Netvisor installs the static route in the RIB only for
active BFD sessions. Netvisor creates the BFD sessions on a per gateway basis, in that
different static routes using the same gateway, use the same BFD session to determine
connectivity.
If the BFD session goes down, Netvisor removes all such static routes from the RIB. The
source address for the BFD session consists of the interface address of any vRouter interface
(vnic) or a loopback interface.
BFD timers can be specified for the vRouter interface or the loopback interface.

CLI Commands

vrouter-static-bfd-add Add a static BFD session

vrouter selector:
vrouter-name name-string Specify the name of the vRouter service
configuration to add BFD
the following static-bfd
arguments:
src-ip ip-address Specify the source IP address for the BFD session.
dst-ip ip-address Specify the destination IP address for the BFD
session.
type single-hop|multi-hop Specify the BFD type.

vrouter-static-bfd-remove Remove a static BFD session

vrouter selector:
vrouter-name name-string Specify the name of the vRouter service
configuration to add BFD.
the following static-bfd
arguments:
src-ip ip-address Specify the source IP address for the BFD session.
dst-ip ip-address Specify the destination IP address for the BFD
session.

172
Pluribus Networks
www.pluribusnetworks.com
vrouter-static-bfd-show Displays a static BFD session
vrouter selector:
vrouter-name name-string Displays the name of the vRouter service
configuration to add BFD
the following static-bfd
arguments:
src-ip ip-address Displays the source IP address for the BFD session
dst-ip ip-address Displays the destination IP address for the BFD
session
type single-hop|multi-hop BFD type

The source IP address is a vRouter interface (vnic) or a loopback interface.

CLI network-admin@switch > vrouter-static-bfd-show

vrouter-name src-ip dst-ip type

------------ -------------- --------- ----------
vr1 100.1.1.1 200.1.1.1 single-hop
vr1 100.1.1.1 201.1.1.1 multi-hop

The vrouter-static-route-show command displays if a route is BFD-enabled or not. If you

configure BFD, Netvisor displays the configured source IP for the session.
Example output of static routes configured with BFD:

CLI network-admin@switch > vrouter-static-route-show

vrouter-name network gateway-ip distance bfd bfd-src-ip

------------ ------------ ---------- -------- --- ---------
vr1 101.1.1.0/24 201.1.1.1 1 single-hop 100.1.1.1
vr1 102.1.1.0/24 202.1.1.1 1 multi-hop 100.1.1.1
vr1 103.1.1.0/24 203.1.1.1 1

In addition, the existing command, vrouter-bfd-neighbor-show displays any static BFD

neighbor information.
The existing commands, vrouter-interface-config-add and
vrouter-interface-config-modify can be used to configure BFD timers.

Adding IPv6 Link-Local Addresses for Static Routing

IPv6 link-local addresses have a specific prefix fe80::/10 and are relevant only on the local
link. This makes it possible to have the same link-local address on multiple IPv6 interfaces. If
you add a static route reachable through a link-local address, you must specify the outgoing
interface.
A new parameter now allows you to specify the interface for the static route:

CLI network-admin@Leaf1 > vrouter-static-route-add

vrouter-name name-string Specify the name of the vRouter service.

network ip-address Specify the IP subnet of the network that
you want to add a static router.

Pluribus Networks
www.pluribusnetworks.com
173
netmask netmask Specify the netmask of the IP subnet.
gateway-ip ip-address Specify the IP address of the gateway
that you want to route packets destined
for the network IP address.
bfd-dst-ip ip-address Specify the destination IP address for
BFD monitoring.
distance number Specifies the administrative distance in a
number from 0 to 255.
• 0 — Connected interface
• 1 — Static route
• 110 — OSPF
• 120 — RIP
• 200 — Internal BGP
interface vrouter interface nic Specify the vRouter interface for the
static route.

You remove or display the state using the commands, vrouter-static-route-remove, and
vrouter-static-route-show.

Configuring Multicast Listener Discovery (MLD)

In IPv4, Layer 2 switches use IGMP snooping to limit the flooding of multicast traffic by
dynamically configuring Layer 2 interfaces so that multicast traffic forwards only to those
interfaces associated with IP multicast address. In IPv6, MLD snooping performs a similar
function. With MLD snooping, Netvisor selectively forwards IPv6 multicast data to a list of
ports configured to receive the data, instead of flooding to all ports in a VLAN. MLD
constructs the list by snooping IPv6 multicast control packets.
IPv6 multicast routers use MLD to discover the presence of multicast listeners (nodes
configured to receive IPv6 multicast packets) on directly attached links and to discover which
multicast packets interest the neighboring nodes. MLD derives from the IGMP protocol. MLD
version 1 (MLDv1) is equivalent to IGMPv2, and MLD version 2 (MLDv2) is equivalent to
IGMPv3. MLD is a subprotocol of Internet Control Message Protocol version 6 (ICMPv6), and
MLD messages are a subset of ICMPv6 messages, identified in IPv6 packets by a preceding
Next Header value of 58.
The switch snoops on both MLDv1 and MLDv2 protocol packets and bridge IPv6 multicast
data based on destination IPv6 multicast MAC addresses. The switch can be configured to
perform MLD snooping and IGMP snooping simultaneously.
To display MLD routers on the network, use the mld-router-show command:

CLI network-admin@switch > mld-router-show

group-ip:
node-ip:
vlan:
port:
source:
node-type:

174
Pluribus Networks
www.pluribusnetworks.com
expires:
The show ouput displays the following information:
Multicast group IP address in IPv6 format
Host node IP address in IPv6 format
HostVLAN ID
Portnumber
Multicast traffic source IP address in IPv6 format
Node type as host or router
Expires as the ageout time
To display MLD group membership on the network, use the mld-show command:

CLI network-admin@switch > mld-show

group-ip:
node-ip:
vlan:
port:
source:
node-type:
expires:

The show ouput displays the following information:

Multicastgroup IP address in IPv6 format
Host node IP address in IPv6 format
Host VLAN ID
Port number
Multicast traffic source IP address in IPv6 format
Node type as host or router
Expires as the ageout time
To enable or disable MLD snooping or modify the scope, use the mld-snooping-modify
command:

CLI network-admin@switch > mld-snooping-modify

To modify the scope from fabric to local, use the following syntax:

CLI network-admin@switch > mld-snooping-modify scope fabric

To display information about MLD snooping, use the mld-snooping-show command.

Configuring an IGMP Querier IP Address

You can configure an IGMP querier IP address for a VLAN or as a global IGMP querier. The
IGMP querier sends IGMP General Query messages. on the network.
If you do not specify a querier IP address, then Netvisor uses 0.0.0.0 as the default value.
There can be an unique querier IP for each VLAN, or you can configure the same Querier IP
address for all the VLANs participating in IGMP snooping. The Querier IP address should have
a local scope and every switch should have a unique Querier IP address.

Pluribus Networks
www.pluribusnetworks.com
175
With a valid source IP address on IGMP Query packets, Netvisor adds the VLAN receiving the
Query to an IGMP Snoop switch list, and now reflects in the igmp-switches-show output and
the IGMP queries sent to the peer Switch as well. This solicits a report from the hosts
listening on the peer switch.
Use these Netvisor OS commands to configure an IGMP querier IP address.

igmp-querier-ip-modify Modify IGMP Querier IP configuration

querier-ip ip-address Specify the Snooping Querier IP address
vlans-on-querier-ip vlan-list Specify the VLAN map.

igmp-querier-ip-show Display IGMP Querier IP config

querier-ip is Specify the Snooping Querier IP address
vlans-on-querier-ip vlan-list VLAN MAP

MLD Snooping of IPv6 Neighbors

For MLD snooping, solicited nodes use the IPv6 address form of FF02::1:Fxx:xxxx, where
xx:xxxx consists of 24 bits of the host or VM MAC address. This allows 224 possible unique
groups and if you enable MLD snooping for IPv6 addresses, scaling the number of hosts and
VMs on a network can deplete the number of multicast groups. Therefore, the ability to
enable or disable MLD snooping for IPv6 addresses is necessary.
Note that a Neighbor Discovery Solicited Node uses the range of ff02::1:ff00:0/104
whereas a link-local address uses the range of ff02::/10. When you disable link-local
snooping, Netvisor determines MLD snooping of ff02::1:ff00:0/104 by the parameter
no-snoop-nd|snoop-nd for the command mld-snooping-modify.
CLI (network-admin@Spine1)>mld-snooping-modify
no-snoop-linklocal|snoop-linklocal no-snoop-nd|snoop-nd

By default, Netvisor disables MLD snooping. If you enable MLD snooping, snoop-linklocal
and snoop-nd are enabled.

Multicast Listener Discovery (MLD) Snooping per VLAN

MLD snooping provides support per individual VLANs. In addition to current global enable and
disable support, you can do the following using the mld-snooping-modify command:
Specify a range of VLANs for determining the use of which MLDv1 and MLDv2
messages. Netvisor identifies the two VLAN ranges as mutually exclusive. When a
VLAN configured for MDv2 receives a MLDv1 query, the MLDv2 query moves to a
VLAN configured with MLDv1 if only one multi-cast router exists for this VLAN. If the
same VLAN receives a MLDv2 message, Netvisor removes existing entries and the
supported version changes to MLDv2.
Specify VLAN ranges for snooping link-local addresses and ND Solicited Node
addresses. You must include these VLANs in one MLDv1 and MLDv2 VLAN list.

Command Options
Use these command options for the mld-snooping-modify command:

176
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > mld-snooping-modify

mld-snooping-modify Enables or disables Multicast Listener

Discovery (MLD) snooping
one or more of the following
options:
scope local|fabric Specify the MLDsnooping scope - fabric or
local
enable|disable Enable or disable MLD snooping
mldv1-vlans vlan-list Specify the VLANs to enable snooping and
use MLDv1 protocol. Default: none
mldv2-vlans vlan-list Specify the VLANs on which to enable
snooping and use MLDv2 protocol. Default 1
- 4092
snoop-linklocal-vlans vlan-list Allow snooping of link-local
groups(ff02::/16) on these vlans. Default 1
- 4092
snoop-nd-vlans vlan-list Allow snooping of ND SN Multicast
addresses (ff02::1:ff/104) on these vlans.
Default 1 - 4092

These command options replace these Netvisor OS versions earlier than 2.5.2 command
options:

Show Output
The mld-snooping-modify show format all command displays the following sample output:

CLI network-admin@switch > mld-snooping-show format all

switch: Name of Switch

enable: no
mldv1-vlans: none
mldv2-vlans: 1-4092
snoop-linklocal-vlans: 1-4092
snoop-nd-vlans: 1-4092
nvOS-managed-vlans: none
interop-v1-vlans: none
vlans: 1-4092

Creating MLD Static Sources and Static Groups

To determine how to forward multicast traffic, a switch with MLD snooping enabled maintains
information about the following interfaces in its multicast forwarding table:
Multicast-router interfaces —These interfaces lead toward multicast routers or MLD
queriers
Group-member interfaces —These interfaces lead toward hosts that are members
of multicast groups.
The switch learns about these interfaces by monitoring MLD traffic. If an interface receives
MLD queries, the switch adds the interface to the multicast forwarding table as a
multicast-router interface. If an interface receives membership reports for a multicast group,
the switch adds the interface to the multicast forwarding table as a group-member interface.

Pluribus Networks
www.pluribusnetworks.com
177
Table entries for interfaces that the switch learns about are subject to aging. For example, if
a learned multicast-router interface does not receive MLD queries within a certain interval,
the switch removes the entry for that interface from the multicast forwarding table.
You can create MLD static sources using IPv6 addresses and then create static groups using
the static sources.
To create an MLD static source for IPv6 address, ff02::1:ff11:111 as the group, and IPv6
2001:db8::2:1 as the source on VLAN 25, port 44-45, use the following syntax:

CLI network-admin@switch > mld-static-source-create source-ip 2001:db8::2:1

group-ip ff02::1:ff11:111 vlan 25 ports 44-45

The parameter, ports, is an optional parameter. You can delete an MLD static source, but you
cannot modify the parameters. To display MLD static sources, use the
mld-static-source-show command.
To create an MLD static group for IPv6 address, ff02::1:ff11:1111, on VLAN 25, ports 44-45,
use the mld-static-group-create command:

CLI network-admin@switch > mld-static-group-create group-ip ff02::1:ff11:1111

vlan 25 ports 44-45

You can delete an MLD static group, but you cannot modify the parameters. To display MLD
static groups, use the mld-static-group-show command.

Displaying MLD Statistics for a VLAN

To display MLD statistics for a VLAN, use the mld-stats-show command:

CLI network-admin@switch > mld-stats-show

switch:
vlan:
v1-queries:
v2-queries:
v1-member-reports:
v1-done-group:
v2-member-reports:
queries-sent:
drops:
ignored:

Configuring Virtual Router Redundancy Protocol

Virtual Router Redundancy Protocol (VRRP), an election protocol that enabling virtual routing
functions for a master or standby routing infrastructure for a given IP address. Netvisor
defines avirtual router by a virtual router identifier (VRID) and a virtual router IP address
(VIP). Netvisor restricts the scope of the virtual routers to a single VLAN.
VRRP provides information on the state of a virtual router, not the routes processed and
exchanged by the router. It increases the availability and reliability of routing paths by
automatic gateway selections on an IP subnetwork.

178
Pluribus Networks
www.pluribusnetworks.com
VRRP provides rapid transition from master to standby and from standby to master. The
master router sends advertisements every second. If the master VRRP advertisements are
not received within a window of time, three (3) seconds, then the standby virtual router
becomes the master virtual router and begins performing routing for the virtual router. If the
master router becomes active again, it can become the master again or allow the standby to
continue as the master router. The role depends on the value assigned to VRRP priority.
vRouter interfaces now support both IPv4 and IPv6 addresses.

Configuring VRRP Priority

VRRP Routers use a Priority value for master election. Netvisor uses a priority range for a
virtual router from 1 to 254. Netvisor uses 1 as the lowest priority and 254 as the highest
priority. The default value used by standby routers is 100. Higher values indicate higher
priority for the virtual router.

Configuring the VRRP ID

You configure the Virtual Router Identifier as a value between 1 and 255. There is no default
value.

Example Configuration
In this example, you have the following configurations on two switches (SW1 and SW2) on
the network:
VLAN 100 with IP address range 192.168.11.0/24
VNET with the name vrrp-router and scope fabric
1. On SW1, configure a vRouter:

CLI network-admin@switch > vrouter-create name vrrp-rtr1 vnet vrrp-router

router-type hardware enable

2. Add the first vRouter interface:

CLI network-admin@switch > vrouter-interface-add vrouter-name vrrp-rtr1 ip

192.168.11.3 netmask 24 vlan 100 if data

3. Use the vrouter-interface-show command to see the name of the interface:

CLI network-admin@switch > CLI network-admin@switch > vrouter-interface-show

format all layout vertical

vrouter-name: vrrp-rtr1
nic: eth0.100
ip: 192.168.11.3/24
assignment: static
mac: 66:0e:94:dd:18:c4
vlan: 100
vxlan: 0
if: data
alias-on:
exclusive: no
nic-config: enable
nic-state: up

Pluribus Networks
www.pluribusnetworks.com
179
4. Create the VRRP interface:

CLI network-admin@switch > CLI (server-switch)>vrouter-interface-add

vrouter-name vrrp-rtr1 ip 192.168.11.2 netmask 24 vlan 100 if data vrrp-id 10
vrrp-primary eth0.100 vrrp-priority 100

5. Create the vRouter and interfaces on SW2:

CLI network-admin@switch > CLI network-admin@switch > vrouter-create name

vrrp-rtr2 vnet vrrp-router router-type hardware dedicated-vnet-service

6. Add the vRouter interface:

CLI network-admin@switch > CLI network-admin@switch > vrouter-interface-add

vrouter-name vrrp-rtr2 ip 192.168.11.4 netmask 24 vlan 100 if data

7. Use the vrouter-interface-show command to see the name of the interface:

CLI network-admin@switch > CLI network-admin@switch > vrouter-interface-show

format all layout vertical

vrouter-name: vrrp-router2
nic: eth2.100
ip: 192.168.11.3/24
assignment: static
mac: 66:0e:94:21:a9:6c
vlan: 100
vxlan: 0
if: data
alias-on:
exclusive: no
nic-config: enable
nic-state: up

8. Create the VRRP interface:

CLI network-admin@switch > CLI network-admin@switch > vrouter-interface-add

vrouter-name vrrp-rtr2 ip 192.168.11.2 netmask 24 vlan 100 if data vrrp-id 10
vrrp-primary eth0.100 vrrp-priority 50

9. Display the information about the VRRP setup:

CLI network-admin@switch > CLI network-admin@switch > vrouter-interface-show

format all layout vertical

vrouter-name: vrrp-router1
nic: eth0.100
ip: 192.168.11.3/24
assignment: static
mac: 66:0e:94:dd:18:c4
vlan: 100
vxlan: 0
if: data
alias-on:
exclusive: no
nic-config: enable

180
Pluribus Networks
www.pluribusnetworks.com
nic-state: up
vrouter-name: vrrp-router1
nic: eth1.100
ip: 192.168.11.2/24
assignment: static
mac: 00:00:5e:00:01:0a
vlan: 100
vxlan: 0
if: data
alias-on:
exclusive: no
nic-config: enable
nic-state: up
vrrp-id: 10
vrrp-primary: eth1.100
vrrp-priority: 100
vrrp-state: master
vrouter-name: vrrp-router2
nic: eth3.100
ip: 192.168.11.4/24
assignment: static
mac: 66:0e:94:21:54:07
vlan: 100
vxlan: 0
if: data
alias-on:
exclusive: no
nic-config: enable
nic-state: up
vrouter-name: vrrp-router2
nic: eth3.100
ip: 192.168.11.2/24
assignment: static
mac: 00:00:5e:00:01:0a
vlan: 100
vxlan: 0
if: data
alias-on:
exclusive: no
nic-config: enable
nic-state: down
Pluribus Networks Configuration Guide
pluribusnetworks.com 87
vrrp-id: 10
vrrp-primary: eth3.100
vrrp-priority: 50
vrrp-state: slave

When you intentionally disable the VRRP interface, the slave interface becomes the master
interface:
vrouter-name: vrrp-router2
nic: eth3.100
ip: 192.168.11.1/24
assignment: static
mac: 00:00:5e:00:01:0a
vlan: 100

Pluribus Networks
www.pluribusnetworks.com
181
vxlan: 0
if: data
alias-on:
exclusive: no
nic-config: enable
nic-state: up
vrrp-id: 10
vrrp-primary: eth3.100
vrrp-priority: 50
vrrp-state: master

When you re-enable the VRRP interface, it becomes the master again, and the second
interface returns to the slave:
vrouter-name: vrrp-router2
nic: eth3.100
ip: 192.168.11.2/24
assignment: static
mac: 00:00:5e:00:01:0a
vlan: 100
vxlan: 0
if: data
alias-on:
exclusive: no
nic-config: enable
nic-state: down
vrrp-id: 10
vrrp-primary: eth3.100
vrrp-priority: 50
vrrp-state: slave

Layer 3 Table Validation

Layer 3 entries can become unsynchronized between the software table and the hardware
table when you modify routes while the routes update on the network. This feature adds
parameters to the l3-setting-modify command to automatically check Layer 3 entries:

CLI network-admin@switch > l3-setting-modify

one or more of the following

options:
aging-time seconds Specify the aging-time for Layer 3 entries, use 0
to disable aging.
convergence-time seconds Specify the unicast convergence time on bootup
(seconds)
l3-checker|no-l3-checker Specify if you want to check Layer 3 consistency
l3-checker-interval Specify the interval for Layer 3 consistency
duration: #d#h#m# checker
l3-checker-fix| Enable fixing of inconsistent entries
no-l3-checker-fix

Nevisor uses two commands to manually check and fix Layer 3 inconsistencies:

182
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > l3-check-fix

CLI network-admin@switch > l3-check-show

Active-Active VLAG with Link-State Layer 3 Unicast Protocols

Netvisor supports VLAG egress filtering rules for packets ingressing cluster links that prevent
packets from egressing VLAGs. In addition to the properties of control packets in unicast
routing protocols, such as OSPF, that set TTL=1, make it difficult to support routing protocols
from creatingLayer3 adjacencies over VLAGs. This feature allows unicast routing protocols
such as OSPF to form Layer3 adjacencies over VLAGs, by implementing a dual-forwarder
logic, while at the same time allowing routing control packets to be directed to the right
cluster switch. Netvisor enables by default, but you enable or disable it on a per vRouter
basis. When you enable active-active VLAG, the cluster vRouter implements special logic to
synchronize vRouter gateway MAC addresses as well as vrouter interface IP addresses with
the cluster peer.
When you configure the ports for active-active VLAG, you should use the
port-config-modify command and add the parameter jumbo to support jumbo frames on
the network.

CLI (network-admin@spine1)>vrouter-modify vrouter-name name-string

cluster-active-active-routing|no-cluster-active-active-routing

To display the configuration, use the vrouter-show command:

CLI (network-admin@spine1)>vrouter-show

switch...cluster-active-active-routing...
------ ---------------------
spine1 enable

Using an L3 Network to Establish the Netvisor Fabric

You can establish a Netvisor fabric over the management network or an in-band network
over Layer 2. Fabric over a Layer 3 network uses an existing or new Layer 3 network in order
to establish the Netvisor OS fabric. Currently Netvisor only supports the feature over a BGP
network. Before you can configure this feature on your network, make sure you configure
BGP routing on your switches.

Basic Concepts
In a simple topology with a network of two BGP neighbors, Netvisor specifies the BGP
neighbor information is specified, while using the two in-band networks to establish TCP
sessions between the two switches.
To establish a fabric communication between the two switches over the BGP network, you
must perform the following steps:

Pluribus Networks
www.pluribusnetworks.com
183
Figure 10:Example Network of Two BGP Neighbors Topology
1. Create a vrouter in each switch.
2. Set up BGP peering and redistribute routes so that they can reach each other’s in-band
networks.
3. Setup up global static routes using the local vrouter as the gateway. You need to do this
because the routes reside inside the vrouter and each switch needs to reach the other
switch in-band network.
4. Enable switch-B to join the fabric that was created by switch-A.

Creating a vRouter for Fabric Communication

Use the fabric-comm-vrouter-bgpcreate command to create a vRouter for fabric
communication. This command has the following options:
name: vrouter name
bgp-as: BGP As number of the vrouter
bgp-redistribute: (default connected)
bgp-max-paths: (optional)
bgp-ibgp-multipath: (optional)
interface to communicate with other router:
IP address/netmask
vlan - vlan of the interface, or
l3-port - layer 3 port to use
bgp neigbor info: (optional)
neighbor: IP address
remote-as: neighbor AS number

184
Pluribus Networks
www.pluribusnetworks.com
interface info to communicate with current switch inband: (optional)
IP address/netmask
fabric network:
network/netmask to reach inband network of the switch that created the fabric
bfd: To use BFD for rapid BGP peer loss detection
allowas-in:
Accept routes with own AS in AS_PATH
This is only required for cluster connections

This CLI performs the majority of the actions in Steps 1 - 4 in the above example using the
information provided in the CLI. However, it does not:
change the inband IP address
create the actual fabric
If the inband network uses a /30 netmask, then it automatically determines the vrouter
inband IP address from the main inband IP address. The switch creating the fabric (the root
switch) does not specify the fabric network, but does need to create a route to the other
switch inband network.
switch-A> fabric-create name myfabric
switch-A> fabric-comm-vrouter-bgp-create name vrouter-a bgp-nic-l3-port 10
bgp-as 65001 bgpnic-
ip 192.169.0.5/30 in-band-nic-ip 12.1.1.10/24 remote-as 65002 neighbor
192.169.0.6 bgpredistribute
connected
switch-A> fabric-in-band-network-create 12.1.2.0/30
switch-B> fabric-comm-vrouter-bgp-create name vrouter-b bgp-nic-l3-port 20
bgp-as 65002 bgpnic-
ip 192.169.0.6/30 in-band-nic-ip 12.1.2.10/24 remote-as 65001 neighbor
192.169.0.5
fabric-network 12.1.1.0/24 bgp-redistribute connected
switch-B> fabric-join switch-ip 12.1.1.1
Joined fabric myfabric. Restarting nvOS...

Pluribus Networks
www.pluribusnetworks.com
185
Multiple Neighbors over a Layer 3 Fabric
With more than two switches, the first BGP peer for each switch can be created using the
fabric-comm-vrouter-bgp-create CLI. You need to create the rest separately. Here is an
example with switch-C added to the previous topology .

Figure 5: An Example Fabric over Layer 3 Topology

The configuration for this proceed along the same lines as before. In addition, Netvisor
requires the following steps to bring switch-B into the fabric:
1. In switch-A, create BGP peering to swtich-C and create a static route to reach switch-C
inband network. The way to extend this beyond three switches is to use this extra step in
every switch that has more than one BGP peer:
switch-A> vrouter-interface-add vrouter-name vrouter-a ip 192.169.0.9/30
l3-port 11
switch-A> vrouter-bgp-add vrouter-name vrouter-a neighbor 192.169.0.10
remote-as 65003
switch-A> fabric-in-band-network-create 12.1.3.0/30

2. On switch-C, use the fabric-comm-vrouter-bgp-create command as before and con-

nect:

switch-C> fabric-comm-vrouter-bgp-create name vrouter-c bgp-nic-l3-port 21

bgp-as 65003 bgpnic-
ip 192.169.0.10/30 in-band-nic-ip 12.1.3.10/24 remote-as 65001 neighbor
192.169.0.9
fabric-network 12.1.1.0/24 bgp-redistribute connected
switch-C> fabric-join switch-ip 12.1.1.1
Joined fabric myfabric. Restarting nvOS...

186
Pluribus Networks
www.pluribusnetworks.com
Support for Bidirectional Forwarding Detection (BFD) and Static Routes
Currently when you create a static route, Netvisor OS installs a route entry in the routing
information database (RIB) regardless of the reachability of the gateway. When you
configure static routes conditionally over BFD neighbor-ship formation, Netvisor OS alleviates
this issue.
Netvisor OS supports static route reachability detection by using BFD protocol, and a static
route entry is installed in the RIB only if BFD is able to communicate with the gateway. After
installation, BFD periodically monitors reachability and removes the route if connectivity is
interrupted. The Pluribus vRouter acts as a gateway, and sends out BFD hellos periodically to
specified neighbors. You can configure static BFD sessions.
Netvisor OS allows you to configure static BFD sessions between a vRouter and any remote
routers supporting BFD. The vRouter sends out periodic BFD packets to these neighbors, so
that they can determine if PN router, acting as a gateway, is alive or not.
If the BFD destination IP address matches a static route gateway IP address, Netvisor OS
considers the static route as BFD enabled. This means the static route is installed in the RIB
only if the BFD session is up. Note that the BFD sessions are per gateway, and different static
routes with the same gateway IP address use the same BFD session to determine
connectivity. If the BFD session goes down, Netvisor removes all static routes from the RIB.
The source IP address for the BFD session can be the interface address of any vRouter
interface or a loopback interface. BFD timers can be specified for the vRouter interface or the
loopback interface.
Use the following new commands to configure BFD on static routes:

CLI network-admin@switch > vrouter-static-bfd-add

CLI network-admin@switch > vrouter-static-bfd-remove

CLI network-admin@switch > vrouter-static-bfd-show

There are new parameters for vrouter-interface-config-add, -modify, and -show:

bfd-interval milliseconds Specify the transmission interval in milliseconds. The

default value is 750ms with a range of 200 to 3000
milliseconds.
bfd-min-rx milliseconds Specify the transmission interval in milliseconds. The
default value is 500ms with a range of 200 to 3000
milliseconds.
bfd-multiplier integer Specify the BFD detection multiplier from 1 to 20. The
default value is 3.

Support for Policy-based Routing

Policy-based Routing (PBR) enables flexible packet forwarding and routing through user
defined policies. Unlike traditional routing based on destination IP address only, PBR allows
you to define routes based on other parameters such as source and destination IP addresses,
protocol, or souce and destination port numbers.
Policy-based routes can match packets based on the following criteria:
All Layer 4 and Layer 3 fields similar to those in vFlow configuraions.
Policy based routes are higher priority than static and dynamic routes.

Pluribus Networks
www.pluribusnetworks.com
187
 If no match or next-hop is not resolved, then traffic is dropped until the next-hop is
resolved.
You configure PBR using vFlow commands. Internally, policy routing of the packets uses a
vFlow entry. Netvisor creates PBR vFlow entries in a new vFlow table, System-L3-L4-PBR.
To enable PBR, use the following command:

CLI network-admin@Leaf1 > system-settings-modify policy-based-routing

To disable PBR, use the following command:

CLI network-admin@Leaf1 > system-settings-modify no-policy-based-routing

To display the vFlow table, use the following command:

CLI network-admin@Leaf1 > vflow-table-show

switch name flow-max flow-used flow-tbs-slices

capability flow-profile
-------- -------------------- -------- --------- --------------- ----------
--- ----------------
Spine1 System-L3-L4-PBR-1-0 set-metada
ta system=>PBR Table

Now you configure a vFlow for the routing policy, using the following syntax:

CLI network-admin@Leaf1 > vflow-create name name-string vrouter-name

name-string scope local next-hop-ip gateway-ip-address table-name
System-L3-L4-PBR-1-0

You can only specify the scope as local.

Cluster Active-Active Routing Support for IPv6 Addresses

With Cluster Active-Active Routing, the router peers act as a proxy for the other router
interface DMACs. But if the two routers try to communicate with each other, packets do not
route correctly on the network.
For two vRouters in a cluster pair with cluster active-active routing enabled, when a switch
receives a packet with the destination MAC address of the cluster peer, Netvisor attempts to
route the packet. For neighbor discovery, IPv6 routing uses Layer 3 packets instead of Layer
2 for ARP communication.
A vRouter sends a neighbor solicitation packet and the peer vRouter responds with neighbor
advertisements containing the destination IP address of the other peer router. On the first
peer vRouter, the packet destination MAC matches the second peer MAC and the switch
routes this packet back to the CPU port of the same switch. Netvisor adds two copies of a
neighbor advertisement packet on the internal port of the second switch and the Netvisor
neighbor advertisement packet does not return to the first switch.

188
Pluribus Networks
www.pluribusnetworks.com
To alleviate this issue, Netvisor adds a host route in the hardware for the link-local IPv6
vRouter peer.Now, when the first cluster peer sends a unicast packet with a link-local
destination IPv6 address to the second cluster peer, the first cluster peer has this host route
entry which properly routes the packet.

Support for PIM Source Specific Multicast (PIM-SSM) Forwarding

Protocol Independent Multicast (PIM) distributes multicast data using routes gathered by
other protocols.
PIM builds and maintains multicast routing trees using reverse path forwarding (RPF) based
on a unicast routing table. PIM can use routing tables consisting of OSPF, BGP, RIP, and static
routes. Each host (senders and receivers) is associated with a Designated Router (DR) that
acts for all directly connected hosts in PIM-SSM transactions.
Netvisor has the following enhancements:
A PIM-SSM vRouter acts as first hop designated router connected to multicast
receivers.
Netvisor performs multicast forwarding in hardware.
Netvisor scales for multicast routes.
Netvisor ensures PIM protocol operation by mapping PIM protocol messages to proper
CoS queues.
Netvisor OS supports PIM-SSM in a cluster environment by providing active-active multicast
forwarding.
Netvisor OS also has enhanced support for mapping IGMPv1 and IGMPv2 requests to specific
SSM ranges.
New commands support extended SSM IP addresses. The default IP address is 232.0.0.0/8.

CLI network-admin@Leaf1 > vrouter-pim-ssm-range-add

vrouter-name name-string Specify the name of the vRouter.

Add the following PIM-SSM arguments:
id id Specify the ID as a number between 1 and 255.
group ip-address Specify a group PIM-SSM range IP address. The
default is 232.0.0.0
netmask netmask Specify the netmask. The default is 8.

CLI network-admin@Leaf1 > vrouter-pim-ssm-range-remove

vrouter-name name-string Specify the name of the vRouter.

Add the following PIM-SSM arguments:
id id Specify the ID as a number between 1 and 255.
group ip-address Specify group PIM-SSM range IP address. The
default is 232.0.0.0

Pluribus Networks
www.pluribusnetworks.com
189
CLI network-admin@Leaf1 > er-pim-ssm-range-show

vrouter-name name-string Displays the name of the vRouter.

Add the following PIM-SSM arguments:
id id Displays the ID as a number between 1 and 255.
group ip-address Displays a group IP address. The default is
232.0.0.0
netmask netmask Displays the netmask. The default is 8.

Configuring PIM-SSM Mapping

SSM mapping introduces a means for the last hop router to discover sources sending to
groups. When SSM mapping is configured, if a router receives an IGMPv1 or IGMPv2
membership report for a particular group G, the router translates this report into one or more
(S, G) channel memberships for the well-known sources associated with this group.
The following new commands support this feature:

CLI network-admin@Leaf1 > vrouter-pim-ssm-map-add

vrouter-name name-string Displays the name of the vRouter.

Add the following PIM-SSM Mapping arguments:
id id Displays the ID as a number between 1 and 255.
group ip-address Displays a group IP address. The default is
232.0.0.0.
netmask netmask Displays the netmask. The default is 8.

CLI network-admin@Leaf1 > vrouter-pim-ssm-map-remove

vrouter-name name-string Displays the name of the vRouter.

Add the following PIM-SSM Mapping arguments:
id id Displays the ID as a number between 1 and 255.
group ip-address Displays a group IP address. The default is
232.0.0.0.
netmask netmask Displays the netmask. The default is 8.

CLI network-admin@Leaf1 > vrouter-pim-ssm-map-add

vrouter-name name-string Displays the name of the vRouter.

Add the following PIM-SSM Mapping arguments:
id id Displays the ID as a number between 1 and 255.

190
Pluribus Networks
www.pluribusnetworks.com
 group ip-address Displays a group IP address. The default is
232.0.0.0.
netmask netmask Displays the netmask. The default is 8.

Configuring PIM-SSM on vRouter Interfaces

In previous releases, when you configured PIM-SSM on a vRouter, Netvisor applied the
configuration to all vRouter interfaces. Now you can configure PIM-SSM on separate vRouter
interfaces using the following syntax:

CLI network-admin@Leaf1 > vrouter-interface-add

pim|no-pim Specify if the vRouter interface is a PIM

interface.
pim-dr-priority integer Specify the direct router (DR) priority as an
integer between 1 and 4294967295.
Netvisor selects the vRouter interface with
higher DR priority as the designated router.
By default, all interfaces are priority 1. In
case of same priority, Netvisor selects the
vRouter interface with highest network
address as DR.
pim-cluster|no-pim-cluster Specify if you want to provide alternative
routing for Layer 3 attached sources for
leaf cluster routing configuration. It allows
for traffic to failover when a unicast path to
the Spine fails.

You can modify or display these parameters using the commands

vrouter-interface-modify and vrouter-interface-show.

Support for Active-Active Multicast Routing with VRRP

In the VRRP Spine and Leaf configuration, multi-cast traffic for a source, group, or VLAN, for
example <S,G, VLAN> tuple can drop traffic since only one of the multicast routers forwards
traffic while the other multicast router is not the DR router for the subnet.
To resolve this issue, configure active-active multicast forwarding to send traffic correctly. In
the VRRP Spine and Leaf topology, you must configure both multicast routers as a designated
router (DR). Designated Routers are the multicast routers designated to forward traffic to a
specific subnet such as a VLAN. By enabling both routers in the Spine and Leaf VRRP
topology as DR routers, all multicast routers are aware of the multicast receivers whatever
the location in the Spine and Leaf topology.
Configuring active-active routing is simple and dynamic by enabling on a per interface basis.
If the interface is a VRRP primary interface, then Netvisor enables active-active multicast
forwarding on this interface by automatically designating the interface as the designated
router for the Layer 3 subnet, no matter how PIM resolves the interface through the DR
election process.

Pluribus Networks
www.pluribusnetworks.com
191
Group membership does not ensure traffic is not black-holed. For a given <S,G,VLAN>,
Netvisor routes traffic to the outgoing set of VLANs without causing a PIM assert on the peer
or local router. Netvisor routes traffic only to local nodes, and the route excludes cluster link
ports on the outgoing interface list.
The following command displays the interfaces of a PIM multicast router:

CLI network-admin@Leaf1 > vrouter-pim-interface-show

vrouter-name name local-address index if-state

------------ ------------- ------------- ----- ----------
vr0 eth0.110 10.16.110.1 0 dr,no-nbrs
vr0 eth0.111 10.16.111.1 1 dr,no-nbrs
vr0 eth0.112 10.16.112.1 2 pim
vr0 eth0.113 10.16.113.1 3 dr,no-nbrs
vr0 register_vif0 10.16.110.1 4
vr1 lo0:1 10.16.116.1 0 dr,no-nbrs
vr1 eth1.112 10.16.112.2 1 dr,pim
vr1 eth0.114 10.16.114.1 2 pim
vr1 register_vif0 10.16.116.1 3
vr2 eth1.113 10.16.113.2 0 dr,no-nbrs
vr2 eth1.114 10.16.114.2 1 dr,no-nbrs
vr2 eth0.115 10.16.115.1 2 dr,no-nbrs
vr2 register_vif0 10.16.113.2 3

To display PIM neighbors, use the following command:

CLI network-admin@Leaf1 > vrouter-pim-neighbor-show

vrouter-name interface address neighbor

------------ --------- ----------- -----------
vr0 eth0.112 10.16.112.1 10.16.112.2
vr0 eth0.113 10.16.113.1 10.16.113.2
vr1 eth1.112 10.16.112.2 10.16.112.1
vr1 eth0.114 10.16.114.1 10.16.114.2
vr2 eth1.113 10.16.113.2 10.16.113.1
vr2 eth1.114 10.16.114.2 10.16.114.1

To display the routing state for router or group, use the following command:

CLI network-admin@Leaf1 > vrouter-pim-join-show 224.0.1.0

vrouter-name source group incoming-if outgoing-ifs joined-ifs

------------ ------------- --------- ------------- -----------------
----------
vr0 :: 224.0.1.0 register_vif0 eth0.113 eth0.113
vr0 10.16.110.101 224.0.1.0 eth0.110 eth0.113 eth0.113
vr4 :: 224.0.1.0 eth0.125,eth0.127

pruned-ifs asserted-ifs leaf-ifs mrt-state

---------- ------------ ----------------- ------------
wc,rp
spt,cache,sg
eth0.125,eth0.127 wc,rp

192
Pluribus Networks
www.pluribusnetworks.com
Configuring the Hello Interval for PIM-SSM
You can configure the PIM-SSM Hello Interval using the following command:

CLI network-admin@Leaf1 > vrouter-pim-config-modify

vrouter-name name-string Specify the vRouter name to modify for PIM-SSM.

query-interval seconds Specify the query interval from 1 to 300 seconds.
hello-interval seconds Specify the hello interval from 1 to 300 seconds.
querier-timeout seconds Specify the querier timeout from 1 to 900 seconds.

Configuring the Designated Router Priority for PIM-SSM

In a PIM SSM-configured network, a host subscribes to an SSM channel through IGMPv3,
announcing a desire to join group G and source S. The directly connected PIM-SSM vRouter,
the Designated Router (DR) for the receiver, sends an (S,G) join message to a Reverse Path
Forwarding (RPF) neighbor for the source.
To configure the DR for PIM, SSM, use the following command:

CLI network-admin@Leaf1 > vrouter-interface-add

pim-dr-priority integer Specify the direct router (DR) priority as an integer

between 1 and 4294967295. Netvisor selects the
vRouter interface with higher DR priority as the
designated router.
By default, all interfaces are priority 1. In case of
same priority, Netvisor selects the vRouter interface
with highest network address as DR.

Virtual Routing and Forwarding (VRF) Support

Netvisor OS supports VRF (virtual routing and forwarding instances) to maintain Layer 3
isolation. VRFs are created without a vRouter and do not support running any routing
protocols within the VRF. Locally on each node, for each active VRF instance, hardware VRID
is allocated to provide Layer 3 isolation. VRFs provides the capability to route between
connected networks by leveraging the Netvisor OS vPort database within the fabric. You
configure VRF and an anycast gateway subnets to provide the distributed routing capability
for tenant endpoints. The distributed routing capability hosted on each leaf node avoids hair
pinning traffic to the centralized vRouter.
Netvisor OS supports anycast gateway routing using a virtual MAC address, anycast gateway
MAC address, which is associated with the subnet anycast gateway IP address. Netvisor OS
provides a default fabric-wide anycast gateway MAC address, and it is also configurable.
Since VRF supports connected networks only, each VRF is provided with a configurable option
of VRF gateway which installs a default route to provide connectivity to subnets outside of
VRF. For redundancy purposes, two VRF default gateways can be configured per leaf node.
Currently VRF only support IPv4 routing.
Netvisor OS assigns the anycast gateway MAC address to VRF from the MAC address in
fabric-anycast-gatway-show output. You can modify the MAC address using the
fabric-anycast-gateway-modify command.
The default MAC address for the anycast gateway is 64:0e:94:40:00:02.

Pluribus Networks
www.pluribusnetworks.com
193
Configuring VRF and Distributed Routing with an Anycast Gateway

The following commands are used to configure VRF:

CLI network-admin@Leaf1 > vrf-create

name name-string Specify a name for the VRF.

vnet vnet-name Specify the name of the VNET to assign the VRF. If
you only have a global VNET configured, omit this
parameter.
scope Specify the scope for the VRF.
local|cluster|fabric
vrf-gw ip-address Specify the gateway IP address.
vrf-gw2 ip-address Specify the second gateway IP address.

CLI network-admin@Leaf1 > vrf-delete

name name-string Specify a name for the VRF.

vnet vnet-name Specify the name of the VNET to assigned the VRF.

CLI network-admin@Leaf1 > vrf-modify

name name-string Specify a name for the VRF.

vnet vnet-name Specify the name of the VNET to assign the VRF.
scope Specify the scope for the VRF.
local|cluster|fabric
vrf-gw ip-address Specify the gateway IP address.
vrf-gw2 ip-address Specify the second gateway IP address.

CLI network-admin@Leaf1 > vrf-show

name name-string Displays the name of the VRF.

vnet vnet-name Displays the name of the VNET assigned the VRF.
scope Displays the scope of the VRF.
local|cluster|fabric
vrf-gw ip-address Displays the gateway IP address.
vrf-gw2 ip-address Displays the second gateway IP address.

The following commands configure the subnet:

CLI network-admin@Leaf1 > subnet-create

name name-string Specify the name of the subnet.

scope Specify the scope for the VRF.
local|cluster|fabric

194
Pluribus Networks
www.pluribusnetworks.com
 vnet vnet-name Specify the name of the VNET to assign the VRF.
vlan vlan-id Specify the VLAN ID to assign to the subnet.
vxlan vxlan-id Specify the VXLAN ID to assign to the subnet.
network ip-address Specify the network IP address.
netmask netmask Specify the netmask for the IP address.
vrf name-string Specify the VRF to assign the subnet.
anycast-gw-ip ip-address Specify the anycast gateway IP address.

CLI network-admin@Leaf1 > subnet-delete

name name-string Specify the name of the subnet.

vnet vnet-name Specify the name of the VNET to assign the VRF.
vrf name-string Specify the VRF to assign the subnet.

CLI network-admin@Leaf1 > subnet-modify

name name-string Specify the name of the subnet.

scope Specify the scope for the VRF.
local|cluster|fabric

CLI network-admin@Leaf1 > subnet-show

name name-string Displays the name of the subnet.

scope Displays the scope for the VRF.
local|cluster|fabric
vnet vnet-name Displays the name of the VNET to assign the VRF.
vlan vlan-id Displays the VLAN ID to assign to the subnet.
vxlan vxlan-id Displays the VXLAN ID to assign to the subnet.
network ip-address Displays the network IP address.
netmask netmask Displays the netmask for the IP address.
vrf name-string Displays the VRF to assign the subnet.
anycast-gw-ip ip-address Displays the anycast gateway IP address.
state init|ok|vxlan not Displays the subnet state.
found|vxlan deactivated|
subnet is not installed
in hw
hw-state|no-hw-state Displays if there is a hardware state present.

The following commands allow you to modify and display anycast gateway information on the
fabric:

Pluribus Networks
www.pluribusnetworks.com
195
CLI network-admin@Leaf1 > fabric-anycast-mac-modify

mac mac-address Modify the MAC address for anycast. The default
MAC address is 64:0e:94:40:00:02.

CLI network-admin@Leaf1 > fabric-anycast-mac-show

mac: 64:0e:94:40:00:02

Example Configuration
To add VRF to all switches installed on the network, use the following syntax:

CLI network-admin@Leaf1 > vrf-create name BLUE vnet coke scope

[local|fabric|cluster vrf-gw1 100.1.1.1 vrf-gw2 100.1.1.2

CLI network-admin@Leaf1 > subnet-create name subnet-1 scope

[local|fabric|cluster] vnet coke vxlan 10 network 10.0.10.0/24 vrf BLUE
anycast-gateway-ip 10.0.10.1

196
Pluribus Networks
www.pluribusnetworks.com
Configuring Virtual Networks

Informational Note: Netvisor ONE does not supported on Dell

platforms. Netvisor uses a global VNET to configure features that require
a VNET.

 Implementing Virtual Networks

 VNET High Availability (HA)

Implementing Virtual Networks

Netvisor uses Virtual Networks (VNETs), an abstract network resource realized across a fabric
of Pluribus Networks switches. Using VNETs, you can segregate a physical fabric into many
logical networks, each with separate resources, network services, and Quality of Service
(QoS) guarantees. A VNET allows you to completely separate all traffic in one VNET from the
traffic of other VNETs.
Figure 1:Using VNETs with Netvisor

Pluribus Networks
www.pluribusnetworks.com
197
Each VNET has a single point of management. As the fabric administrator, you can create
VNETs and assign ownership of each VNET to individuals with responsibility for managing
those resources. You can create separate user names and passwords for each VNET manager.
Using the separate VNET administration credentials, the VNET admin can use Secure Shell
(SSH) to connect to the VNET manager and access a subset of the Netvisor CLI commands to
manage that VNET. This way, multiple tenants can share a fabric with each managing a VNET
with security, traffic, and resource protection from other VNETs.
VNETs can create very flexible and complex network architectures. For example, a Pluribus
Networks switch, or a fabric of switches, can be used to create multiple tenant environments
in an OpenStack deployment. In Figure 1 Using VNETs with Netvisor, the diagram
displays three VNETs, each with a management interface and a data interface.Netvisor
assigns each VNET an IP address pool used for DHCP assignment of IP addresses to each
node, server, or OS component.
Underlying each VNET lies the VNET manager. Each VNET manager runs in a zone. When you
create services for a VNET, the services occupy the same zone on a switch. Netvisor
designates a shared service and created by default when creating services. However, each
zone can only support a single instance of a service. If you need a second service instance for
a VNET, then it needs to occupy a separate zone. Netvisor designates the zone as a dedicated
service. In most cases, you can create services as shared unless you specifically want to
create a dedicated service.
When you create a fabric, Netvisor automatically creates a VNET with the name
fabric-name-global. This VNET owns all resources within the fabric, and as you create new
VNETs, resources move from the default VNET to the new VNETs. Global services remain in
the default VNET unless assigned specifically to a VNET.

Specifying the Type of VNET Interface

The mgmt, data, and span keywords used in different commands specify the path used to
connect to the network service. For example, to specify an out-of-band connection to a
management interface of a VNET, the interface specifies the mgmt keyword. If you require
in-band access to the management interface of the VNET, then use the data or span
keywords in the specific command. The keywords, data and span, are essentially equivalent
but apply to two separate paths. To maximize throughput between the server and the switch
components, Pluribus Networks recommends using both. The data keyword applies to port
65, and the span keyword applies to port 66.
Each VNET can have one or more isolating zones and Netvisor applies network services to
each zone. Network services use a separate zone or share the zone with the VNET manager
which is the zone that the VNET user logs into to manage the VNET. In shared zones, the
network interfaces are available to all network services in the shared zones, regardless of the
service that created the network interface.

Informational Note: This is an important concept as you can use service

commands such as vnet-manager-interface-add to add interfaces to a VNET. If you
want the service to be specific to a VNET as a dedicated service, then add the
interfaces using the exclusive|no-exclusive parameters.

198
Pluribus Networks
www.pluribusnetworks.com
Creating a Virtual Network (VNET)
To separate resources, including switch ports, IP addresses, VLANs, and VXLANs, into
separate management spaces, create a VNET and place the resources in the VNET. Then
configure a separate VNET administrator to manage the network.

Informational Note: You cannot create another VNET inside of a VNET.

The switch has no performance impact when you send network traffic through a VNET.
Netvisor switches packets in the hardware with full line-rate bandwidth and the same latency
even if the packets reside on a VNET or not. But, the VNET allows you to provide different
Service Level Agreements (SLAs) to each VNET when multiple VNETs exist on a physical
switch and no resource contention based on traffic loads.

VNET High Availability (HA)

VNET HA provides high availability for switch access through a VNET manager. The VNET
manager is a zone, typically with one IP interface, allows a VNET administrator to log into
and administer a VNET using the CLI or a REST API.
Netvisor provides HA functionality by allowing you to create multiple VNET managers.
Netvisor enables access to the VNET managers when you add either standard or VRRP VIP
interfaces to the VNET managers.
Without VNET HA, VNET administrators have only a single point of access where the VNET
manager zone runs on a particular switch. If that switch fails, the administrator cannot log
into the fabric and administer the VNET.
This feature now provides the following:
Create or delete a VNET manager zone.
The vnet-manager-interface command now accepts VRRP interfaces. This allows you
to create VRRP interfaces on VNET manager zones
The vnet-create command now provides an option to not create a VNET manager zone
when creating a VNET
Copy and paste SSH host keys used on different VNET managers. You need to perform
this task when using a VRRP VIP to access the VNET managers to avoid SSH host key
violations if the VIP fails over to the standby VNET manager. Install these keys on the
client machine used to connect to the VNET managers.

Creating a VNET Manager Zone

To create additional VNET manager zones, use the vnet-manager-create command.

CLI network-admin@switch > vnet-manager-create name name-string vnet vnet-name

[disable|enable][location fabric-node name][storage-pool storage-pool name]

vnet-manager-create Creates a VNET manager

name name-string Specify the name of service
configuration.
vnet vnet-name Specify the VNET assigned to the
service.

Pluribus Networks
www.pluribusnetworks.com
199
any of the following options:
disable|enable Specify to enable or disable the
service.
location fabric-node name Specify the location of the service.
storage-pool storage-pool name Specify the storage pool assigned to
the service.

Deleting a VNET Manager Zone

To delete a VNET manager zone, use the vnet-manager-delete command.
vnet-manager-delete
vnet-manager-delete Deletes a VNET manager.
name name-string Specify the name of service
configuration.

Copying and Pasting SSH Keys

To output SSH host keys to copy and paste to ~/.ssh/known_hosts file of the client host,
use the vnet-manager-ssh-host-key-show command. This allows you to SSH to any VNET
manager zone and avoid issues with invalid key hosts.

CLI network-admin@switch > vnet-manager-ssh-host-key-show [vnet vnet name]

vnet-manager-ssh-host-key-show Displays the VNET Manager host keys

to copy and past to
~/.ssh/known_hosts.
name name-string Displays the name of service
configuration.

VNET Manager Command Options

This feature uses a new option [no-]create-vnet-mgr] which controls whether or not to
create a VNET manager. Netvisor creates a VNET manager by default.

CLI network-admin@switch > vnet-create name-string scope local|cluster|fabric

create-vnet|no-create-vnet-mgr

vnet-create Creates a virtual network (VNET)

name name-string Specify the VNET name.
scope local|cluster|fabric Specify the VNET scope as local,
cluster, or fabric.
create-vnet-mgr|no-create-vnet-mgr Create or not create a VNET manager
service.

VRRP Interfaces Option

This feature now accepts options for VRRP interfaces. This allows you to create VRRP
interfaces on VNET manager zones.

CLI network-admin@switch > vnet-manager-interface-add vnet-manager-name

name-string [vrrp-id][vrrp-primary vrrp-primary-string][vrrp-priorty
0..254][vrrp-ad-int]

200
Pluribus Networks
www.pluribusnetworks.com
vnet-manager-interface-add Adds an interface to a VNET manager.
vnet-manager-name name-string Specify the name of service
configuration.
vrrp-id 0..255 Specify the ID assigned to VRRP.
vrrp-primary vrrp-primary-string Specify the VRRP primary interface.
vrrp-priority 0..254 Specify the VRRP priority for the
interface.
vrrp-adv-int 10..40950 Specify the VRRP Advertisement
Interval in mseconds. The minimum
interval is 10ms and the maximum
interval is 40950ms. The default
interval is 1000ms.

Pluribus Networks
www.pluribusnetworks.com
201
Configuring Network Security
 Creating and Implementing Access Control Lists (ACLs)
 MAC ACLs
 IP ACLs
 Support for DHCP Snooping
 Support for Router Advertisement (RA) Guard

Creating and Implementing Access Control Lists (ACLs)

Access Control Lists (ACLs) allow you to configure basic traffic filtering for IP addresses and
MAC addresses. The ACL controls if the network forwards or blocks routed packets. The
switch examines the packet and then determines if the packet forwards or drops based on
the criteria configured in the ACLs. Netvisor supports Layer 2 (MAC) or Layer 3 (IP) ACLs.
ACL criteria can be based on source or destination addresses or the protocol type. Netvisor
supports UDP, TCP, IGMP, and IP protocols.
You can use ACLs to restrict contents of routing updates or provide traffic flow control. ACLs
can allow one host to access part of your network and prevent another host from accessing
the same area. You can also use ACLs to decide what types of traffic are forwarded or
blocked.
If you need more background on ACLs and using them on your network, refer to the many
networking resources available.

MAC ACLs
Using MAC ACLs to Deny Network Traffic
Create ACLs based on MAC addresses to deny network traffic from a specific source. MAC
addresses are Layer 2 protocols and most often assigned by the hardware manufacturer.
Figure 1 MAC ACL Blocking Access shows an example of a MAC address and Ethernet type to
block from the network.

202
Pluribus Networks
www.pluribusnetworks.com
Figure 1: MAC ACL Blocking Access

Configuring a MAC ACL to Deny Network Traffic

To deny IPv4 network traffic from MAC address, 01:80:c2:00:00:0X, for the scope fabric,
create the MAC ACL, deny-MAC, using the following syntax:

CLI network-admin@switch > acl-mac-create name deny-mac action deny src-mac

01:80:c2:00:00:0X ether-type ipv4 scope fabric

To review the configuration, use the acl-mac-show command:

CLI network-admin@switch > acl-mac-show name deny-mac layout vertical

name: deny-mac
id: b000015:12
action: deny
src-mac: 01:80:c2:00:00:0X
dst-mac: 00:00:00:00:00:00
dst-mac-mask: aa:aa:aa:aa:aa:aa
ether-type: ipv4
vlan: 0
scope: fabric
port: 0

Pluribus Networks
www.pluribusnetworks.com
203
Using MAC ACLs to Allow Network Traffic
So now that you have blocked the MAC address, reverse the scenario and allow IPv4 network
traffic from the MAC address to the network.
Figure 2:MAC ACL Allowing Access

See Configuring a MAC ACL to Allow Network Traffic to review the example configuration.

Configuring a MAC ACL to Allow Network Traffic

To allow IPv4 network traffic from MAC address, 01:80:c2:00:00:0X, for the scope fabric,
create the MAC ACL, allow-MAC, using the following syntax:

CLI network-admin@switch > acl-mac-create name allow-mac action permit src-mac

01:80:c2:00:00:0X ether-type ipv4 scope fabric

To review the configuration, use the acl-mac-show command:

204
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > acl-mac-show name deny-mac layout vertical

name: deny-mac
id: b000015:12
action: deny
src-mac: 01:80:c2:00:00:0X
dst-mac: 00:00:00:00:00:00
dst-mac-mask: aa:aa:aa:aa:aa:aa
ether-type: ipv4
vlan: 0
scope: fabric
port: 0

To delete the ACL configuration, use the acl-mac-delete command.

To modify the ACL configuration, use the acl-mac-modify command.

Configuring a MAC ACL to Deny Network Traffic

To deny IPv4 network traffic from MAC address, 01:80:c2:00:00:0X, for the scope fabric,
create the MAC ACL, deny-MAC, using the following syntax:

CLI network-admin@switch > acl-mac-create name deny-mac action deny src-mac

01:80:c2:00:00:0X ether-type ipv4 scope fabric

To review the configuration, use the acl-mac-show command:

CLI network-admin@switch > acl-mac-show name deny-mac layout vertical

name: deny-mac
id: b000015:12
action: deny
src-mac: 01:80:c2:00:00:0X
dst-mac: 00:00:00:00:00:00
dst-mac-mask: aa:aa:aa:aa:aa:aa
ether-type: ipv4
vlan: 0
scope: fabric
port: 0

IP ACLs
Using a Deny IP ACL to Block Network Traffic
This example displays a network with a Finance server on one part of the network, and an
Engineering server on another part. You want to block the Engineering server from the
Finance server in order to protect company sensitive information. See Configuring an
Internal Deny ACL to review the configuration sample.

Pluribus Networks
www.pluribusnetworks.com
205
Figure 1: Network Example - IP ACL for Internal Servers

Or you may discover that an external source is attempting to access your network, and ping
your servers for IP addresses. You can use an ACL to block the specific source using an IP
ACL.

206
Pluribus Networks
www.pluribusnetworks.com
Figure 2:IP ACL Blocking External Access

See Configuring an External Deny ACL to review the configuration example.

Using IP ACLs to Allow Network Traffic

In the same manner, you allow specific traffic to a destination such as the external server in
Figure 2 IP ACL Blocking External Access. To allow HTTP traffic to 209.225.113.24, see
Configuring an External Allow IP ACL to review the configuration example.

Configuring IP ACLs
From Figure 1 Network Example - IP ACL for Internal Servers, the following information is
available:
Source IP address
Source netmask
Destination IP address
Destination netmask
Type of protocol to deny - IP
Ports
VLAN

Configuring an Internal Deny ACL

Let’s configure the ACL for denying traffic from the Engineering server to the HR server and
name the ACL, deny-hr:

Pluribus Networks
www.pluribusnetworks.com
207
CLI network-admin@switch > acl-ip-create name deny-hr action deny scope local
src-ip 192.168.10.2 src-ip-mask 24 dst-ip 192.168.200.3 dst-ip-netmask 24
proto ip src-port 55 dst-port 33 vlan 1505

To review the configuration, use the acl-ip-show command:

CLI network-admin@switch > acl-ip-show name deny-hr layout vertical

name: deny-ip
id: b00011:20
action: deny
proto: ip
src-ip: 192.168.10.2/24
src-port: 55
dst-ip: 192.168.200.3/24
dst-port: 33
vlan: 1505
scope: local
port: 0
Now, when you attempt to access the Finance server from the Engineering server, the
network drops the packets.

Configuring an External Deny ACL

From Figure 2 IP ACL Blocking External Access, you can see the following information:
IP Address
Port Number
To configure an ACL to deny traffic from the external server, use the acl-ip-create
command to create an ACL named deny-external:

CLI network-admin@switch > >acl-ip-create name deny-external scope fabric

src-ip 209.255.113.24/28

To review the configuration, use the acl-ip-show command:

CLI network-admin@switch > acl-ip-show name deny-external layout vertical

name: deny-external
id: b000022:20
action: deny
proto: tcp
src-ip: 209.225.113.24/28
src-port: 0
dst-ip: ::/0
dst-port: 0
vlan: 0
scope: fabric
port: 0

Configuring an External Allow IP ACL

To allow HTTP traffic to the external server, 209.225.113.24 with a netmask of
255.255.255.240 and a scope of fabric, you can create an IP ACL called allow-http using
the following syntax:

208
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > acl-ip-create name allow-http permit scope fabric
src-ip 0.0.0.0. src-ip-mask 255.255.255.255 dst-ip 209.225.113.24 dst-ip-mask
255.255.255.240 protocol tcp dst-port 57

To review the configuration, use the acl-ip-show command:

CLI network-admin@switch > >acl-ip-show name allow-http layout vertical

name: allow-http
id: b000025:20
action: allow
proto: tcp
src-ip: 0.0.0.0/255.255.255.255
src-port: 0
dst-ip: 209.225.113.24/28
dst-port: 57
vlan: 0
scope: fabric
port: 0

To delete the ACL configuration, use the acl-ip-delete command.

To modify the ACL configuration, use the acl-ip-modify command.

Support for DHCP Snooping

Netvisor supports DHCP snooping as a security feature allowing the network to avoid
denial-of-service (DoS)attacks from rogue DHCP servers. You define trusted ports to connect
to the known DHCP servers. DHCP snooping also maintains a mapping table for current
assignments.
In a DHCP packet flow, there are the following packet types:
DHCPDISCOVER/DHCPREQUEST — Packets from the DHCP client to server (UDP
dest-port = 67)
DHCPOFFER/DHCPACK — Packets from the DHCP Server to client (UDP dest-port =
68)
Netvisor must snoop the DHCP packets in order to leverage this feature, and achieves this by
installing a copy-to-cpu vFlow with the parameter, bw-max, to set packet rate limits.
DHCP-client-vflow — Packets with UDP dest-port=67, copy-to-cpu
DHCP-server-vflow — Packets with UDP dest-port=68, copy-to-cpu
A trusted port receives the DHCP server messages from a trusted DHCP server. Netvisor
validates any DHCP server message, such as OFFER/ACKNOWLEDGE, received from trusted
ports. Netvisor designates ports not specifically configured as trusted as untrusted ports.
Netvisor drops any DHCP server message received from an untrusted port, and ensures that
a rogue DHCP server cannot assign IP addresses to devices on your network.

Pluribus Networks
www.pluribusnetworks.com
209
Enable DHCP snooping and specify the list of trusted server ports using the following set of
commands:
(CLI network-admin@Spine1)>dhcp—filter-create name name-string trusted-ports
port-list
name name-string Specify a name for the filter.
trusted-ports port-list Specify a list of trusted ports.

(CLI network-admin@Spine1)>dhcp-filter-modify name name-string trusted-ports

port-list
name name-string Specify the name of the filter to modify.
trusted-ports port-list Specify a list of trusted ports.

CLI network-admin@Spine1)>dhcp-filter-delete name name-string

name name-string Specify the name of the filter to delete.

(CLI network-admin@Spine1)>dhcp-filter-show name name-string trusted-ports

port-list vlan vlan-list

name name-string Displays the name of the filter.

trusted-ports port-list Displays a list of trusted ports.
vlan vlan-list Displays a list of VLANs.

In order to drop the packets from rogue DHCP servers, connected through untrusted ports,
Netvisor has a new system vFlow, DHCP-LOG-DROP.
The vFlow sends the packets to the CPU, to track the untrusted server messages, and then
drop the untrusted DHCP server packets. The vFlow has a higher precedence than the DHCP
trusted ports vFlow. The vFlow includes the untrusted port list for the ingress port.
Untrusted ports typically connect to hosts where DHCP clients can send messages, and
Netvisor ensures the DHCP messages are rate limited using dhcp CPU class. All the DHCP
messages use the dhcp CPU class. Use the existing command for cpu-class-modify:
CLI (network-admin@Spine1)>cpu-class-modify name dhcp rate-limit
rate-limit-number

The show output for the command, dhcp-lease-show, has two new parameters to display
trusted and rogue DHCP servers:
CLI (network-admin@Spine1)>dhcp-lease-show trusted-server|no-trusted-server

CLI (network-admin@Spine1)>dhcp-lease-show
switch ip mac port vnet vlan db-state
server
------------- ------------------- ----------------- ---- ---- ---- ---------
------
Spine1 6053:23a7:0:0:200:: 00:12:c0:80:1f:b8 9 1 unknown

server-ip server-port trusted-server last-msg

---------- ----------- -------------- --------
10.1.1.100 65 no offer

Log messages indicate the presence of an unknown or rogue DHCP servers:

DHCP server message received from untrusted port=<x> server-ip=<ip-addr>

210
Pluribus Networks
www.pluribusnetworks.com
Support for Router Advertisement (RA) Guard
The IPv6 RA Guard feature provides support for allowing the network administrator to block
or reject unwanted or rogue RA guard messages arriving at the network device platform. RAs
are used by devices to announce themselves on the link. The IPv6 RA Guard feature analyzes
these RAs and filters out RAs sent by unauthorized devices. In host mode, all RA and router
redirect messages are not allowed on the port. The RA Guard feature compares configuration
information on the Layer 2 (L2) device with the information found in the received RA frame.
Once the L2 device has validated the content of the RA frame and router redirect frame
against the configuration, it forwards the RA to the unicast or multicast destination. If
Netvisor does not validate the RA frame content, Netvisor drops the RA.

Informational Note: Internal ports and cluster ports are not

blacklisted.

Figure 1: Route Advertisement (RA) Configuration

In Figure 1, the Layer 2 device receives a RA from the router and floods the RA on the ports.
The attacker host, attempting to gain control over the network, sends a misleading RA with
different prefixes, link-local or global IP addresses.The host assumes the attacker to be the
router, based on priority or arrival order. When you configure RA Guard, you can disallow any
RAs sent from ports connected to host ports using RA policies. The RA sent by the router, the
source IP address, from the port and prefixes are whitelisted by the policies defined by the
configuration.
To configure the RA Guard feature, follow these steps:
1. Create an access list using the command, access-list-create.
2. Create a prefix list using the command, prefix-list-create.
3. Create the IPv6 security profile using the command, ipv6security-raguard-create.

Pluribus Networks
www.pluribusnetworks.com
211
This creates two vFlows for RA Guard:
One vFlow drops RAs sent by devices with the role host as assigned using the
ipv6security-raguard-create command
The second vFlow sends RAs to the CPU on qualified ports or VLANs with the action,
to-cpu, and the device role as router.
Netvisor receives and examines the RAs then takes the necessary action based on the
access and prefix lists or port and VLAN policies. The RA accepts and floods back to all
ports.
New commands support this feature:
(CLI network-admin@Spine1)>access-list-create
name name-string Specify a name for the access list.
scope scope Specify if the scope is local or fabric.

(CLI network-admin@Spine1)>access-list-delete name name-string

(CLI network-admin@Spine1)>access-list-show
switch name scope
-------- ---- -----
dorado03 test local

(CLI network-admin@Spine1)>access-list-ip-add
name name-string Specify a name for the access list.
ip ip-address Specify the IP address for the access list.

(CLI network-admin@Spine1)>access-list-ip-delete name name-string ip

ip-address

(CLI network-admin@Spine1)>access-list-ip-show
switch name ip
-------- ---- -------
dorado03 test 1.1.1.4

(CLI network-admin@Spine1)>prefix-list-create
name name-string Specify a name for the prefix list.
scope scope Specify if the scope is local or fabric.

(CLI network-admin@Spine1)>prefix-list-delete name name-string

(CLI network-admin@Spine1)>prefix-list-show
name name-string Displays the name for the prefix list.
scope scope Displays if the scope is local or fabric.

(CLI network-admin@Spine1)>prefix-list-network-add
name name-string Specify the name for the prefix network list.
network ip-address Specify the IP address for the network.
netmask netmask Specify the netmask.

(CLI network-admin@Spine1)>prefix-list-network-delete name name-string

212
Pluribus Networks
www.pluribusnetworks.com
(CLI network-admin@Spine1)>prefix-list-network-show
name name-string Displays the name for the prefix network list.
network ip-address Displays the IP address for the network.
netmask netmask Displays the netmask.

(CLI network-admin@Spine1)>ipv6security-raguard-create

name name-string Specify the RA policy name.

device host|router Specify the type of device as host or router.
router-priority Specify the router priority as low, medium, or high.
low|medium|high
access-list name-string Specify the access list name.
prefix-list name-string Specify the prefix list name.

(CLI network-admin@Spine1)>ipv6security-raguard-delete

name name-string Specify the RA policy name.

(CLI network-admin@Spine1)>ipv6security-raguard-modify
name name-string Specify the RA policy name.
device host|router Specify the type of device as host or router.
router-priority Specify the router priority as low, medium, or high.
low|medium|high
access-list name-string Specify the access list name.
prefix-list name-string Specify the prefix list name.

(CLI network-admin@Spine1)>ipv6security-raguard-show

name name-string Displays the RA policy name.

device host|router Displays the type of device as host or router.
router-priority Displays the router priority as low, medium, or high.
low|medium|high
access-list name-string Displays the access list name.
prefix-list name-string Displays the prefix list name.

(CLI network-admin@Spine1)>ipv6security-raguard-port-add
name name-string Specify the name of the RA Guard policy to add
ports.
ports port-list Specify the list of ports to add to the policy.

Pluribus Networks
www.pluribusnetworks.com
213
(CLI network-admin@Spine1)>ipv6security-raguard-port-remove
name name-string Specify the name of the RA Guard policy to remove
ports.
ports port-list Specify the list of ports to remove from the policy.

(CLI network-admin@Spine1)>ipv6security-raguard-port-show
name name-string Displays the name of the RA Guard policy.
ports port-list Displays the list of ports.

(CLI network-admin@Spine1)>ipv6security-raguard-vlan-add
name name-string Specify the name of the RA Guard policy to add
VLANs.
vlans vlan-id Specify the VLANs to add to the policy.

(CLI network-admin@Spine1)>ipv6security-raguard-vlan-remove
name name-string Specify the name of the RA Guard policy to remove
VLANs.
vlans vlan-id Specify the VLANs to remove from the policy.

(CLI network-admin@Spine1)>ipv6security-raguard-vlan-show
name name-string Displays the name of the RA Guard policy to add
VLANs.
vlans vlan-id Displays the VLANs to add to the policy.

214
Pluribus Networks
www.pluribusnetworks.com
Administering your Switches and Fabric
 Fabric Administration
 Displaying Fabric Statistics
 More Information About Undo Commands and Transactions
 Configuring Logging
 Forwarding Log Files to an External Linux Server
 Configuring SNMP
 Modifying the SNMP Engine ID

Fabric Administration
Using the Fabric Transaction Commands
You can roll back the fabric to a specific fabric transaction number. If a failure occurs on the
fabric, transactions on nodes in the fabric can go out of sync. Once transactions go out of
sync, Netvisor cannot execute further transactions across the scope of local, fabric, or
cluster. Unjoining and rejoining the fabric causes the node to lose the configuration.
As part of a single node transaction recovery, you can roll back the transaction number to a
previous one. If you find multiple nodes out of sync, you must recover each node separately.
If you find that the transaction ID becomes unsynchronized with the rest of the fabric, you
can roll the fabric transaction ID forward on a node.
In the previous example, the switch, CBF-Switch2, became out of sync with the rest of the
fabric. The fabric transaction ID displayed as 327 and the rest of the nodes have a
transaction ID of 328. In this case, you can roll the node, CBF-Switch2, forward to
transaction ID 328. Enter the following command on node CBF-Switch2:

CLI network-admin@switch > transaction-rollforward-to scope fabric tid 328

This command produces output when an error occurs during the transaction. If there no
output displays, the transaction has successfully completed.
To display transaction information for CBF-Switch2,use the transaction-show command:

CLI network-admin@switch > transaction-show format all layout vertical

start-time: 03-19,13:46:42
end-time: 03-19,13:46:43
scope: fabric
tid: 33
state: remote-commit
command: --unrecoverable-- vlan-delete id 22
undo-command: --unrecoverable-- vlan-create id 22 nvid a000030:16 scope fabric
name vlan-22 active yes stats vrg 0:0 ports 1-72,128-129,255 untagged-ports
none send-ports 31,41,47-48,51,65-66 active-edge-ports none ports-specified
false flags
----------------------------------------
start-time: 09:36:09

Pluribus Networks
www.pluribusnetworks.com
215
end-time: 09:36:09
scope: fabric
tid: 34
state: remote-commit
command: vlan-create id 35 scope fabric stats ports-specified true

The scope parameter indicates which set of transactions to display as each scope has an
independent set of transactions associated with it. Netvisor uses fabric as the default scope
unless another scope is specified.
You cannot copy and paste commands and undo-commands because they include
information that cannot apply to new commands. Netvisor displays the fields as
informational-only and allow you to see exactly what happens to the configuration when you
roll forward or roll back the transaction ID.
Once you decide which node to modify and the transaction to roll forward or roll back, you
use the transaction-rollforward-to or transaction-rollback-to commands to re-run
the command (roll forward) or undo the command (rollback) on the node. This applies only
to the local node.

More Information About Undo Commands and Transactions

You may see output similar to this output:
start-time: 21:54:53
end-time: 21:54:53
scope: local
tid: 3
state: commit
command: port-config-modify port 9 enable
undo-command: port-config-modify port 9 enable

Netvisor takes the undo info from the current state on the fabric. So if you enable the port ,
and you try to enable it again, you see the undo-command in the output, since the previous
state also enabled. If you actually disable the port first, and then enable it, you see the
expected undo info in the transaction log.

start-time: 10:05:22
end-time: 10:05:22
scope: local
tid: 20
state: commit
command: port-config-modify port 12 disable
undo-command: port-config-modify port 12 enable
----------------------------------------
start-time: 10:05:48
end-time: 10:05:48
scope: local
tid: 21
state: commit
command: port-config-modify port 12 enable
undo-command: port-config-modify port 12 disable

The transaction undo is not the opposite of the current command, but allows you to go back
to the state before the command was issued. This may be the exact same state as before.

216
Pluribus Networks
www.pluribusnetworks.com
Displaying Fabric Statistics
To display fabric statistics, use the following command:

CLI network-admin@switch > fabric-stats-show

switch: pleiades23
id: 0
servers: 0
storage: 0
VM: 0
vlan: 0
vxlan: 0
tcp-syn: 229K
tcp-est: 171
tcp-completed: 7.19K
tcp-bytes: 3.53G
udp-bytes: 0
arp: 0
vlan: 0
vxlan: 0
switch: pleiades24
id: 0
servers: 0
storage: 0
VM: 0
vlan: 0
vxlan: 0
tcp-syn: 85.6K
tcp-est: 125
tcp-completed: 11.6K
tcp-bytes: 3.95G
udp-bytes: 0
arp: 0
switch: pleiades25
id: 0
servers: 0
storage: 0
VM: 0
vlan: 0
vxlan: 0
tcp-syn: 179K
tcp-est: 20.9K
tcp-completed: 1.60M
tcp-bytes: 485G
udp-bytes: 0
arp: 0

Troubleshooting the Fabric

There may be instances when you need to troubleshoot the fabric. The following is a list of
helpful port numbers, multicast information, and communication on the fabric.
Internal Keepalive
Multicast IP: 239.4.9.7

Pluribus Networks
www.pluribusnetworks.com
217
UDP Destination Port: 23399
This packet is sent from the CPU to the internal port to ensure that the CPU path to the
switch is working and the internal port is up.
Fabric Keepalive
UDP Destination Port: 23394
Point to point UDP fabric keepalive
If these messages don't get through, the fabric node may go to offline state.
Global Discovery
Multicast IP: 239.4.9.3
UDP destination port: 23399
Each node periodically multicasts a message about the fabric. This enables fabric-show
on L2-connected nodes to show available packets and also enables fabric-join name
name. It also enables you to join a fabric over Layer 3 connectivity by specifying an IP
address.
Proxy commands
TCP Destination Port: 23397 SSL
Used for Netvisor OS-to-Netvisor OS communication. Used for internal purposes and also
to implement commands executed on other switches from a local switch.
Status propagation
TCP Destination Port: 23398 SSL
Port changes and vport changes propagated to other nodes in the fabric.
TCP API clients
TCP Destination Port: 23396 SSL
C API clients connect to this port. Disable using admin-service-modify if <mgmt/data>
no-net-api command.
FileSystem replication
TCP Destination Port: 23392 SSL
For ZFS send and ZFS receive messages when replicating file systems across the fabric.
L2 ARP/DMAC miss/Broadcast encapsulation
UDP Destination Port: 23389
These are VXLAN-encapsulated packets sent from CPU to CPU between two L2 connected
switches.
L3 ARP/DMAC miss/Broadcast encapsulation
UDP Destination Port: 23388
These are VXLAN-encapsulated packets sent from CPU to CPU between two L3 connected
switches.
vPORT status
Multicast IP: 239.4.9.4
UDP Destination Port: 23390
vPort updates from hypervisors or hosts in the fabric.
vFlow CPU packets
UDP Destination Port: 23398

218
Pluribus Networks
www.pluribusnetworks.com
 These packets are sent point-to-point for vflow-snoop of a fabric-scoped vFlow.

All of these messages must get sent in order to keep an L2 fabric healthy. The multicast
messages do not propagate through routers so Netvisor does not use the messages for L3
fabrics.
fabric-node-show displays information about internal data structures for each node in the
fabric. If Netvisor does not receive no keepalive or other messages from a fabric node for
about 20 seconds, the node is marked as offline.
Anything that prevents keepalive or other kinds of messages from flowing freely between
fabric nodes can cause problems for fabric connectivity.
If the fabric transaction IDs become unsynchronized, use the transaction commands to either
roll forward or back the transaction IDs. See Fabric Administration<TextRegular> on
page 215.

Displaying System Statistics on a Switch

Display system statistics on a server-switch using the system-stats-show command:

CLI network-admin@switch > system-stats-show layout vertical

switch: Leaf-
uptime: 1h22m26s
used-mem: 27%
used-swap: 0%
swap-scan: 0
cpu-user: 0%
cpu-sys: 1%
cpu-idle: 98%

The swap-scan output displays the number of scans performed on the swap. A nonzero
number indicates that Netvisor pages memory from the physical memory (RAM) to virtual
memory (disk or swap). A consistently high value indicates that all memory, both physical
and virtual, is exhausted and the system may stop responding.

Configuring Logging
logs all important activities that occur on the switch and fabrics created on them. Netvisor
enables logging by default and viewable using the CLI. You can also configure system logging
to send syslog-formatted messages to other servers configured to receive them as part of
centralized logging and monitoring.

Pluribus Networks
www.pluribusnetworks.com
219
Figure 1: Switch with Syslog Server

There are three types of activities logged:

Table 1: Log Events
Log Type Description
Event Records action observed or performed by switches.
Each Event type can be enabled or disabled. Events
are collected on a best effort basis. If events occur
too rapidly to be recorded, Netvisor annotates the
event log with the number of events lost.Netvisor
logs the following examples of event types:
• Port state changes
• TCP connections
• STP port changes
Audit When you make an administrative change to the
configuration, an audit log is recorded. An audit log
consists of the command and parameters along with
the success or failure indication. When a command
fails, Netvisor records an error message.

220
Pluribus Networks
www.pluribusnetworks.com
 Log Type Description
System The system log records error conditions and
conditions of interest. Netvisor has four levels in the
system log:
• critical
• error
• warn
• note
Perror The perror log records messages on standard error
output, describing the last error encountered.

Each log message includes the following information:

Category - event, audit, or system
Timestamp within a microsecond
Process name and process ID of the process producing the message
Unique message name
Unique five digit numerical message code
Message: additional message-specific parameters and explanation
A log message may include optional parameters, including associated VLAN, VXLAN, or
switch port. An audit log message includes additional information:
User
Process ID
Client IP of the remote computer issuing the command
An event log also includes the event type.
The maximum number of repeated messages detected by Netvisor OS is ten (10). After five
seconds, if Netvisor detects repeated messages, then the log prints "Last X messages(s)
repeated Y time(s)”. If the log message detects "X" and "Y" as both 1, then Netvisor prints
the message rather than "Last 1 message(s) repeated 1 time(s)". Netvisor prints the log
events after a five (5) second delay.
To view event logs using the CLI, enter the following command:

CLI network-admin@switch > log-event-show

category time name code event-type port message

event 2013-06-04,13:12:18.304740 port_up 62 port 62 up
event 2013-06-04,13:12:18.304740 port_up 62 port 50 up
event 2013-06-04,13:12:18.304740 port_up 62 port 10 up
...

To view audit log entries, enter the following command:

CLI network-admin@switch > log-audit-show

category time name code user message

audit 2013-06-04,13:12:18.304740 command 1101 network-admin Command create
id=b000011:! name=1 scope=fabric vrg=b000011:0 vlans=100 _mgr_id=b00001

Pluribus Networks
www.pluribusnetworks.com
221
audit 2013-06-04,13:12:18.304740 command 1101 network-admin Command create
vrouter id=b000011:! name=1 scope=fabric vrg=b000011:0 vlans=100
_mgr_id=b00001

To view system log entries, use the following command:

CLI network-admin@switch > log-system-show

time: 2013-09-17, 06:28:09.351514-07:00

name: 11006
level: warn
time: 2013-09-17, 11:28:09.351514-07:00
name: 11006
level: warn
time: 2013-09-17, 13:28:09.351514-07:00
name: 11006
level: warn

Currently, accessing system log information may require assistance from TAC to retrieve the
logs from Netvisor OS. To enable log auditing in Netvisor OS, use the following command

CLI network-admin@switch > log-admin-audit-modify enable|disable

To display auditing status, use the following command:

CLI network-admin@switch > log-admin-audit-show

Modifying and Displaying Log Event Settings

By default, Netvisor logs only system and port events. You can configure Netvisor to include
other logging, and you can add other events using the log-event-settings-modify
command. You can modify the way logs events by using the log-event-settings-modify
command to remove or add log events. For instance to remove logging of STP events, use
the following command:

CLI network-admin@switch > log-event-settings-modify no-stp

To display log event settings information, use the log-event-settings-show command.

Sending Log Messages to Syslog Servers

To configure the switch to send all log messages to a syslog server with an IP address of
172.16.21.67, use the following command:

CLI network-admin@switch > admin-syslog-create name log-all scope fabric host

172.16.21.76

To display the configuration use the admin-syslog-show command:

CLI network-admin@switch > admin-syslog-show

name scope host port message-format

----- ----- ------------ ---- ---------------
log-all fabric 172.16.21.67 514 legacy

222
Pluribus Networks
www.pluribusnetworks.com
To specify sending the syslog messages in structured format, per RFC5424, add the
message-format option to the configuration.

CLI network-admin@switch > admin-syslog-modify name log-all message-format

structured

You can also modify the port that the service listens on to another port. More than one syslog
listening service can be configured and Netvisor sends appropriate syslog messages to each
one.
By default, Netvisor forwards all log messages to syslog servers. To filter the log messages,
use the msg-level option to specify the severity or other options:

CLI network-admin@switch > admin-syslog-match-add syslog-name log-all name

critical-msgs msg-level critical

You can modify syslog matching using the admin-syslog-match-modify command, or

remove matching criteria using the admin-syslog-match-remove command.
To display the configuration, use the show command:

CLI network-admin@switch > admin-syslog-match-show

syslog-name msg-level name

log-all critical critical-msgs

Forwarding Log Files to an External Linux Server

Informational Note: Netvisor supports only one external server.

You can forward log files to an external Linux server and encrypt them using Transport Layer
Security (TLS) over Transmission Control Protocol (TCP). The command,
admin-syslog-create accepts a new parameter, transport tcp-tls|udp, to add TLS
encryption and you can specify a port number for TCP.

CLI network-admin@switch > admin-syslog-create name audit-logs scope local host

172.16.21.33 transport tcp-tls port 10514

You can create TLS certificates using the following command:

CLI network-admin@switch > syslog-tls-cert-request-create country US state CA

city Palo Alto organization QA organizational-unit engineering common-name
pluribusnetworks.com

This command creates a Certificate Signing Request (CSR) and places it in the directory
/sftp/export used by Netvisor OS. You must copy and the CSR to the CA server and sign it.
To import the signed certificate to Netvisor OS, you must copy the certificate and the ca.pem
file to /sftp/import directory in Netvisor OS. Then use the following command to import the
files:

Pluribus Networks
www.pluribusnetworks.com
223
CLI network-admin@switch > syslog-tls-cert-import file-ca ca.pem file-cert
my-cert.pem

To enable TLS-TCP logging export, use the following syntax:

CLI network-admin@switch > admin-syslog-create name audit-logs scope local host

172.16.21.33 transport tcp-tls port 10514

To display the export information, use the admin-syslog-show command:

CLI network-admin@switch > admin-syslog-show

switch name scope host port transport message-for

mat
------- -------- ------ ----------- ----- --------- -----------
---
leaf-pst-1 MYTLS local 172.21.16.33 10514 tcp-tls legacy

Other new commands

syslog-tls-cert-clear Clears the certificates
syslog-tls-cert-request-show Displays the certificate information
syslog-tls-cert-show Displays syslog TSL import certificate
config

Saving Diagnostic Files and Exporting to an External Server

1. Use the save-diags export-sftpcommand.
2. The signed *.tar file is saved to /sftp/export directory in Netvisor.
3. Enable SFTP on the switch using the admin-sftp-modify enable command.
4. Copy the file to the external server using SFTP to the Netvisor switch.

Using Facility Codes with Log Messages

Netvisor labels log messages with a facility code indicating the area of the software that
generated the log message. Netvisor uses the following facility codes by default:
Log_Daemon for events and system messages
Log_AUDIT for audit messages
The following severities are used by default:
Log_INFO for events and audit messages
Log_Critical = critical
Log_ERROR = error
Log_WARNING = warn
Log_NOTICE = note
You can override the default values by configuring matches for each syslog configuration
which allows Netvisor to translate log messages into fields that the syslog servers
understand.

224
Pluribus Networks
www.pluribusnetworks.com
Displaying Log Counters Information
You can display information about the number of events occurring on the network by using
the log-system-counters-show command:

CLI network-admin@switch > log-system-counters-show layout vertical

switch: pleiades24
critical: 0
error: 0
warn: 1061
note: 9

To reset the log counters, use the log-system-counters-reset command.

Formatting and Filtering of Logging Messages

Netvisor allows many options for filtering and formatting of log messages returned by these
commands. Use the <tab> completion method and ? to explore them.
You can access the log files using SFTP, switch-ip:/sftp//logs and NFS,
/net/switch-name//logs if you have enabled the services.
Many systems support a syslog facility for sending or receiving log messages. The
infrastructure can send messages to syslog servers using either RFC 5424 (Structure) or RFC
3164 (legacy) formats.

Viewing Log Events

For information about specific log events and the meaning, see the Log Message Reference
Guide.
A log message consists of common parameters separated by spaces and a colon (:), and
optional parameters such as key and value pairs, another colon, and then the log-specific
message.
To view event logs using the CLI, enter the following command:

CLI network-admin@switch > log-event-show

category: event
time: 2014-07-17,07:37:17.466173-07:00
switch: pleiades24
program: nvOSd
pid: 6344
name: mac_ip_changed
code: 11023
event-type: port
: global-default
port: 65
vlan: 200
message: ip address change: mac=50:33:a5:e0:7f:fd ip=172.16.23.7
category: event
time: 2014-07-17,07:37:50.109133-07:00
switch: pleiades24
program: nvOSd
pid: 6344
name: mac_ip_changed

Pluribus Networks
www.pluribusnetworks.com
225
code: 11023
event-type: port
: vlb-web-svr
port: 65
vlan: 200
message: ip address change: mac=50:33:a5:e0:7f:fd ip=172.16.23.1
category: event
time: 2017-05-05,07:42:17.418349-07:00...

To view audit log entries, enter the following command:

CLI network-admin@switch > log-audit-show layout vertical

category: audit
time: 2017-04-01,14:56:40.763626-07:00
name: user_command
code: 11001
user: network-admin
message: Command "vlan-create id 25
category: audit
time: 2017-04-01,14:56:40.765839-07:00
name: logout
code: 11100
user: network-admin
message: logout
category: audit
time: 2017-04-01,14:56:40.847912-07:00
name: login
code: 11099
user: network-admin
message: login
category: audit
time: 2017-04-01,14:56:40.888363-07:00
name: logout
code: 11100
...
To view system log entries, use the following command:

CLI network-admin@switch > log-system-show

time: 2013-09-17, 06:28:09.351514-07:00

name: 11006
level: warn
time: 2013-09-17, 11:28:09.351514-07:00
name: 11006
level: warn
time: 2013-09-17, 13:28:09.351514-07:00
name: 11006
level: warn

226
Pluribus Networks
www.pluribusnetworks.com
Displaying Log Counters Information
You can display information about the number of events occurring on the network by using
the log-system-counters-show command:

CLI network-admin@switch > log-system-counters-show layout vertical

switch: pleiades24
critical: 0
error: 0
warn: 1061
note: 9

To reset the log counters, use the log-system-counters-reset command.

Displaying System Statistics on a Switch

Display system statistics on a server-switch using the system-stats-show command:

CLI network-admin@switch > system-stats-show layout vertical

switch: Leaf-
uptime: 1h22m26s
used-mem: 27%
used-swap: 0%
swap-scan: 0
cpu-user: 0%
cpu-sys: 1%
cpu-idle: 98%

The swap-scan output displays the number of scans performed on the swap. A nonzero
number indicates that memory is paged from the physical memory (RAM) to virtual memory
(disk or swap). A consistently high value indicates that all memory, both physical and virtual,
is exhausted and the system may stop responding.

Exceptions for Audit Logging

Use the commands log-audit-exception-create, log-audit-exception-delete, and
log-audit-exception-show to control which CLI, shell and vtysh commands subjected to
auditing. If Netvisor subjects a command to auditing, Netvisor logs the command in the audit
log and sends it to the TACACS+ server as authorization and accounting messages.

CLI network-admin@Spine1>log-audit-exception-create

Create an audit logging exception.

cli|shell|vtysh Specify the type of audit exception.
pattern pattern-string Specify a regular expression to match
exceptions.
any|read-only|read-write Specify the access type to match exceptions.
scope local|fabric Specify the scope of exceptions.

CLI network-admin@Spine1>log-audit-exception-delete

Delete an audit logging exception.

cli|shell|vtysh Specify the type of audit exception.

Pluribus Networks
www.pluribusnetworks.com
227
pattern pattern-string Specify a regular expression to match
exceptions.
any|read-only|read-write Specify the access type to match exceptions.

CLI network-admin@Spine1>log-audit-exception-show

Display audit logging exceptions.

cli|shell|vtysh Display the type of audit exception.
pattern pattern-string Display a regular expression to match
exceptions.
any|read-only|read-write Display the access type to match exceptions.
scope local|fabric Display the scope of exceptions.

By default, Netvisor audits every command except for read-only CLI commands and
^/usr/bin/nvmore which is the pager for the Netvisor CLI:
CLI (network-admin@switch) > log-audit-exception-show
switch type pattern access scope
------ ----- ---------------- --------- -----
switch cli read-only local
switch shell ^/usr/bin/nvmore any local

To enable auditing of ALL CLI commands, you can delete the cli/read-only exception:

CLI (network-admin@switch) > log-audit-exception-delete cli read-only

Modifying User Roles to Allow Shell Access

You can add privileges to a user by adding new parameters available for roles. To add shell
access to a user’s role, use the following syntax:
CLI (network-admin@switch) >role-create
name name-string Specify a name for the user role.
scope local|fabric Specify a scope for the user role.
One or more of the following options:
access read-only|read-write Specify the type of access for the user role.
The default is read-write.
running-config|no-running-config Specify if the user role allows access to the
switch running configuration.
shell|no-shell Specify if the user role allows access to the
shell.
sudo|no-sudo Specify if the user role allows the sudo
command.

The new parameters are also available for the role-modify command.

228
Pluribus Networks
www.pluribusnetworks.com
Configuring SNMP
Networks use Simple Network Management Protocol (SNMP) for monitoring the health of
network equipment such as routers, computer equipment and even devices such as UPS.
Netvisor supports SNMP v1, v2, and v3 . The SNMP daemon runs as a service and launches
by using the following command:

CLI network-admin@switch > admin-service-modify if mgmt snmp

This command launches the daemon, sub-agents, and opens the firewall so that remote
queries can reach the daemon.

SNMP Communities
SNMPv1 uses communities as a method of controlling access to information. You can create a
community using the following command:

CLI network-admin@switch > snmp-community-create community-string name-string

community-type read-only|read-write

To create a SNMP community string named, snmp-group, with read-only privileges, use the
following command:

CLI network-admin@switch > snmp-community-create community-string snmp-group

community-type read-only

To modify the SNMP community, snmp-group, to read-write, use the following command:

CLI network-admin@switch > snmp-community-modify community-string snmp-group

community-type read-write

To display information about the SNMP community, snmp-group, use the following command:

CLI network-admin@switch > snmp-community-show community-string snmp-group

switch community-string community-type

------ ---------------- --------------
pleiades24 snmp-group read-only

To delete the SNMP community, snmp-group, use the following command:

CLI network-admin@switch > snmp-community-delete community-string snmp-group

Users and SNMPv3

SNMPv3 creates users as access control mechanisms, and creating users is secure and
flexible. You can also require that users must authenticate and use encryption.

Informational Note: Prior to Version 2.6, MD5 was the default authentication protocol. With
Version 2.6, Netvisor OS supports SHA1 and is the default authentication protocol. You must specify MD5
if MD5 authentication is required.

Pluribus Networks
www.pluribusnetworks.com
229
Use the following command to create a user:

CLI network-admin@switch > snmp-user-create user-name name-string

auth-password [auth|no-auth] priv-password [priv|no-priv]

To create the user, snmp-admin, with authentication, password m0nk3ys, use the following
command:

CLI network-admin@switch > snmp-user-create user-name snmp-admin auth-password

auth

auth password: ********

confirm password: ********
To modify the SNMP user and add the password, b33h!v3, use the following command:

CLI network-admin@switch > snmp-user-modify user-name snmp-admin auth-password

auth priv-password priv

priv-password priv
auth password: ********
confirm password: ********
priv password: ******
confirm password: ******

To display information about the SNMP user, use the following command:

CLI network-admin@switch > snmp-user-show user-name snmp-user

switch user-name auth priv

-------- --------- ---- ----
pleiades24 snmp-user yes yes

To delete the SNMP user, use the snmp-user-delete command.

After you create the user, you must grant permission, using View Access Control Model
(VACM) to view SNMP objects:

CLI network-admin@switch > snmp-vacm-create user-name name-string user-type

[rouser|rwuser] oid-restrict string [auth|no-auth] [priv|no-priv]

The command parameter, oid-restrict, an optional argument, specifies a MIB sub-tree

with a restricted view. . In other words, if you specify an OID, you can only see that OID and
the descendants in the tree .
To continue with the previous example, Netvisor restricts snmp-user as a read-only user for
sysContact OID:

CLI network-admin@switch > snmp-vacm-create user-name snmp-user user-type

rouser oid-restrict sysContact no-auth no-priv

To modify the VACM configuration and change no authentication to authentication, use the
following command:

CLI network-admin@switch > snmp-vacm-modify user-name snmp-user user-type

rouser auth

230
Pluribus Networks
www.pluribusnetworks.com
To display information about the VACM configuration, use the snmp-vacm-show command:
switch user-type user-name oid-restrict view auth priv
------ --------- --------- ------------ ---- ---- ----
pleiades24 rouser snmp-user sysContact no no

To delete the VACM user from the SNMP configuration, use the snmp-vacm-delete command:

CLI network-admin@switch > snmp-vacm-delete user-name snmp-user

Modifying the SNMP Engine ID

Netvisor allows you to modify the SNMP Engine ID and retrieve previous SNMP agent
information for a switch no longer in use. If you have to remove a switch from the network,
you can modify the SNMP Engine ID to use the old SNMP Engine ID so Netvisor can query
and maintain the same history records for the new switch.
(CLI network-admin@Spine1)>snmp-engineid-modify

engineid string Specify the 28 character unique ID for the SNMP

engine.

(CLI network-admin@Spine1)>snmp-engineid-modify engineid

0x80001f8880077f7820da49395a00000000
Warning: All SNMP users will be erased.

Please confirm y/n (Default: n):y

Modified snmp engineID, Deleted all SNMP users.Please re-create SNMP users.

Supported MIBs
customized MIBs:
IfTable
IfXTable
EntPhySensorTable

 EntPhySensorTable

Additional commands that support SNMPv1, SNMPv2, and SNMPv3:

snmp-engineid-show — The SNMP engine ID is a unique string of 24 characters that identifies
the device for administrative purposes. This command displays the identification of the local
SNMP engine and all remove engines configured on the switch.
snmp-trap-enable-modify — Used to enable notifications about link conditions and
common system errors. This is used with the snmp-monitor commands.
snmp-trap-enable-show — Display enabled SNMP traps.
snmp-trap-sink-create — Used to specify a SNMPv1 trap receiver.
snmp-trap-sink-delete — Delete SNMP trap sinks
snmp-trap-sink-modify — Modify SNMP trap sinks.
snmp-trap-sink-show — Display SNMP trap sinks.
snmp-v3-trap-sink-create - Used to specify a SNMPv3 trap receiver.
snmp-v3-trap-sink-delete — Used to delete a SNMPv3 trap receiver.
snmp-v3-trap-sink-delete — Used to delete a SNMPv3 trap receiver.

Pluribus Networks
www.pluribusnetworks.com
231
 snmp-v3-trap-sink-modify — Used to modify a SNMPv3 trap receiver.
snmp-v3-trap-sink-show — Used to display a SNMPv3 trap receiver.

SNMP Traps (MIBs) for Link Congestion

SNMP MIBs send traps when a link becomes congested and when a node state in the fabric
changes.
Link congestion can be detected when Netvisor drops ingress or egress packets due to link
congestion. Netvisor logs this information into the system logs and you can enable Netvisor
to send “Link Congestion detected” SNMP traps.
When Netvisor detects a fabric node as dead or keepalives are detected, Netvisor checks the
previous state of the nod and determines if the previous state was online or offline. SNMP
sends out traps as “Node status changed.”
To enable the traps, use the snmp-trap-enable-modify command:
CLI (network-admin@Spine1)snmp-trap-enable-modify
one or more of the following options:

link-up-down|no-link-up-down Specify if you want to enable a link

up|down trap.
default-monitors|no-default-monitors Specify if you want to enable a default
monitoring trap.
physical-sensors|no-physical-sensors Specify if you want to enable a physical
sensor trap.
low-disk-space|no-low-disk-space Specify if you want to enable a low disk
low-disk-space-threshold-string space trap.
system-usage|no-system-usage Specify if you want to enable a system
usage trap.
high-system-usage-threshold Specify if you want to enable a low disk
high-system-usage-threshold-string space trap.
login-failure|no-login-failure Specify if you want to enable login failure
trap.
lacp-status|no-lacp-status Specify if you want to enable a LACP trap.
vport-modified|no-vport-modified Specify if you want a enable a trap when
a vPort is modified.
mirror-to-cpu|no-mirror-to-cpu Specify if you want a mirror to CPU trap.
stp-port-state-failed| Specify if you want to enable a trap when
no-stp-port-state-failed the STP port state is failed.
link-congestion-detected| Specify if you want to enable a trap when
no-link-congestion-detected a link congestion state is detected.
fabric-node-state-changed|no-fabric-no Specify if you want a trap send when a
de-state-changed fabric node state changes.

Topic Feedback
Was this topic useful to you? Please provide feedback to improve the content.

232
Pluribus Networks
www.pluribusnetworks.com
Additional Supported MIBs

MIB Description
Entity-Sensor This module defines Entity MIB extensions for physical
sensors.
The Entity Sensor MIB contains a single group called the
entitySensorValueGroup, which allows objects to
convey the current value and status of a physical sensor.
The entitySensorValueGroup contains a single table,
called the entPhySensorTable, which provides a small
number of read-only objects:
entPhySensorType
This object identifies the type of data units associated with
the sensor value.
entPhySensorScale
This object identifies the (power of 10) scaling factor
associated with the sensor value.
entPhySensorPrecision
This object identifies the number of decimal places of
precision associated with the sensor value.
entPhySensorValue
This object identifies the current value of the sensor.
entPhySensorOperStatus
This object identifies the current operational status of the
sensor (as it's known to the agent).

Pluribus Networks
www.pluribusnetworks.com
233
MIB Description
entPhySensorOperStatus
This object identifies the current operational status of the
sensor (as it's known to the agent).
entPhySensorUnitsDisplay.1 STRING: Temp Outlet. >60
entPhySensorUnitsDisplay.2 STRING: Temp Inlet.
entPhySensorUnitsDisplay.3 STRING: Temp 2
BCM56960.
entPhySensorUnitsDisplay.4 STRING: Temp 3
OptoPHYJ5.
entPhySensorUnitsDisplay.5 STRING: Temp 4
OptoPHYJ9.
entPhySensorUnitsDisplay.6 STRING: Fan1 Inlet.
entPhySensorUnitsDisplay.7 STRING: Fan1 Outlet.
entPhySensorUnitsDisplay.8 STRING: Fan2 Inlet.
entPhySensorUnitsDisplay.9 STRING: Fan2 Outlet.
entPhySensorUnitsDisplay.10 STRING: Fan3 Inlet.
entPhySensorUnitsDisplay.11 STRING: Fan3 Outlet.
entPhySensorUnitsDisplay.12 STRING: Fan4 Inlet.
entPhySensorUnitsDisplay.13 STRING: Fan4 Outlet.
entPhySensorUnitsDisplay.14 STRING: Fan PSUR.
entPhySensorUnitsDisplay.15 STRING: PSUR Temp1.
entPhySensorUnitsDisplay.16 STRING: PSUR Temp2.
entPhySensorUnitsDisplay.17 STRING: AN Temp1.
entPhySensorUnitsDisplay.18 STRING: ROV1 Temp1.
entPhySensorUnitsDisplay.19 STRING: ROV2 Temp1.
entPhySensorUnitsDisplay.20 STRING: OP1 Temp1.
entPhySensorUnitsDisplay.21 STRING: OP2 Temp1.
entPhySensorUnitsDisplay.22 STRING: PS1 Status.
entPhySensorUnitsDisplay.23 STRING: PS2 Status.
entPhySensorUnitsDisplay
This object provides a textual description of the data units
represented by the entPhySensorType and
entPhySensorScale objects.
entPhySensorValueTimeStamp
The object identifies the value of sysUpTime at the time
the agent last updated the information in the entry. This
object is only relevant if the agent uses a polling
implementation strategy, for example, the associated
entPhySensorValueUpdateRate object is greater than
zero.
The RFC and be found here.

234
Pluribus Networks
www.pluribusnetworks.com
 MIB Description
Host-Resource Use this MIB in managing host systems. The term `host' is
s construed to mean any computer that communicates with
other similar computers attached to the Internet and that
is directly used by one or more human beings. Although
this MIB does not necessarily apply to devices whose
primary function is communications services, for example,
terminal servers, routers, bridges, monitoring equipment),
such relevance is not explicitly precluded. This MIB
instruments attributes common to all Internet hosts
including, for example, both personal computers and
systems that run variants of Unix. The RFC is found here.
IF The MIB module to describe generic objects for network
interface sub-layers. This MIB is an updated version of the
ifTable for MIB-II, and incorporates the extensions defined
in RFC 1229.

Supported Traps

Trap Name Description Trap Type Trigger

link-up-down Port link is up Event-based If enabled, SNMP
or down. generates a trap
when a port is up or
down.
default-monitors Use default
SNMP
monitoring.
physical-sensors Physical If enabled, SNMP
sensors are generates a trap for
enabled. physical sensors such
as power supplies
and fans.
low-disk-space Monitors for Event-based If enabled, SNMP
low-disk-space generates a trap if
. disk space is lower
than threshold.
SNMP checks the
output of the
command,
storage-pool-show.

Pluribus Networks
www.pluribusnetworks.com
235
Trap Name Description Trap Type Trigger
low-disk-space-thresh The threshold Event-based SNMP generates the
old value of trap with
low-disk-space low-disk-space.
in %.
system-usage Monitors Event-based If enabled, SNMP
memory & CPU generates a trap if
usage. system-usage[Total
CPU-sys + user]
space is greater than
threshold. SNMP
checks the output of
the command,
"system-stats-show"
system-usage-thresho system-usage- SNMP generates the
ld threshold system usage trap.
login-failure Monitors login Message-based If enabled, SNMP
failures. generates a trap
when user login with
wrong password.
lacp-status Monitors LACP Message-based If enabled, SNMP
enable or generates a trap
disable. when the LACP state
changes from enable
to disabled or
disabled to enable.
vport-modified Monitors vPort Message-based If enabled, SNMP
modifications. generates a trap
when vPort
modifications occur
on the switch.
stp-port-modified Monitors STP Message-based If enabled, SNMP
port status. generates a trap
when STP port state
is modified using the
command,
switch-local
stp-port-modify
port 1
<block|edge|bpdu|
root-guard>

236
Pluribus Networks
www.pluribusnetworks.com
 Trap Name Description Trap Type Trigger
stp-port-state-failed Monitors STP Message-based If enabled, SNMP
port state generates a trap
failures. when STP port state
is modified using the
command,
switch-local
stp-port-modify
port 128 edge
bpdu-guard
mirror-to-cpu Monitors Message-based If enabled, SNMP
mirror-to-cpu generates a trap
configuration. when created a
vlflow using the
command,
vflow-create name
mirror scope local
action copy-to-cpu
and also generates a
trap for perror.log.
link-congestion-detect Monitors Message-based If enabled, SNMP
ed congestion generates a trap
drop at port. indicating a link is
congested.
fabric-node-state-cha Monitors fabric Message-based If enabled, SNMP
nged node states. generates a trap
when the a fabric
node changes state.
stp-new-root Monitors new Event-based If enabled, SNMP
STP root. generates a trap to
monitor a new root
for STP.
stp-topology-changed Monitors STP Event-based If enabled, SNMP
topology generates a trap to
change monitor topology
changes for STP.
interface-up-down Monitors Event-based If enabled, SNMP
vRouter generates a trap for
interfaces. an interface with the
state up or down.
disable-start-stop Monitors Event-based
disable traps
for start and
stop.
fabric-node-state-cha Monitors fabric Event-based If enabled, SNMP
nged node states. generates a trap to
monitor fabric node
state changes.

Pluribus Networks
www.pluribusnetworks.com
237
Trap Name Description Trap Type Trigger
system-usage Monitors Event-based If enabled, SNMP
memory & cpu generates a trap to
usage. monitor memory and
CPU changes.
vrrp-new-master Monitors VRRP Event-based If enabled, SNMP
master generates a trap to
changes. monitor VRRP master
state changes.

Topic Feedback
Was this topic useful to you? Please provide feedback to improve the content.

238
Pluribus Networks
www.pluribusnetworks.com
Using Analytics
 Configuring vFlow for Analytics
 Redirecting Analytics to a Rear Facing NIC
 Using vFlows to Disable Communication
 Configuring Mirroring for vFlows and Ports
 Port Mirroring to a Remote Host
 Managing Traffic Classes with vFlow
 Applying CoS Queue Mapping based on Re-Marked DSCP in vFlow
 Configuring Burst Size in vFlow for Maximum Bandwidth
 Displaying Multiple Objects for Show Commands
 Support for Policy-based Routing
 Using Application Flows and Statistics
 Support for Policy-based Routing
 Configuring vFlows in Virtual Wire Mode
 Support for TCP Parameters using vFlows
 Configuring vFlows with User Defined Fields (UDFs)
 Configuring Priority-based Flow Control
 Configuring Priority-based Flow Control Port Statistics
 About sFlow
 Using Wireshark to Analyze Packets in Real Time

Configuring vFlow for Analytics

A vFlow can be used to capture packets for analysis, and you can determine if the vFlow
captures packets across the fabric or on a single switch. Packets are captured by forwarding
them from the data plane of the switch to the control plane.
A flow that directs packets to the switch CPU can be configured to save packets to a file by
enabling the log-packets parameter. The file is written using a libcap compatible format so
that programs like TCPdump and Wireshark can be used to read the file. The file is exported
to clients using NFS or SFTP.

Pluribus Networks
www.pluribusnetworks.com
239
Packet capture data is available with switch or fabric scope. The pcap files are stored over
NFS in the following locations:

/net/<ServerSw_Name>/ONVL/global/flow/<Flow_Name>/switch/<Switch_Name>/p
cap

/net/<ServerSw_Name>/ONVL//<_Name>/flow/<Flow_Name>/
switch/<Switch_Name>/pcap

/net/<ServerSw_Name>/ONVL/global/flow/<Flow_Name>/fabric/pcap

/net/<ServerSw_Name>/ONVL//<_Name>/flow/<Flow_Name>/
fabric/pcap

Snooping only works if you use the parameters, copy-to-cpu or to-cpu. The copy-to-cpu
parameter ensures that the data plane forwards the packets and sends a copy to the CPU.
Use this parameter if you want traffic to flow through the switch. The to-cpu parameter
doesn’t forward packets and interrupts traffic on the switch. To snoop all application flow
packets of protocol type TCP, enter the following CLI commands at the prompt:

CLI network-admin@switch > vflow-create name snoop_all scope local proto tcp
action copy-to-cpu

Then use the following command to display the output:

CLI network-admin@switch > vflow-snoop

switch: pleiades24, flow: snoop_all, port: 65, size: 66, time:

20:07:15.03867188
smac: 64:0e:94:28:00:fa, dmac: 64:0e:94:2c:00:7a, etype: ip
sip: 192.168.2.51, dip: 192.168.2.31, proto: tcp
sport: 42120, dport: 33399

switch: pleiades24, flow: snoop_all, port: 65, size: 184, time:

20:07:15.03882961
smac: 64:0e:94:28:00:fa, dmac: 64:0e:94:2c:00:7a, etype: ip
sip: 192.168.2.51, dip: 192.168.2.31, proto: tcp
sport: 42120, dport: 33399

switch: pleiades24, flow: snoop_all, port: 43, size: 66, time:

20:07:15.03893740
smac: 64:0e:94:2c:00:7a, dmac: 64:0e:94:28:00:fa, etype: ip
sip: 192.168.2.31, dip: 192.168.2.51, proto: tcp
sport: 33399, dport: 42120

To restrict the flows captured to TCP port 22, SSH traffic, create the following vFlow:

CLI network-admin@switch > vflow-create name snoop_ssh scope local action

copy-to-cpu src-port 22 proto tcp vflow-add-filter name snoop_ssh

240
Pluribus Networks
www.pluribusnetworks.com
Then use the vflow-snoop command to display the results:
switch: pleiades24, flow: snoop_ssh, port: 41, size: 230, time:
10:56:57.05785917 src-mac: 00:15:17:ea:f8:70, dst-mac: f4:6d:04:0e:77:60,
etype: ip src-ip: 10.9.11.18, dst-ip: 10.9.10.65, proto: tcp src-port: 22,
dst-port: 62356
switch: pleiades24, flow: snoop_ssh, port: 41, size: 118, time:
10:56:57.05922560 src-mac: 00:15:17:ea:f8:70, dst-mac: f4:6d:04:0e:77:60,
etype: ip src-ip: 10.9.11.18, dst-ip: 10.9.10.65, proto: tcp src-port: 22,
dst-port: 62356

The optional parameter vflow-add-filter restricts the output of the vflow-snoop

command to the packets matching the snoop_ssh flow definition.
To capture traffic packets for a flow across the entire fabric, you create a flow with the scope
of fabric. To copy the packets to a pcap file, add the log-packets option:

CLI network-admin@switch > vflow-create name fab_snoop_all scope fabric action

copy-to-cpu port 22 log-packets yes

If you enable log-packets, the separate pcap files for all switches are available on any switch.
In addition a consolidated pcap file is available that aggregates the packets from all switches
in the entire fabric.

Support for IPv6 Addresses and vFlow Configurations

You must modify the vFlow table profile using the new command,
vflow-table-profile-modify:

CLI (network-admin@Spine1) >vflow-table-profile-modify profile ipv6 hw-tbl

switch-main percent 10

You must reboot the switch in order for the settings to take effect. To ensure that the profile
is available after rebooting, use the vflow-table-show command:

CLI (network-admin@Spine1) >vflow-table-show

switch name flow-max flow-used flow-tbl-slices capability

flow-profile
-------- --------------------- -------- --------- ---------------
-------------- ------------
lev-leo2 Egress-Table-1-0 512 0 1 match-metadata
system
lev-leo2 IPv6-Table-1-0 2048 0 1 none
ipv6 <=========
lev-leo2 System-L1-L4-Tun-1-0 1536 41 2 set-metadata
system
lev-leo2 System-VCAP-table-1-0 512 0 1 none
system

Redirecting Analytics to a Rear Facing NIC

This feature provides an option to copy analytics traffic to one of the rear facing NICs instead
of the CPU. By redirecting analytics traffic to a dedicated port instead of the common CPU
port, Netvisor OS alleviates the load on the CPU port.

Pluribus Networks
www.pluribusnetworks.com
241
The analytics port is part of all VLANs and is removed from the flood map. When the
connection-stats-setting is enabled to redirect analytics, Netvisor OS receives
information from the switch vPort database. There is no HA (High Availability) impact as this
feature is local to the switch.
The following command specifies the target of analytics traffic on the supported switch:
(CLI network-admin@Spine1)>connection-stats-settings-modify
redirect-analytics-vflow span3

The following command entered through the CLI returns to the default setting of redirecting
the traffic to CPU:
(CLI network-admin@Spine1)>connection-stats-settings-modify
redirect-analytics-vflow none

Using vFlows to Disable Communication

Flows can be used to specify communications that are not allowed with a switch or a fabric.
Use the following steps to create a vFlow as a firewall:
1. Define a VLAN and destination IP-based flow and specify that the flow is dropped by the
switch, with statistics monitoring enabled:

CLI network-admin@switch > vflow-create name flow3 scope local vlan 99 dst-ip
172.168.24.1 action drop stats enable

Display the statistics for the new flow above as the traffic is dropped:

CLI network-admin@switch > vflow-stats-show name flow3 show-diff-interval 5

switch name packets bytes cpu-packets cpu-bytes

aquila02 flow3 864 116K 0 0
switch name packets bytes cpu-packets cpu-bytes
aquila02 flow3 5 936K 0 0

There are many options available for creating vFlows, and vFlows can be used to shape
traffic, capture statistics, capture flow metadata, capture packets, or manage
communications. The options include:
 vlan

 in-port

 out-port

 ether-type

 src-mac

 src-mac-mask

 dst-mac

 dst-mac-mask

 src-ip

242
Pluribus Networks
www.pluribusnetworks.com
 src-ip-mask

 dst-ip

 dst-ip-mask

 src-port

 dst-port

 dscp

 tos

 proto

 flow-class

 uplink-ports

 bw-min

 bw-max

 precedence

 action

 action-value

 no-mirror

 mirror

 no-process-mirror

 process-mirror

 no-log-packets

 log-packets

 packet-log-max

 stats

 stats-interval

 duration

 no-transient

 transient

 vxlan

 vxlan-ether-type

 vxlan-proto

Use Case Scenario

Pluribus Networks
www.pluribusnetworks.com
243
In a real use case, the command connection-show server-ip 10.9.10.117 was used to
analyze a suspicious connections to server 10.9.10.117:
switch: switch02
vlan: 1
client-ip: 10.9.9.33
server-ip: 10.9.9.107
service: http
dur(s): 0
latency(us): 65
out-bytes: 0
in-bytes: 0
active: yes
switch: switch02
vlan: 1
client-ip: 10.9.9.33
server-ip: 10.9.9.107
service: http
dur(s): 210
latency(us): 7
out-bytes: 48804
in-bytes: 6120
active: yes
switch: switch02
vlan: 1
client-ip: 10.9.9.33
server-ip: 10.9.9.107
service: http
dur(s): 328
latency(us): 30
out-bytes: 48720
in-bytes: 612620
active: yes

Configuring Mirroring for vFlows and Ports

An Netvisor OS fabric administrator can run services and applications within the switch.
Consider the use case of an application that needs access to data that is flowing through the
switch, but does not want to impede that flow. The port-mirroring feature provides this
functionality.
The system predefines a mirror configuration, but does not insert any traffic into that mirror.
Use the following steps to setup mirroring to send from all of the data ports to the span port
(port 66)The command syntax for mirror-modify is as follows:

CLI network-admin@switch > mirror-modify out-port port-list in-port port-list

[policy port|vflow] mirroring|no-mirroring

CLI network-admin@switch > mirror-show [format fields-to-display]

[parsable-delim character] [sort-asc] [sort-desc] [show dups] [layout
vertical|horizontal] [show-interval seconds-interval]

View the status of mirroring by entering the following at the CLI command prompt:

244
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > mirror-show

switch: T6001-ON
direction: bidirection
out-port:
in-port:
mirroring: disable

The parameter out-port is not configured and mirroring is disabled therefore, no data
mirroring can occur.
To modify the mirroring configuration, use the following steps:
1. Use the mirror-modify command to set the output to the span port. However, if there is
more than 10Gb of traffic on ports 1-64, do not execute this command.

CLI network-admin@switch > mirror-modify in-port 1-64 out-put 66 mirroring

mirror-show
switch: T6001-ON
direction: bidirection
out-put: 66
in-port: 1-64
mirroring: enable
To disable the configuration, use the following command:

CLI network-admin@switch > mirror-modify no-mirroring

mirror-show
switch: T6001-ON
direction: bidirection
out-port: 66
in-port: 1-64
mirroring: disable

Port Mirroring to a Remote Host

A port mirroring configuration that allows mirrored traffic to be transmitted to a remote host
which is located across L2 or L3 IP network. This feature allows you to monitor traffic from
source ports distributed over multiple switches, which means that you can centralize your
network capture devices. Port Mirroring to a remote host works by mirroring the traffic from
the source ports of a mirrored port session onto a VLAN that is dedicated for the port
mirroring session. This VLAN isthen trunked to other switches, allowing session traffic to be
transported across multiple switches. On the switch that contains the destination port for the
session, traffic from the session VLAN is simply mirrored out the destination port. Parameters
are available for the mirror-create command for this feature.

Mirroring Traffic to a Virtual Machine (VM) Interface

Mirroring traffic coming from a switch port rear facing network interface card (NIC) to a VM
NIC is now supported. This feature is useful for several reasons:
 Viewing incoming traffic from front facing ports.

 Troubleshooting issues if traffic is not running as expected.

 Using a firewall, running as an application on a VM, for all incoming traffic.

Pluribus Networks
www.pluribusnetworks.com
245
This feature is related to the existing mirror-create command which mirrors traffic from
any port to a rear facing NIC and uses the parameter option mirror-traffic on the
Netvisor OS-kvm-interface-add command.

Managing Traffic Classes with vFlow

Netvisor OS provides a full set of traffic class features, including the ability to view and create
traffic classes, as well as assign traffic classes to flows to manage the quality of service of the
flow traffic and shape the traffic passing through an Netvisor fabric.
To display the currently defined traffic classes:

CLI network-admin@switch > vflow-class-show

name scope type priority

------------- ------ ------ --------
meter fabric system 0
guaranteed_bw fabric system 9
lossless fabric system 10
control fabric system 11

The higher the priority number, the higher the priority of the class. To add a vflow class, use
the vflow-class-create command:

CLI network-admin@switch > vflow-class-create name traffic-1 scope fabric

priority 5

This creates a traffic class with a scope of fabric and medium priority.
To add a traffic class to a vFlow, create a vFlow and assign a traffic class. In this case the flow
is for a single IP address:

CLI network-admin@switch > vflow-create name losslessflow scope local src-ip

10.11.1.10 src-ip-mask 255.255.255.255 action none flow-class lossless

246
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > vflow-show name losslessflow layout vertical

switch: aquila12
name: losslessflow
scope: local
type: vflow
vlan: 0
:
in-port:
out-port:
ether-type: 0
src-ip: 10.11.1.10
dst-ip:::
src-port: 0
dst-port: 0
proto: ip
flow-class: lossless
bw-max: 0
pri: 0
action: none
action-value: 0
transient: no

Traffic from IP address 10.11.1.10 now has a very high priority throughout the switch. For a
similar high priority throughout the fabric use scope fabric rather than scope local.
When a TCP session goes through the NPU, and capacity is exceeded, the return traffic with
TCP ACK packets can get dropped from the session. To avoid this, create a flow that matches
the TCP ACK packets and set a higher precedence for it.

Applying CoS Queue Mapping based on Re-Marked DSCP in vFlow

Currently, Netvisor OS allows a vFlow to mark or re-mark matched packets with a DSCP
value on egress. As a result, Netvisor OS does not prioritize any of the traffic in terms of the
egress port CoS queue selected for transmit. Another feature, Enabling DSCP to Priority and
CoS Mappings introduces the ability to create DSCP QoS maps and apply to ports, but the
maps apply to ingress packets. This feature introduces the ability prioritize traffic based on
the remarked DSCP value in a vFlow
Netvisor OS creates named DSCP maps as independent objects, and applies the maps to
ingress ports for prioritization of packets based on the DSCP markings. In this feature, you
can apply the same maps in a vFlow. QoS maps can be applied to ports, but not to Flow
Processor entries corresponding to vFlows. This implementation does the prioritization
explicitly, since flows can be configured with CoSQ values. The implementation has the
following features:
Verify the DSCP map named in the vFlow exists.
Determine the priority and CoS for the DSCP value assigned to the vFlow.
Apply this CoS value to the Flow Processor entry in hardware.
Reconfigure CoS in the flow when the vFlow DSCP setting changes.
Prevent deleting a DSCP map in use by a vFlow.
Update the CoS setting of vFlows using the DSCP map when the DSCP map priority
settings are updated.
You can specify the name of a DSCP map in the vflow-create command:

Pluribus Networks
www.pluribusnetworks.com
247
dscp-map dscp-map name | none Specify the DSCP map to apply on the flow. Please
reapply if map priorities are updated.

Configuring Burst Size in vFlow for Maximum Bandwidth

The vflow-create and vflow-modify commands support a configurable burst-size
parameter. The requirement of a flow-class for attaching a meter by configuring the bw-max
parameter to a vFlow is removed.
This feature is useful because you can now specify different burst-sizes for different types of
metered traffic. For example, you can configure higher burst levels for a metered application
that may produce bursty traffic patterns when you click on it, such as a media-rich Web page
link.
This feature defaults to burst-size auto, which auto-calculates the burst size based on the
maximum bandwidth settings for the vFlow. You can configure a burst-size number between
256B through 128MB.

CLI network-admin@switch > vflow-create name name-string scope local|fabric

in-port port-list bw-max bw-max-number burst-size number

For example, to create a vFlow with a burst size of 12 MB, use the following syntax:

CLI network-admin@switch > vflow-create name flow1 scope local in-port 12

bw-max 5G burst-size 12M

Displaying Multiple Objects for Show Commands

In previous versions of software, Netvisor could not display multiple objects for show
commands. Netvisor displayed either one object or all objects in show ouput. For example,
the show command, vflow-show, displayed all vFlows or just one specified vFlow.
Now, you can specify multiple objects to display. For example, for vflow-show, you can
specify which vFlows to display:
CLI (network-admin@Leaf1)>vflow-show name [TAB]
Internal-Keepalive
System-S
System-F
System-R
System-VLE-S
System-VLE-F
System-VLE-R
VIRT_WIRE_MAS_SLV_17_19
VIRT_WIRE_SLV_MAS_19_17
VIRT_WIRE_MAS_SLV_18_20
VIRT_WIRE_MAS_SLV_20_18

Now you can specify the vFlows to display:

248
Pluribus Networks
www.pluribusnetworks.com
CLI (network-admin@Leaf1)>vflow-show name System-S,System-R
switch name scope type proto tcp-flags precedence action
action-to-ports-value enable
-------- -------- ----- ------ ----- --------- ---------- -----------
--------------------- ------
dorado05 System-S local system tcp syn default copy-to-cpu none
enable
dorado05 System-R local system tcp rst default copy-to-cpu none
enable

CLI (network-admin@Leaf1)>cpu-class-show name arp,dhcp,l3-miss

switch name scope rate-limit hog-protect hog-protect-support queue
-------- ------- ----- ---------- ----------- ------------------- -----
Leaf1 arp local 1000 disable supported 21
Leaf1 dhcp local 1000 disable none 24
Leaf1 l3-miss local 1000 disable none 10

Support for Policy-based Routing

Policy-based Routing (PBR) enables flexible packet forwarding and routing through user
defined policies. Unlike traditional routing based on destination IP address only, PBR allows
you to define routes based on other parameters such as source and destination IP addresses,
protocol, or souce and destination port numbers.
Policy-based routes can match packets based on the following criteria:
All Layer 4 and Layer 3 fields similar to those in vFlow configuraions.
Policy based routes are higher priority than static and dynamic routes.
If no match or next-hop is not resolved, then traffic is dropped until the next-hop is
resolved.
You configure PBR using vFlow commands. Internally, policy routing of the packets uses a
vFlow entry. PBR vFlow entries are created in a new vFlow table, System-L3-L4-PBR.
To enable PBR, use the following command:
(CLI network-admin@Spine1)>system-settings-modify policy-based-routing

To disable PBR, use the following command:

(CLI network-admin@Spine1)>system-settings-modify no-policy-based-routing

To display the vFlow table, use the following command:

(CLI network-admin@Spine1)>vflow-table-show
switch name flow-max flow-used flow-tbs-slices
capability flow-profile
-------- -------------------- -------- --------- --------------- ----------
--- ----------------
Spine1 System-L3-L4-PBR-1-0 set-metada
ta system=>PBR Table

Pluribus Networks
www.pluribusnetworks.com
249
Now you configure a vFlow for the routing policy, using the following syntax:
(CLI network-admin@Spine1)>vflow-create name name-string vrouter-name
name-string scope local next-hop-ip gateway-ip-address table-name
System-L3-L4-PBR-1-0

You can only specify the scope as local.

CLI network-admin@switch >

Using Application Flows and Statistics

Displaying Standard Statistics
You can display standard statistics that consist of flow-based information collected and
tracked continuously by the switch.
To show connection-level statistics, traffic flows between a pair of hosts for an application
service, including current connections and all connections since the creation of the fabric,
enter the following CLI command at the prompt:

CLI network-admin@switch > connection-stats-show

switch: pleiades24
mac: 00:e0:81:e4:02:12
vlan: 200
ip: 100.200.1.3
port: 53
iconns: 80
oconns: 0
ibytes: 0
obytes: 0
total-bytes: 0
last-seen-ago: 4d19h32m23s
switch: pleiades24
mac: 00:12:c0:80:1e:85
vlan: 200
ip: 100.200.1.4
port: 16
iconns: 0
oconns: 70684
ibytes: 578M
obytes: 890M
total-bytes: 1.43G
last-seen-ago: 46s

From the information displayed in the output, you can see statistics for each switch, VLANs,
client and server IP addresses, as well as the services on each connection. Latency and other
information is also displayed.
The latency(us) column displays the running latency measurement for the TCP connection in
microseconds. It indicates end-to-end latency and includes the protocol stack processing for
the connected hosts and all intermediary network hops.

250
Pluribus Networks
www.pluribusnetworks.com
This is not the same latency measurement experience by a packet transiting the switch
port-to-port. The port-to-port latency is platform-dependent and you should refer to the
datasheet for your switch model.
To display specific types of connections, use the additional parameters with the command.
For instance to display VLANs of connections,

CLI network-admin@switch > connection-stats-show vlan

switch vlan vxlan client-ip server-ip service active age

switch12 1 0 10.9.10.152 96.17.77.96 http yes 35m27s
switch12 5 0 10.12.1.47 10.9.10.204 445 yes 7m56s
switch12 1 0 10.9.9.21 23.62.97.88 http yes 3m41s
switch12 1 0 10.9.9.21 23.60.129.224http yes 3m44s
switch12 1 0 10.9.10.72 10.9.99.23 http yes 7s
. . .

To display a summary of traffic statistics for each application service, use the
service-stats-show command.

CLI network-admin@switch > service-stats-show

switch service bytes

pleiades24 53495 584
pleiades24 8084 845M
pleiades24 59475 33.9K
pleiades24 imap 1.83M
pleiades24 35356 106
pleiades24 54341 584

From the information displayed in the output, you can review each switch, service, and the
number of bytes used by each service.

Understanding vFlow Statistics

Virtual network-based flows, vflows, display statistics for packet traffic flows on a switch and
across the
fabric. vFlows are very powerful and provide many features such as quality of service (QoS),
traffic shaping,packet redirect, drop actions, mirror, and capture.
A vFlow can be configured to store log statistics to a file accessible to clients using NFS and
SFTP. If statistics logging is enabled, Netvisor OS periodically polls the switch for the most
recent statistics for each flow and saves the statistics to an exported file. Netvisor OS also
saves individual statistics received from other switches in the fabric and combines the
statistics from all switches to record aggregate statistics for the entire fabric.
The switch consists of two components, the switch and the server. vFlows with operations like
drop are executed within the switch component. Some vFlows operations for QoS take place
in the switch component, while others operate within the co-processor by directing pertinent
traffic to the co-processor.

Pluribus Networks
www.pluribusnetworks.com
251
There, the traffic is managed and then sent back to the switch component.Other actions such
as copy-to-cpu sends the match traffic to the server component where the traffic is managed
and then forwards packets for delivery. In general, the details are managed by Netvisor OS
including fabric scope commands that cause all switches within a fabric to participate in an
operation and then sends the compiled results to the CLI or to log files.
Before you can access the files, you must enable NFS or SFTP access to the log files by using
the admin-service-modify command.

CLI network-admin@switch > CLI network-admin@switch > vflow-share-show

switch enable share-path

pleiades24 fab1-global no pleiades24://fab1-global
pleiades24 fab1-global no pleiades24://fab1-global
pleiades24 fab1-global no pleiades24:///fab1-global
pleiades24 fab1-global no pleiades24://fab1-global
pleiades24 fab1-global no pleiades24://fab1-global

CLI network-admin@switch > CLI network-admin@switch > vflow-share-modify

fab1-global enable

vflow-share-show
switch enable share-path
pleiades24 fab1-global yes pleiades24://fab1-global
pleiades24 fab1-global no pleiades24://fab1-global
pleiades24 fab1-global no pleiades24://fab1-global
pleiades24 fab1-global no pleiades24://fab1-global
pleiades24 fab1-global no pleiades24://fab1-global

You can then access the statistics log files using NFS in the following locations:
For the switch scope, the files are located in
/net/switch-name//-name/flow/flow-name/switch/
switch-name/stats
For the fabric scope, the files are located in
/net/switch-name//-name/flow/flow-name/fabric/
stats
To create a vFLow for example, Host-Agent-Discover, and measure statistics, enter the
following command:

CLI network-admin@switch > CLI network-admin@switch > vflow-create name

Host-Agent-Discover scope local system

To view all vFlows currently tracked by the switch or fabric, use the vflow-show command:

252
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > vflow-show

switch: pleiades24
name: Host-Agent-Discover
scope: local
type: system
dst-ip: 224.4.9.6
precedence: 2
action: copy-to-cpu
switch: pleiades24
name: DHCP-client
scope: local
type: system
in-port: 1-68
src-port: 68
proto: udp
precedence: 2
action: copy-to-cpu
switch: pleiades24
name: Host-Agent-Discover
scope: local
type: system
dst-ip: 224.4.9.6
precedence: 2
action: copy-to-cpu
switch: pleiades24
name: DHCP-client
scope: local
type: system
in-port: 1-68
src-port: 68
proto: udp
precedence: 2
action: copy-to-cpu

From the information displayed in the output, you can review the switch, the name of the
vFlow, scope, type of vFlow, destination IP address, precedence, and action for the vFlow.
To display statistics for all vFlows, use the vflow-stats-show command:

CLI network-admin@switch > vflow-stats-show

switch name packets bytes cpu-packets cpu-bytes

------ ---- ------- ----- ----------- ---------
pleiades24IGMP-Flow 368K 23.0M 392K 23.0M
pleiades24 LLDP-Flow 82.9K 26.3M 82.9K 26.0M
pleiades24 Host-Agent 17.8K 1.11M 0 0
pleiades24 ECP 0 0 0 0

To monitor statistics of a vFlow and update every 10 seconds, use the following syntax:

CLI network-admin@switch > vflow-stats-show name flow1 show-diff-interval 10

To log persistent records of flow statistics, use the logging parameter and collect statistics
every 10 seconds:

Pluribus Networks
www.pluribusnetworks.com
253
CLI network-admin@switch > vflow-create name monitor-flow scope local
ether-type arp stats log stats-interval 5

You can display the statistics logs for the new flow using the vflow-stats-show command.

Informational Note: Conflicting vFlows

Multiple vFlows can be active at once, but Netvisor OS cannot apply them
at the same time. You can use the precedence parameter is used to set the
order of the vFlows. If you set the precedence to a higher value (0 - 10
with 0 as the lowest precedence), the vFlow has a higher precedence than
those with lower values. If you’re seeing error messages about vFlow
conflicts, try adding a precedence value to new or existing vFlows.

Creating vFlows with the Scope Fabric

To create vFlows across the entire fabric, configure the vFlow with the scope fabric and stats
enable option. Using these parameters enables statistics for the flow on all switches that are
members of the fabric and you can display the statistics for any switch in the fabric.
To create a vFlow for VLAN1 with the scope fabric, use the following syntax:

CLI network-admin@switch > vflow-create name fab_flow1 scope fabric stats

enable vlan 1

To display the statistics for the new vFlow for a switch in the fabric, use the following syntax:

CLI network-admin@switch > switch switch-name vflow-stats-show name fab_flow1

name packets bytes cpu-packets cpu-bytes

---- ------- ----- ----------- ---------
fab_flow1 51.4K 13.8M 50.1K 13.1M

If you omit the switch name, all vFlow statistics for the fabric are displayed.
switch name packets bytes cpu-packets cpu-bytes
------ ---- ------- ----- ----------- ---------
pleiades1 fab_flow1 1.32K 305K 1.29K 291K
pleiades2 fab_flow1 910 256K 884 243K

Example Use Cases for vFlows

The following examples illustrate how to use vFlows to impact traffic on the switch. You can
regulate bandwidth, create multiple vFlows, or share bandwidth.

Creating Multiple vFlows

1. You can create multiple vFlows and add precedence values to the vFlows. The packet is
matched to the vFlow with the highest precedence. Create the first vFlow:

CLI network-admin@switch > vflow-create name client-flow1 scope fabric bw

flow-class meter bw-max 2g

2. Create the second vFlow:

254
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > vflow-create name client-flow2 scope fabric bw
flow-class meter bw-max 5g src-ip 192.168.20.1

vflow-create: Flow conflicts with Flow client-flow1, ID68: specify fields to

make flows mutually exclusive or change the flow precedence

The error message is generated because the vFlow configurations conflict with each other.
To differentiate between the two flows, assign a different precedence to client-flow2:

CLI network-admin@switch > vflow-create name client-flow2 scope fabric bw

flow-class meter bw-max 5g src-ip 192.168.20.1 precedence 5

Configuring Bandwidth Sharing for a Single VLAN

In some instances, you may want to configure bandwidth sharing for a single VLAN with
different IP addresses or subnets. To do this, you must create a VRG with the required
bandwidth:

CLI network-admin@switch > vrg-create name admin-vrg vlans 100 data-bw-min 1g

data-bw-max 2g scope fabric

You have now created a VRG with the guaranteed bandwidth of 1 Gbps and limited to a
maximum of 2 Gbps. Now, create a vFLow for each IP address:

CLI network-admin@switch > vflow-create name vfl-1 scope fabric vlan 100 src-ip
1.1.1.1

CLI network-admin@switch > vflow-create name vfl-2 scope fabric vlan 100 src-ip
2.2.2.2

CLI network-admin@switch > vflow-create name vfl-3 scope fabric vlan 100 src-ip
3.3.3.3

CLI network-admin@switch > vflow-create name vfl-4 scope fabric vlan 100 src-ip
4.4.4.4

In this example, the specified IP addresses each have a guaranteed bandwidth between 1
Gbps and 2 Gbps.
If you want to specify a subnet, 100.100.100.0/28, and VLAN 53 with maximum bandwidth
of 50 Mbps, use the following syntax:

CLI network-admin@switch > vrg-create name vrg-custom scope fabric data-bw-min

50M data-bw-max 50M vlan 53

CLI network-admin@switch > vflow-create name vfl-cust scope fabric src-ip

100.100.100.0 src-ip-mask 255.255.255.240 vlan 53

But later on, you found that sixteen IP addresses were not enough and you needed an
additional 8 with the subnet, 101.101.101.8/29 that require the same bandwidth as the
previous subnet. Use the following syntax:

CLI network-admin@switch > vflow-create name vfl-cust-2 scope fabric src-ip

101.101.101.8 src-ip-mask 255.255.255.248 vlan 53

You now have two vFlows on VLAN 53.

Pluribus Networks
www.pluribusnetworks.com
255
Then, you discover that 50 Mbps is not sufficient to support the network traffic affected by
the vFlow, and you want to upgrade to 80 Mbps:

CLI network-admin@switch > vrg-modify name vrg-custom data-bw-min 80M

data-bw-max 80M

CLI network-admin@switch >

Configuring vFlows in Virtual Wire Mode

vFlows can be configured on Virtual Wire platforms. You can configure a vFLow to store log
statistics to a file accessible to clients using NFS and SFTP. If statistics logging is enabled,
Netvisor OS periodically polls the switch for the most recent statistics for each flow and saves
the statistics to an exported file. Netvisor OS also saves individual statistics received from
other switches in the fabric and combines the statistics from all switches to record aggregate
statistics for the entire fabric.

Support for TCP Parameters using vFlows

Packet Broker requires the ability to create flows based on TCP control bits in a packet. The
commands, vflow-create and vflow-modify have a new option tcp-flags. The supported
TCP control bits include FIN, SYN, RST, PUSH, ACK, and URG.
Setting the ACK bit is supported only if it is combined with other TCP bits such as SYN and
FIN and not as a single parameter.
Only to-port and mirror actions are supported by vFlow with tcp-flags filter. The actions
added for vFlows with tcp-flags configured are mirror-to-port. If analytics is enabled, then
copy-to-cpu are also applied on the same vFlow. Also, these flows are created with a
precedence of 3 or above. System vFlows are created with precedence 2 so that analytics can
also work even with these vFlows.
To create a vFlow for the default system table, use the following syntax:
CLI (network-admin@Spine1)>vflow-create name Redirect-TCP-Reset tcp-flags RST
action to-port
CLI(network-admin@Spine1)>vflow-create name Redirect-TCP-ECN-Capable tcp-flags
ECN,RST action to-port
CLI(network-admin@Spine1)>vflow-create name Mirror-TCP-Finished tcp-flags FIN
action mirror

256
Pluribus Networks
www.pluribusnetworks.com
You can use the vflow-table-show command to display vFlow tables:
CLI (network-admin@Spine1)> vflow-table-show format all layout vertical
switch: Spine1
name: Egress-Table-1-0
id: a0000d7:1
flow-max: 1024
flow-used: 0
flow-tbl-slices: 1
capability: match-metadata
flow-tbl-bank: Egress
flow-profile: system
switch: Spine1
name: Decap-Table-1-0
id: a0000d7:2
flow-max: 1024
flow-used: 0
flow-tbl-slices: 2
capability: none
flow-tbl-bank: Match-Metadata
flow-profile: vxlan
switch: tac-f64-sw5
name: OpenFlow-L2-L3-1-0
id: a0000d7:3
flow-max: 1024
flow-used: 0
flow-tbl-slices: 7
capability: none
flow-tbl-bank: Match-Metadata
flow-profile: openflow

Configuring vFlows with User Defined Fields (UDFs)

A User Defined Field (UDF) can match up to 128 bytes of a packet starting from the first byte
of the packet. The relative offset can be given to the match location. The length of the match
can be from 1 to 4 bytes. Hardware with a Trident chip supports the creation of 8 UDF IDs.
Each id can match a 2 byte portion of a packet. Creating a UDF with a length of 3 or 4 bytes
requires 2 UDF IDs whereas a UDF with length of 1 or 2 bytes required 1 UDF id. The length
specified for each UDF determines the total number of UDFs supported by Netvisor OS. If you
specify a length of 3 or 4 bytes, a maximum of 4 UDFs can be created. If you specify a length
of 1 or 2 bytes, a maximum of 8 UDFs can be created.
A UDF adds a qualifier to the vFlow group, and you should create all UDFs before creating
any vFlows.
This feature is disabled by default, and you must enable it using the following command:
CLI(network-admin@Spine1)>vflow-settings-modify enable-user-defined-flow
You must reboot Netvisor OS for the parameter to take effect on the platform.
To disable the feature, use the following command:
CLI(network-admin@Spine1)>vflow-settings-modify no-user-defined-flow
A new command, udf-create, adds the qualifier to the UDF group in the hardware. This
allocates UDF IDs based on the length. The command, vflow-create, also has new fields to
provide the data and mask to be matched by the vFlow. You can create vFlows with either
one or two UDFs.
You cannot modify a UDF after adding it to a vFlow. You must delete the vFlow, modify the
UDF, and re-create the vFlow with the modified UDF.

Pluribus Networks
www.pluribusnetworks.com
257
New Commands for UDF
To create a new UDF, use the following command:
CLI(network-admin@Spine1)>udf-create name u1 scope local offset 10 length 2
header packet-start

udf-create Create the UDF qualifier list

name name-string Create the UDF name
scope local|fabric Scope for the UDF
offset number-bytes The offset in bytes. This is a value between 1 and
128.
length number-bytes The length in bytes. This is a value between 1 and 4
bytes.
header The header from where offset is calculated.
packet-start|l3-outer|l3-inner|l4
-outer|l4-inner

CLI(network-admin@Spine1)>udf-delete name u1

udf-delete Delete UDF qualifier list

name name-string The name of the UDF to delete.

CLI(network-admin@Spine1)>udf-modify name u1 scope local offset 20 length 4

header packet-start
udf-modify Modify UDF qualifier list
name name-string The name of the UDF to modify.
one or more of the following
options:
offset number-bytes The offset in bytes. This is a value between 1 and
128.
length number-bytes The length in bytes. This is a value between 1 and 4
bytes.
header The header from where offset is calculated.
packet-start|l3-outer|l3-inner|l4
-outer|l4-inner

CLI(network-admin@Spine1)>udf-show
switch name scope offset length header
------ ---- ----- ------ ------ ------------
k2 u1 local 20 4 packet-start
k2 u2 local 24 4 packet-start

udf-show Displays the UDF qualifier list

name name-string Displays the UDF name
scope local|fabric Displays the scope for the UDF
offset number-bytes Displays the offset in bytes. This is a value between
1 and 128.

258
Pluribus Networks
www.pluribusnetworks.com
length number-bytes Displays the length in bytes. This is a value
between 1 and 4 bytes.
header Displays the header from where the offset is
packet-start|l3-outer|l3-inner|l4 calculated.
-outer|l4-inner

The command, vflow-create, has the following new parameters:

udf-name1 udf-name Specify the name of the UDF.
udf-data1 udf-data1-number Specify UDF data1q with the format 0xa0a0a01
udf-data1-mask Specify he mask for udf-data with the format
udf-data1-mask-number 0xffffffff.
udf-name2 udf-name Specify the name of the UDF.
udf-data2 udf-data2-number Specify UDF data2 with the format 0xa0a0a01
udf-data2-mask Specify the mask for udf-data with the format
udf-data2-mask-number 0xffffffff.

CLI(network-admin@Spine1)>vflow-create name vf scope local udf-name1 u1

udf-data 0x0a0a0a01 udf-data-mask1 0xffffffff udf-name2 u2 udf-data2
0x0a0a1400 udf-data-mask2 0xffffff00

CLI(network-admin@Spine1)>vflow-show

switch name scope type precedence udf-name1 udf-data1 udf-data-mask1

------ ---- ----- ----- ---------- --------- --------- --------------
K2 vf local vflow default u1 0xa0a0a01 0xffffffff

udf-name2 udf-data2 udf-data-mask2 enable

--------- --------- -------------- ------
u2 0xa0a1400 0xffffff00 enable

Configuring DSCP to CoS Mapping

Netvisor OS supports creating Quality of Service (QoS) maps that configure hardware based
mapping of Differentiated Services Code Point (DSCP) value in a received IP header to a Cost
of Service (CoS) priority. This helps in prioritizing traffic based on DSCP markings by using
the appropriate egress CoS queues to send packets out.
DSCP value is the 6 upper bits in the 8-bit ToS field of an IP header. Details about the specific
values and the proposed traffic disposition can be found in these RFCs :
RFC 2474 (DS Fields Definitions)
RFC 2475 (DiffServ architecture)
RFC 2597 (AF PHB Group)
RFC 2780 (IANA Allocation Guidelines)
A quick summary of DSCP in Netvisor OS:
DSCP values range from 0 to 63, while packet priorities map to 8 CoS values or
priority queues.
Standards (IANA) include specific values in their guidelines. These values are used by
different vendors to facilitate interconnectivity.
Class selector code points (CS0 through CS7, multiples of 8) are backwards
compatible with IP ToS values. These values also serve as base selectors for other
values.

Pluribus Networks
www.pluribusnetworks.com
259
 Assured Forwarding (AF) code points have 4 priority classes, each class has three
code points indicating the drop precedence.
Class1: AF11/12/13 (DSCP 10, 12, 14)
Class2: AF21/22/23 (DSCP 18, 20, 22)
Class3: AF31/32/33 (DSCP 26, 28, 30)
Class4: AF41/42/43 (DSCP 34, 36, 38

0 is best effort (CoS 0, default)

46 is an Expedited Forwarding (EF) code point, indicating critical traffic.
There are new commands to support this feature:

dscp-map-create Create a DSCP priority mapping table with default

DSCP to priority mappings.
name name-string Create a name for the DSCP map

dscp-map-delete Delete a DSCP priority mapping table.

name name-string The name of the DSCP map to delete.

dscp-map-show Display a DSCP priority mapping table

name name-string Display the name of the DSCP map.
This command displays output only if there are maps configured.

dscp-map-pri-map-modify Update priority mappings in tables.

dscp-map selector: Specify the name for the DSCP map to modify.
name name-string
the following pri-map
arguments:
pri number Specify a CoS priority from 0 to 7.
dsmap number-list Specify a DSCP value(s)as a single value, comma
separated list, or a number range.

dscp-map-pri-map-show Display priority mappings in tables.

dscp-map selector: Display the name of the DSCP map.
name name-string
the following pri-map
arguments:
pri number Display a CoS priority from 0 to 7.
dsmap number-list Display a DSCP value(s)a DSCP value(s)as a single
value, comma separated list, or a number range.
The dscp-map-pri-map-show displays output only if there are maps configured.

260
Pluribus Networks
www.pluribusnetworks.com
The default values are listed in the following dscp-map-pri-map-show output:
CLI (network-admin@Spine1)>dscp-map-pri-map-show name dscp-map1
switch name pri dsmap
------- ---- --- -----------
Spine1 ds2 0 none
Spine1 ds2 1 8,10,12,14
Spine1 ds2 2 16,18,20,22
Spine1 ds2 3 24,26,28,30
Spine1 ds2 4 32,34,36,38
Spine1 ds2 5 40
Spine1 ds2 6 48
Spine1 ds2 7 56

The command, port-config-modify, has a new parameter, dscp-map map-name|none to

support this feature. Using the option none deletes or cancels a DSCP map previously
configured on the port.

Configuring Priority-based Flow Control

Priority Flow Control (PFC) is an IEEE standard (802.1qbb) for link level flow control on
Ethernet networks. Functionally, this feature is similar to the IEEE standard 802.3 for PAUSE
mechanism, except that it operates at the granularity of individual packet priorities or traffic
class, instead of port level. When a queue corresponding to traffic with a particular traffic
class reaches a predetermined threshold, either auto or statically set, the switch chip
generates a PFC frame and sends it back to the sender. For PFC to work effectively end to end
on the network, all switches and hosts in the traffic path are configured to enable PFC, and
configured for traffic class to priority mappings.
Netvisor OS introduces a new command to configure priorities, or traffic classes, for PFC. The
configuration allows you to add ports where PFC is enabled. When enabled, Netvisor OS
performs traffic class to CoS queue mappings, as well as to packet priorities, in the
background. The following mappings are performed:
 1 to 1 traffic class to CoS queue mapping (0 through 7)

 1 to 1 packet priority (802.1p) mapping

PFC is enabled to both transmit and receive on the selected port. For transmit, Netvisor OS
pauses traffic corresponding to the traffic class indicated in the received PFC frame. For
receive, Netvisor OS generates a PFC frame when a queue corresponding to a traffic class
reaches the pause threshold. Netvisor OS auto-configures parameters such buffer
threshold,and pause timer value. Disabling PFC turns off PFC for receive and transmit,
although the traffic class priority and queue mappings remain.
On switches with a Broadcom Trident II chip, even with ingress admission control enabled (in
lossless mode), by default, only the traffic class or priority group 7 is set up with the memory
management unit (MMU) buffer resources. Packets of all priorities utilize the resources of the
default priority group unless specifically configured. This implies that when enabling a new
priority group for PFC, the buffer configuration is generated and saved in the chip
configuration file, which is read during system initialization for MMU setup. As a result, when
you enable a new priority for PFC, you must restart Netvisor OS. Adding new ports to an
existing priority group setting, for another port or ports, does not require restarting Netvisor
OS.
Up to three priority group buffer settings can be configured on switches in Netvisor OS. If you
attempt to configure more than three, Netvisor OS returns an error message.

Pluribus Networks
www.pluribusnetworks.com
261
To create a new PFC configuration on port 2 with a priority group of 2, use the following
command:
CLI (network-admin@Spine1)>port-pfc-create priority 2 port 1-10
Priority configuration will be effective after restart.

To modify the ports and change them to 11-15, use the following command:
CLI (network-admin@Spine1)>port-pfc-modify priority 2 port 11-15
Priority configuration will be effective after restart.
To delete the configuration, use the following command:
CLI (network-admin@Spine1)>port-pfc-delete priority 2 port 11-15

To display the configuration, use the port-pfc-show command:

CLI (network-admin@Spine1)>port-pfc-show
switch priority port error
------- -------- ----- -----
Spine1 2 11-20
Spine1 3 11-20

Configuring Priority-based Flow Control Port Statistics

Priority-based Flow Control (PFC) was introduced in Netvisor OS in Version 2.5.4, but the
feature implementation did not include displaying statistics related to PFC. It is helpful to
view the stats to confirm or debug traffic characteristics when PFC is in use. New commands
allow you to display PFC stats per port and adjust the statistics collection.
(CLI network-admin@Spine1)>port-pfc-clear

port port-list Specify the ports to delete PFC statistics.

(CLI network-admin@Spine1)>port-pfc-stats-settings-modify

enable|disable Specify if you want to enable or disable PFC statistics

collection.
interval duration: #d#h#m# Specify the interval between statistics collection.
disk-space disk-space-number Specify the amount of disk space for statistics collection.
(CLI network-admin@Spine1)>port-pfc-stats-settings-show

enable|disable Specify if you want to enable or disable PFC statistics

collection.
interval duration: #d#h#m# Specify the interval between statistics collection.
disk-space disk-space-number Specify the amount of disk space for statistics collection.
(CLI network-admin@Spine1)>port-pfc-stats-show

time date/time: Displays the date and time for statistics collection.
yyyy-mm-ddTHH:mm:ss
start-time date/time: Displays the start date and time for statistics collection.
yyyy-mm-ddTHH:mm:ss
end-time date/time: Displays the end date and time for statistics collection.
yyyy-mm-ddTHH:mm:ss
duration duration: #d#h#m#s Displays the duration for statistics collection.
interval duration: #d#h#m#s Displays the interval between statistics collection.
since-start Displays the statistics since the start time.

262
Pluribus Networks
www.pluribusnetworks.com
older-than duration: #d#h#m#s Displays the statistics older than the specified time.
within-last duration: #d#h#m#s Displays the statistics within a specified time.
port port-list Displays the port list.

About sFlow
Because businesses rely on network services for mission critical applications, small changes
in network usage can impact network performance and reliability. As a result, these changes
can also impact a business’ ability to conduct key business functions and increase the cost of
maintaining network services.
Figure 1: Overview of sFlow

sFlow provides the visibility into network usage and active routes on the network by
providing the data required to effectively control and manage network usage. This ensures
that network services provide a competitive edge to the business.
A few examples of sFlow applications include the following:
Detecting, diagnosing, and fixing network problems
Real-time congestion management
Understanding application mixes such as P2P, Web, DNS
Usage accounting for billing

Pluribus Networks
www.pluribusnetworks.com
263
 Audit trail analysis to identify unauthorized network activity and trace sources of
Denial of Service (DoS) attacks
Route profiling and optimizing peers
Trending and capacity planning
sFlow is an open source sampling tool providing constant traffic flow information on all
enabled interfaces simultaneously. sFlow data is sent to a collector that formats the data into
charts and graphs while recording and identifying trends on the network. You can use this
information for troubleshooting a network, perform diagnostics, and analysis of data.
The sFlow agent on the switch samples packets from data flows and forwards headers of the
sample packet to a collector at regular intervals. You can specify the number of packets to
sample from the total packets which is called the sample rate. The packets are stored and
sent to the collector at an interval that you can configure on the switch. This is called the
polling interval. You can sample different types of packets such as frames sent to the CPU or
interfaces of the switch, routed packets, flooded packets, and multicast packets. However,
the following packet types are not sampled by sFlow:
LACP frames
LLDP frames
STP RPDUs
IGMP packets
Ethernet PAUSE frames
Frames with CRC errors
PIM_HELLO packets
Packets dropped by ACLs
Packets dropped as a result of VLAN violations
Routed packets with IP options or MTU violations

Configuring the sFlow Collector

Before configuring the sFlow agents, you must configure the sFlow collector. The sFlow
collector receives sFlow datagrams from the sFlow agents. In this example, the sFlow
collector has an IP address of 10.1.1.243, and a default port of 6343. The collector name is
net-man-all, and the scope is fabric. If the scope is fabric, then additional switches that
join the fabric receive the sFlow collector configuration. If the scope is local, then the sFlow
collector is configured only on one switch.

CLI network-admin@switch > sflow-collector-create collector-ip 10.1.1.243

collector-port 6343 name net-man-all scope fabric

You can add as many collectors as needed for your configuration.

Enabling sFlow on the Network

You must configure and enable sFlow on each switch that you want to use for monitoring
network traffic. You can only configure one sFlow per switch.
On each switch in the example diagram, use the following command to enable sFlow,
net-monitor, on ingress ports 57-59, sample type raw, sample-rate 4096, sample interval
5 seconds, trunc-length 160 bytes, on VLAN 200:

264
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > sflow-create name net-monitor sample-type raw ports
57-59 sample-rate 4096 trunc-length 160 vlan 200

Adding Additional Ports to sFlow

To add the ports, 61-62, to the sFlow configuration, you must use the following command on
each switch:

CLI network-admin@switch > sflow-port-add sflow-name net-monitor switch

10.1.1.23 ports 61-62

In this example, the IP address of the switch is used as the name of the switch.

Removing Ports from the sFlow Configuration

You can remove ports from the sFlow configuration by using the sflow-port-remove
command:

CLI network-admin@switch > sflow-port-remove sflow-name net-monitor switch

10.1.1.23 ports 61-62

Counter Sampling
For counter sampling, also called polling, the sFlow agent periodically polls the hardware
interface statistics registers, counters, in the switch chip for per port statistics, and stores
them in RAM until it is time to send the next message to the sFlow collector. Overall port
statistics such as the number of broadcasts, errors, are collected by the sFlow agent.
The agent then includes the statistics in the sFlow datagrams sent to the sFlow collector
along with the packet sampling information. From these statistics, the sFlow obtains
information about the actual utilization of each port. For instance, information about
broadcast to multicast to unicast rations is captured.
When you configure the agent for counter sampling, it sends an sFlow datagram at intervals
of a second, at most. The datagram contains a snapshot of the counters cached in RAM from
the most recent polling of interface counters.

Packet Sampling
Packet sampling is used to characterize network traffic. If the sFlow agent is configured for
packet sampling, the agent takes copies of random samples of packets forwarded within the
switch CPU and sends them to the switch for processing. The CPU sends a configured portion
of the sampled packet, containing a number of protocol headers and possibly some of the
payload data to the sFlow collector. Random sampling prevents the synchronization of
periodic traffic patterns. On the average, 1 in every N packets is captured and analyzed. The
sampling can apply to ingress and egress frames independently. The rate that the agent
sends datagrams depends on the sampling rate, the traffic rate, and the configured
maximum datagram size. Typically, several samples are included in the datagram.

Agent to Collector Datagrams

Pluribus Networks
www.pluribusnetworks.com
265
After gathering packet and counter samples, each sFlow agent creates a packet of the data
and sends it to an sFlow collector in UDP datagrams. The datagrams contain the IP address of
the sFlow collector and the standard UDP destination port number of 6343. Using a
standardized port helps avoid configuration between sFlow agents and collectors. If the sFlow
agent is configured for counter sampling or packet sampling, or both, an sFlow datagram can
contain either interface counters, packet samples, or a mixture of both.
The following table provides information about the contents of sFlow datagrams:
Packet Header Information
Version The sFlow version used on the
network.
IP Address Type An IPv4 or IPv6 address
Source IP Address The IP address of the sFlow agent
Sequence Number The sequence number of the
datagram
System Uptime The length of time that the system
is operational.
Sample Count The number of samples in the
datagram
Ingress Interfaces The ifindex of the switch port
where the packets entered the
agent.
Egress Interfaces The ifindex of the switch port
where the packets exited the
agent.
Sample dataset sFlow-specific parameters:
• Sequence Numbers
• Sampling Rate
• Total Packets available for
sampling
• Number of sampled packets
dropped because there was no
processing resource for them.
Packet Samples Packet sample information and
may contain several samples.
Packet data The sampled data that may
include the packet payload data
and the number on length of
protocol headers. This information
depends on the size of the size, up
to 200 bytes.
Counter Sample Counter statistical information -
fitted in where space permits.
If index The ifindex of the interface related
to the counters.

266
Pluribus Networks
www.pluribusnetworks.com
 Packet Header Information
Physical Interface • Speed
Parameters • Duplex mode
• Admin status
• Operational status of the
interface
In Counters • ifInOctets
• ifInUnicastPkts
• ifInMultiPkts
• ifInBroadcastPkts
• ifInDiscards
• ifInErrors
• ifInUnknownProbs
Out Counters • ifOutOctets
• ifOutUcastPkts
• ifOutDiscards
• ifOutErrors
Promiscuous Mode The private VLAN promiscuous
mode of the interface
Ethernet Statistics • Alignment Errors
• FCS Errors
• SQE Errors
• Deferred Transmission
• Internal MAC errors
• Carrier sense errors
• Overlength frame errors
• Symbol errors

Analyzing Live Traffic Using Wireshark

Wireshark is a well known network protocol analyzer and one of many applications used for
network protocol analysis. Wireshark can interactively browse packet data from a live
network or from a previously save pcap file.

Informational Note:You can download Wireshark from

http://www.wireshark.org

Pluribus Networks
www.pluribusnetworks.com
267
To use Wireshark to decode a previously saved packet flow capture file, export the file from
the switch and analyze it with Wireshark.

Informational Note:
The path to a Netvisor switch pcap file has the format:
/net/<ServerSw_Name>/ONVL/global/flow/<Flow_Name>/<Switch_Na
me>/pcap

Using Wireshark to Analyze Packets in Real Time

To use Wireshark to interactively analyze packets in real time, you need to capture a packet
traffic flow, either on a specific switch or across the entire fabric using the scope option.
Include the log-packets option to send packets to the associated pcap files, for example

CLI network-admin@switch > vflow-snoop scope fabric src-ip 112.168.3.105 action

copy-to-cpu log-packets

Next, create a fifo on the host running Wireshark.

mkfifo /tmp/pcap

Start Wireshark, and select Options from the Capture menu.

Enter the fifo path that you created in the Interface field: /tmp/pcap

268
Pluribus Networks
www.pluribusnetworks.com
Figure 2:Wireshark Capture Options

Use tail to copy the pcap file to the FIFO:

tail +0f \
/net/ServerSw_Name//global/flow/Flow_Name/switch/Switch_Name/
pcap/tmp/pcap

You need to substitute ServerSw_Name, Flow_Name and Switch_Name to match your

environment. Live capture continues until the packet capture file is rotated. By default, the
maximum packet capture file size is 10MB but it is configurable with the packet-log-max
option of the vflow-create and vflow-modify commands.

Informational Note:
The mkfifo command used in this task is a standard feature of
Linux-like operating systems, including MacOS. For Windows platforms,
you may need to install the GNU CoreUtils package available at
http://gnuwin32.sourceforge.net/packages/coreutils.htm.

Pluribus Networks
www.pluribusnetworks.com
269
Internet Protocol Flow Information Export (IPFIX)
IPFIX (Internet Protocol Flow Information Export) is an IETF protocol created by the need for
a common, universal standard of export for Internet Protocol flow information from routers,
probes and other devices that are used by mediation systems, accounting/billing systems
and network management systems to facilitate services such as measurement, accounting
and billing. The IPFIX standard defines how IP flow information is to be formatted and
transferred from an exporter to a collector.

IPFIX Architecture
A Metering Process collects data packets at an Observation Point, optionally filters them and
then aggregates information about these packets. Using the IPFIX protocol, an Exporter then
sends this information to a Collector. Exporters and Collectors are in a many-to-many
relationship as one Exporter can send data to many Collectors and one Collector can receive
data from many Exporters.

IPFIX Protocol
IPFIX considers a flow to be any number of packets observed in a specific timeslot and
sharing a number of properties such as same source, same destination, or same protocol.
Using IPFIX, devices such as routers can send information to a central monitoring station
about their view of a potentially larger network.
IPFIX is a push protocol, meaning each sender periodically sends IPFIX messages to
configured receivers without any interaction by the receiver.
The actual makeup of data in IPFIX messages is largely up to the sender. IPFIX introduces
the makeup of these messages to the receiver with the help of special Templates. The sender
also accepts user-defined data types in the messages, so the protocol is freely extensible and
can adapt to different scenarios.
IPFIX prefers the Stream Control Transmission Protocol (SCTP) as the transport layer
protocol, but also allows the use of the Transmission Control Protocol (TCP) or User Datagram
Protocol (UDP). SCTP provides some of the same service features of both TCP and UDP. SCTP
is message-oriented like UDP and ensures reliable, in-sequence transport of messages with
congestion control like TCP. It differs from the two protocols when providing multi-homing
and redundant paths to increase resilience and reliability.

IPFIX Collector
Flow collectors are able to dynamically read the templates exported by flow capable
hardware and store the flows being sent. Most IPFIX collectors provide reporting on the data
and some even provide behavior analysis to help detect network threats.
On each Pluribus switch, Netvisor OS embeds a real-time non-sampled IPFIX metering
process, and each switch can be configured as an IPFIX exporter. In addition, Netvisor OS
supports exporting to multiple collectors.

270
Pluribus Networks
www.pluribusnetworks.com
Bidirectional Flow Support
Pluribus Networks supports bidirectional flows for IPFIX in that every flow record contains the
attribute of both endpoints. Many flow analysis tasks benefit from association of the
upstream and downstream flows of a bidirectional communication, for example, separating
answered and unanswered TCP requests, calculating round trip times, and more. Metering
processes that are not part of an asymmetric routing infrastructure, especially those
deployed at a single point through which bidirectional traffic flows, are well positioned to
observe bidirectional flows (Biflows). In such topologies, the total resource requirements for
Biflow assembly are often lower if the Biflows are assembled at the measurement interface
as opposed to the IPFIX Collector. The IPFIX Protocol requires only information model
extensions to be complete as a solution for exporting Biflow data.

Information Elements
Information in messages of the IPFIX protocol is modeled in terms of Information Elements
of the IPFIX information model.
All Information Elements specified for the IPFIX protocol has the following properties defined:
name - a unique and meaningful name for the Information Element
elementId - A numeric identifier of the Information Element. If this identifier is used
without an enterprise identifier, then it is globally unique, and the list of allowed
values is administered by IANA. It is used for compact identification of an Information
Element when encoding Templates in the protocol.
description - The semantics of this Information Element. It describes how the
Information Element is derived from the Flow or other information available to the
observer. Information Elements of dataType string or octetArray that have length
constraints such as fixed length, minimum and/or maximum length, state these
constraints in the descriptions.
dataType - One of the types listed in DataTypes or registered in the IANA "IPFIX
Information Element Data Types" subregistry. The type space for attributes is
constrained to facilitate implementation. The existing type space encompasses most
primitive types used in modern programming languages, as well as some derived
types such as ipv4Address, that are common to this domain.
status - The status of the specification of this Information Element. Allowed values
are current and deprecated. All newly defined Information Elements are in the current
status.
enterpriseId - You can define Information Elements without registering them with
IANA, for example, for enterprise internal purposes. For such Information Elements,
the Information Element identifier is not sufficient when the Information Element is
used outside the enterprise. If specifications of enterprise-specific Information
Elements are made public and if enterprise-specific identifiers are used by the IPFIX
protocol outside the enterprise, then the enterprise-specific identifier is globally
unique by combining it with an enterprise identifier. Valid values for the enterpriseId
are defined by IANA as Structure of Management Information (SMI) network
management private enterprise numbers, defined at [IANA-PEN]

Pluribus Networks
www.pluribusnetworks.com
271
Abstract Data Types Supported by IPFIX
Abstract data types unsigned8, unsigned16, unsigned32, unsigned64, signed8,
signed16, signed32, and signed64 are integral data types. These data type semantics can
be further specified, for example, by totalCounter, deltaCounter, identifier, or flags.
Abstract Data Type Description
unsigned8 Represents a non-negative integer value in the range
of 0 to 255.
unsigned16 Represents a non-negative integer value in the range
of 0 to 65535.
unsigned32 Represents a non-negative integer value in the range
of 0 to 4294967295.
unsigned64 Represents a non-negative integer value in the range
of 0 to 18446744073709551615.
signed8 Represents an integer value in the range of -128 to
127.
signed16 Represents an integer value in the range of -32768 to
32767.
signed32 Represents an integer value in the range of
-2147483648 to 2147483647.
signed64 Represents an integer value in the range of
-9223372036854775808 to 9223372036854775807
float32 Corresponds to an IEEE single-precision 32-bit
floating-point type
float64 Corresponds to an IEEE single-precision 64-bit
floating-point type
boolean Represents a binary value. The only allowed values are
true and false.
macAddress Represents a MAC-48 address
octetArray Represents a finite-length string of octets.
string Represents a finite-length string of valid characters
from the Unicode coded character set. Unicode
incorporates ASCII and the characters of many other
international character sets.
dateTimeSeconds Represents a time value expressed with second-level
precision.
dateTimeMilliseconds Represents a time value expressed with
millisecond-level precision.
dateTimeMicrosecond Represents a time value expressed with
microsecond-level precision
dateTimeNanoseconds Represents a time value expressed with
nanosecond-level precision.
ipv4Address Represents an IPv4 address.

272
Pluribus Networks
www.pluribusnetworks.com
 Abstract Data Type Description
ipv6Address Represents an IPv6 address.
basicList Supports structured data export.
subTemplateList Supports structured data export.
subTemplateMultiList supports structured data export.

Data Type Semantics Supported by IPFIX

These semantics apply only to numeric types, as noted in the description of each semantic
below.
Abstract Data Type Description
quantity A numeric (integral or floating point) value representing a
measured value pertaining to the record. This is
distinguished from counters that represent an ongoing
measured value whose "odometer" reading is captured as
part of a given record. This is the default semantic type of
all numeric data types.
totalCounter An integral value reporting the value of a counter.
Counters are unsigned and wrap back to zero after
reaching the limit of the type. For example, an
unsigned64 with counter semantics continues to
increment until reaching the value of 2**64 - 1. At this
point, the next increment will wrap its value to zero and
continue counting from zero. The semantics of a total
counter is similar to the semantics of counters used in the
Simple Network Management Protocol (SNMP), such as
Counter32 . The only difference between total counters
and counters used in SNMP is that the total counters have
an initial value of 0. A total counter counts independently
of the export of its value.
deltaCounter An integral value reporting the value of a counter.
Counters are unsigned and wrap back to zero after
reaching the limit of the type. For example, an
unsigned64 with counter semantics continues to
increment until reaching the value of 2**64 - 1. At this
point, the next increment wraps its value to zero and
continue counting from zero. The semantics of a delta
counter is similar to the semantics of counters used in
SNMP, such as Counter32. The only difference between
delta counters and counters used in SNMP is that the
delta counters have an initial value of 0. A delta counter is
reset to 0 each time it is exported and/or expires without
export.

Pluribus Networks
www.pluribusnetworks.com
273
Abstract Data Type Description
identifier An integral value that serves as an identifier. Specifically,
mathematical operations on two identifiers (aside from
the equality operation) are meaningless. For example,
Autonomous System ID 1 * Autonomous System ID 2 is
meaningless. Identifiers MUST be one of the signed or
unsigned data types.
flags An integral value that represents a set of bit fields. Logical
operations are appropriate on such values, but other
mathematical operations are not. Flags MUST always be
of an unsigned data type.

Information Elements Supported by Netvisor OS and IPFIX

Data Field Eleme Name Description Data Units Data

nt ID Type Type
Semant
ic
proto 4 protocolIdentifi The value of the unsigne identifie
er protocol number d8 r
in the IP packet
header.
The protocol
number identifies
the IP packet
payload type.
Protocol numbers
are defined in the
IANA Protocol
Numbers
registry.

274
Pluribus Networks
www.pluribusnetworks.com
 Data Field Eleme Name Description Data Units Data
nt ID Type Type
Semant
ic
cur-state 6 tcpControlBits TCP control bits unsigne flags
observed for the d16
packets of this
Flow. This
information is
encoded as a bit
field. For each
TCP control bit,
there is a bit in
this set. The bit
is set to 1 if any
observed packet
of this Flow has
the
corresponding
TCP control bit
set to 1. The bit
is cleared to 0
otherwise.
src-port 7 sourceTransport The source port unsigne identifie
Port identifier in the d16 r
transport header.
For the transport
protocols UDP,
TCP, and SCTP,
this is the source
port number in
the respective
header. This field
MAY also be used
for future
transport
protocols with
16-bit source
port identifiers.
src-ip 8 sourceIPv4Addr The IPv4 source ipv4Add default
ess address in the IP ress
packet header.

Pluribus Networks
www.pluribusnetworks.com
275
Data Field Eleme Name Description Data Units Data
nt ID Type Type
Semant
ic
src-switch-po 10 ingressInterfac The index of the unsigne identifie
rt e IP interface d32 r
where packets of
this Flow are
received. The
value matches
the value of
managed object
'ifIndex'. Note
that ifIndex
values are not
assigned
statically to an
interface and
that the
interfaces may
be renumbered
every time the
device's
management
system is
re-initialized,
dst-port 11 destinationTran The destination unsigne identifie
sportPort port identifier in d16 r
the transport
header. For the
transport
protocols UDP,
TCP, and SCTP,
this is the
destination port
number in the
respective
header. This field
MAY also be used
for future
transport
protocols
with 16-bit
destination port
identifiers.
dst-ip 12 destinationIPv4 The IPv4 ipv4Add default
Address destination ress
address in the IP
packet header.

276
Pluribus Networks
www.pluribusnetworks.com
 Data Field Eleme Name Description Data Units Data
nt ID Type Type
Semant
ic
dst-switch-po 14 egressInterface The index of the unsigne identifie
rt IP interface d32 r
where packets of
this Flow are
sent. The value
matches the
value of
managed object
'ifIndex' .
Note that ifIndex
values are not
assigned
statically to an
interface and
that the
interfaces may
be renumbered
every time the
device's
management
system is
re-initialized.

Pluribus Networks
www.pluribusnetworks.com
277
Data Field Eleme Name Description Data Units Data
nt ID Type Type
Semant
ic
41 exportedMessa The total number unsigne messag totalCo
geTotalCount of IPFIX d64 es unter
Messages the
Exporting
Process has sent
since the
Exporting
Process
(re-)initialization
to a particular
Collecting
Process. The
reported number
excludes the
IPFIX Message
that carries the
counter value. If
this Information
Element is sent
to a particular
Collecting
Process, then by
default, it
specifies the
number of IPFIX
Messages sent to
the Collecting
Process.

278
Pluribus Networks
www.pluribusnetworks.com
 Data Field Eleme Name Description Data Units Data
nt ID Type Type
Semant
ic
The total number
of Flow Records
that the
Exporting
Process has sent
as Data Records
since the
Exporting
Process
(re-)initialization
to a particular
Collecting
Process. The
reported number
excludes Flow
Records in the
IPFIX Message
with the counter
value.
If this
Information
Element is sent
to a particular
Collecting
Process, then by
default, it
specifies the
number of Flow
Records sent to
the process.
src-mac 56 sourceMacAddr The IEEE 802 macAdd default
ess source MAC ress
address field.
vlan 58 vlanId Virtual LAN unsigne identifie
identifier d16 r
associated with
ingress interface.
dst-mac 80 destinationMac The IEEE 802 macAdd default
Address source MAC ress
address field.
started-time 150 flowStartSecon The absolute dateTim second default
ds time stamp of eSecond s
the first packet of s
this Flow.

Pluribus Networks
www.pluribusnetworks.com
279
Data Field Eleme Name Description Data Units Data
nt ID Type Type
Semant
ic
ended-time 151 flowEndSecond The absolute dateTim second default
s time stamp of eSecond s
the last packet of s
this Flow.
216 unsigne identifie
d16 r
217 unsigne identifie
d16 r
218 unsigne packets totalCo
d64 unter
219 unsigne packets totalCo
d64 unter
222 unsigne packets totalCo
d64 unter
obytes 231 InitiatorOctets The total number unsigne octets deltaCo
of Layer 4 d64 unter
payload bytes in
a flow from the
initiator. The
initiator is the
device triggering
the session
creation, and
remains the
same for the life
of the session.
ibytes 232 responderOctet The total number unsigne octets deltaCo
s of Layer 4 d64 unter
payload bytes in
a flow from the
responder. The
responder is the
device that
replies to the
initiator, and
remains the
same for the life
of the session.
0x01 239 unsigne identifie
d8 r
src-switch-po 252 unsigne unsigne
rt d32 d32

280
Pluribus Networks
www.pluribusnetworks.com
 Data Field Eleme Name Description Data Units Data
nt ID Type Type
Semant
ic
dst-switch-po 253 egressPhysicalI The index of a unsigne identifie
rt nterface networking d32 r
device's physical
interface for
example, a
switch port,
where the flow
packets are sent.
ether-type 256 ethernetType The Ethernet unsigne identifie
type field of an d16 r
Ethernet frame
identifying the
MAC client
protocol carried
in the payload.
257 unsigne identifie
d8 r
258 dateTim millisec default
eSecond onds
s
259 unsigne identifie
d16 r
260 dateTim second default
eSecond s
s
261 dateTim second default
eSecond s
s
262 octetArr default
ay
349 octetArr default
ay
350 string default

Pluribus Networks
www.pluribusnetworks.com
281
Data Field Eleme Name Description Data Units Data
nt ID Type Type
Semant
ic
351 layer2Segment The identifier of a unsigne identifie
Layer 2 network d64 r
segment in an
overlay network.
The most
significant byte
identifies the
Layer 2 network
overlay network
encapsulation
type:
• 0x00 reserved
• 0x01 VxLAN
• 0x02 NVGRE
The three lowest
significant bytes
hold the value of
the Layer 2
overlay network
segment
identifier.
For example:
• a 24 bit
segment ID
VXLAN Network
Identifier (VNI)
• a 24 bit Tenant
Network
Identifier (TNI)
for NVGRE
368 unsigne identifie
d32 r
369 unsigne identifie
d32 r
401 unsigne octets deltaCo
d64 unter

Netvisor also supports the following private Information Elements:

latency 47269 LatencyMicroseco TCP session dateTimeSeco microsecon default

.1 nds latency nds ds

282
Pluribus Networks
www.pluribusnetworks.com
Configuring IPFIX
To configure IPFIX from the CLI, you must have a host IP address as the destination for the
IPFIX collector. Netvisor OS uses port 9090 by default, and the default transport protocol
type is TCP.

CLI network-admin@switch > ipfix-collector-create name ipfix-host1 port 9090

transport-protocol tcp dscp 3

To enable the IPFIX service, use the command, ipfix-service-modify enable. You can
also set the collection interval using this command. To set the collection interval to one hour,
use the following syntax:

CLI network-admin@switch > ipfix-service-modify enable export-interval

0d1h0m0s

Pluribus Networks
www.pluribusnetworks.com
283
Configuring vCenter
 vCenter Connection Service
 Configuring a vCenter Service
 Auto Provisioning for vCenter
 Automatic Link Aggregation on EXSi-facing Ports for vCenter
 Support for VLAN Alarms in vCenter

vCenter Connection Service

Connectivity to vCenter allows a network administrator visibility into physical and/or virtual
VMware entities behind each port of the Pluribus fabric for operational simplification. Netvisor
categorizes the properties of the entities into three classes:
Server, hypervisor,and one or more physical NICs.
Virtual Machine including guest OS, virtual NIC properties such as MAC address, IP
address, and VLAN.
VM Kernel ports and the associated services like vMotion and vSAN.
Netvisor now provides vCenter Connection Service allowing you to connect to VMware
vCenter and allows you to do the following tasks:
Identify the switch port used for each Virtual Machine (VM).
Track the movement of VMs from one host (ESXi) to another host.
Identify the VLAN requirements of each VM on the network.
Track network statistics at a granular level for troubleshooting purposes.
Track VM configuration changes such as additions, deletions, or modifications of
VLANs, and configure VLANs on Netvisor OS switch ports.
Track the additions or deletions of VMs and hosts, and configure VLANs on Netvisor OS
switches.
Track the state of VMs and dynamically provisions VLANs on the servers facing
physical ports.
vCenter Connection Service also synchronizes with VMware vCenter to obtain the following
information:
The host where the VMs exist.
The Netvisor OS switch ports connected to the VM.
The virtual network interface card (vNIC) that connects the VM to a virtual switch.
The Power status of the VM as on or off.
VLAN information of port groups.
The port groups required for the VM.
This information is associated with MAC addresses and VLANs on the network.
vCenter Connection Service currently has the following limitations for this Netvisor OS
release:
Support for only 1 VMware vCenter connections
Supported on ESXi 5.x and 6.x versions of VMware vCenter.

284
Pluribus Networks
www.pluribusnetworks.com
Pluribus Networks recommends checking the status of the connection. To check the status of
the connection, use the connection-show command. If the connection status shows an error
message with the state as "enabled”, then you should first disable the connection and then
enable it to restart the connection service.

Figure 1: Example LAN and VLAG Topology for vCenter

Example output with a list of VMs running on the two VMware ESXi servers along with
visibility into the associated VLANs, attached port group, vswitches:

CLI network-admin@switch > vport-show format ip,entity,status vnic-type

owner ip hostname entity power os portgroup vswitch

----- --------- -------- ------- ------- --------- --------- ------
SW-2 10.9.8.13 esx-1 MyTestVM powered-on Ubuntu Linux VM Network vSwitch0

vs-type vnic-type.. status

------- --------- ------
host-vs tagged vm

Pluribus Networks
www.pluribusnetworks.com
285
Configuring a vCenter Service
To create a vCenter service, use the vcenter-connection-create command:

CLI network-admin@switch > vcenter-connection-create name name-string host

host-string user user-string password password-string enable|disable vlans
vlan-list

To modify a vCenter service, use the following syntax:

CLI network-admin@switch > vcenter-connection-modify name name-string host

host-string user user-string password password-string enable|disable vlans
vlan-list

To delete a vCenter service, use the following syntax:

CLI network-admin@switch > vcenter-connection-delete name name-string

To display information about a vCenter service, use the following syntax:

CLI network-admin@switch > vcenter-connection-show name name-string host

host-string host host-string state init|ok|error connected-time
connected-time-string connection-error connection-error-string vlans vlan-list

Auto Provisioning for vCenter

Auto-provisioning allows a network administrators to provide a range of VLANs that other
administrators, for example, server administrators, can use to associate with port Groups
used in vCenter and applied to Virtual Machines(VMs).
For auto-provisioning VLANs, the vcenter-connection-create command extends to include
a vlans keyword to allow one VLAN or a range of VLANs tied to the service. If you do not
configure VLANs as part of starting the service, then vCenter does not auto-provision VLANs.
In this initial release, a maximum of 500 VLANs can be provisioned with per connection
service instance. Overlapping VLANs across connection service instances is allowed. VMs
connect to portGroups on a ESXi server, and the PortGroups include definition of VLAN or
VLAN range used. In order for the port Group VLAN or VLAN range to be provisioned in the
fabric it must be part of the range specified in the connection service command. The VLANs
are created with the scope local and no ports added.
Netvisor creates VLANs as persistent even after rebooting the switch. When you delete the
connection service, Netvisor deletes the VLANs or cleans up VLAN port membership added by
the service. For port membership on a VLAN, each switch manages the local host facing
ports. NEtvisor relies on LLDP objects stored on per physical NIC (pNIC) per ESXi basis and
distributed vSwitch (DVS) object attached to the pNIC. VMs attached to associated DVS are
port members. Server administrators must enable LLDP for the distributed portGroup.
When you remove the last VM attached to the local host por, then the port is removed from
associated VLAN. Port membership is validated and if necessary, updated with each poll cycle
of the vCenter database.
To display the range of provisioned VLANs, use the vcenter-connection-show command:

286
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > vcenter-connection-show

switch name host user enable state

-------------- ---- ----------- -------------------------- ------ -----
Spine1 svc2 10.9.34.204 [email protected] yes ok
Spine1 svc2 10.9.34.204 [email protected] yes ok
connected-time
--------------------------------
connected at 2017-01-14 03:07:23
connected at 2017-01-14 03:07:21

vlans
-----------
20-30,33-40
20-30,33-40

Automatic Link Aggregation on EXSi-facing Ports for vCenter

Currently, automatic Link Aggregation (LAG) is only supported between Pluribus Networks
switches. With this new feature, automatic Link Aggregation (LAG) includes ports between
Pluribus Networks switches and ESXi hosts .
Netivsor implements LLDP and LACP to bundle the ports. Since Netvisor does not proviide
custom type-length-value (TLV) probes, Netvisor implements standard LLDP TLVs to uniquely
identify ESXi hosts. Netvisor uses the system description TLV to identify ESXi hosts and the
system name uses to uniquely identify a specific ESXi host.
Netvisor enables LACP mode and set to active on auto-lag with a fallback-option of
individual to ensure extra robustness in bundling and ensure ports are bundled only if ESXi
host runs LACP to avoid any data path issues.
Netvisor provides a new parameter to enable or disable trunking towards hosts. The default
value is off.
The current global parameter auto-trunk needs to be ON (default) for this new settings to
take effect. Setting auto-trunk to off turns off all trunking.
CLI (network-admin@Leaf1) > system-settings-show
auto-host-trunk: on

Example output from trunk-show, port-show and lldb-show.

CLI (network-admin@Leaf1) > trunk-show format
trunk-id,switch,name,ports,lacp-mode,lacp-fallback,lacp-individual,status,

trunk-id switch name ports lacp-mode lacp-fallback

-------- -------------- -------- ----- --------- -------------
129 Leaf1 auto-129 42,44 active individual

lacp-individual status
--------------- -----------
none up,PN-other

Pluribus Networks
www.pluribusnetworks.com
287
CLI (network-admin@Leaf1) > port-show port 42,44 format all
port bezel-port vnet hostname status
---- ---------- ---- -------- ---------------------------------------
42 42 up,PN-other,LLDP,trunk,LACP-PDUs,vlan-up
44 44 up,PN-other,LLDP,trunk,LACP-PDUs,vlan-up
rswitch rem-ip rem-mac lport config trunk
----------------- ----------------- ------ ------ --------
2987 :: 00:00:00:00:00:00 42 fd,10g auto-129
2987 :: 00:00:00:00:00:00 44 fd,10g auto-129

CLI (network-admin@Leaf1) > lldp-show

local-port chassis-id port-id port-desc
---------- ---------- ----------------- ------------------------------
42 vmnic2 00:50:56:98:07:56 port 25 on dvSwitch DEV-CN-Tests-1
44 vmnic3 00:50:56:98:07:57 port 24 on dvSwitch DEV-CN-Tests-1
sys-name
------------------------------
esx-dev01.pluribusnetworks.com
esx-dev01.pluribusnetworks.com

Support for VLAN Alarms in vCenter

Netvisor can be configured with a vCenter Connection Service to read metadata from a
vCenter server and updates to the vPort table when learned from an inventory scan. A
remote plugin for vCenter is now capable of auto provisioning a VLAN range for a
DvPortgroup associated with uplinks going to the Netvisor OS fabric.
vCenter Alarm integration allows you to see the vCenter connection service running with
supported VLAN ranges configured on DVPortgroup.
If you configure a DVPortgroup associated with a VM for VLAN trunking using VLANs
unsupported by auto provisioning, then you receive alarms on the vSphere Web Client
Server. When you receive an alarm, you can modify the VLAN range or request additional
VLANs from the Netvisor switch administrator.
To configure the vCenter Connection Service with VLANs, use the following syntax:

CLI network-admin@switch > vcenter-connection-create name svc12 host

10.13.37.211 user [email protected] vlans 10-15

288
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > vcenter-connection-show

switch name host user enable

state
------------------ -----------
------------ ------------------------- ------- -----
switch-1 svc15 10.13.37.211 [email protected] yes
ok
switch-2 svc15 10.9.34.204 [email protected] yes
ok

connected-time vlans
-------------------------------- -----
connected at 2017-03-02 20:14:55 10-15
connected at 2017-03-02 20:14:57 10-15

If you configure a VM with a VLAN that is not in the 10-15 VLAN range, then an alarm is
triggered.

Pluribus Networks
www.pluribusnetworks.com
289
Configuring Open Virtual Switch

Open vSwitch is a virtual switch that enables network automation, while supporting standard
management interfaces and protocols, like NetFlow. Open vSwitch also supports distribution
across multiple physical servers.
In an Open vSwitch implementation, a database server and a switch daemon are used. The
OVSDB protocol is used in a control cluster, along with other managers and controllers, to
supply configuration information to the switch database server. Controllers use OpenFlow to
identify details of the packet flows through the switch. Each switch may receive directions
from multiple managers and controllers, and each manager and controller can direct multiple
switches.

Configuring OVSDB with Netvisor OS

There are a number of steps required to configure OVSDB using Netvisor OS.
1. Configure the VNET with the number of private VLANs, VXLANs, and managed ports:

CLI network-admin@switch > vnet-create name name-string vlans vlan-range

num-private-vlans integer vxlans vxlan-id managed-ports port-list

2. Configure the underlay network:

CLI network-admin@switch > vrouter-create name-string vnet name-string router

type hardware

CLI network-admin@switch > vrouter-interface-add vrouter-name name-string vlan

vlan-id ip ip-address netmask netmask

3. Configure the tunnel:

CLI network-admin@switch > vnet-tunnel-network-add name name-string network

ip-address netmask netmask

CLI network-admin@switch > trunk-modify name name-string trunk-id trunk-id port

port-list

4. Create the SSL/TLS certificate:

CLI network-admin@switch > cert-create name name-string country country-string

state state-string city city-string organization organization-string
organizational-unit organizational-unit-string common-name common-name-string
container zone name-string

5. Configure OVSDB:

CLI network-admin@switch > openvswitch-create name name-string vnet name-string

global-vtep tunnel-ip ip-address dedicated-service cert-name name-string

CLI network-admin@switch > openvswitch-interface-add ovs-name name-string ip

ip-address netmask netmask if data|mgmt vlan vlan-id

290
Pluribus Networks
www.pluribusnetworks.com
 Configuring the interface as data or management depends on the location of the controller,
on the data network or the management network.
If the controller is on a Layer 3 network several hops away, use openvswitch-modify to
configure a gateway IP address. This is required in order for the configuration to work
properly.
6. Add the hardware VTEP manager:

CLI network-admin@switch > opensvswitch-hwvtep-manager-add name name-string

manager-type odl|nsx connection-method ssl ip ip-address username
username-string password password-string port port-number

A VXLAN tunnel automatically establishes between the local and remote hardware and
software VTEPs.
If you connects to VMware NSX controllers, you must use SSL or TLS to securely connect
with the hardware VTEP.

Using OpenSSL TLS Certificates for OVSDB and other Services

This feature provides a common Transport Layer Socket (TLS) within Netvisor OS that you
can use for any service such as the Open vSwitch Database Management Protocol (OVSDB)
or a Web service. TLS is needed for any SSL connection to a Netvisor OS service. For OVSDB,
it is needed to connect to a controller using SSL. For HTTPS communication between a REST
API client and the Tomcat application server which is running a switch, you need to configure
and deploy a server certificate in a Tomcat server.
You can create one common certificate for all Netvisor OS services or create multiple named
certificates. Each service can use a different certificate identified by name or container name
or zone.
The Certificate facility keeps track of certificate use by using various applications. It notifies
the applications when a certificate is updated and it also prevents a certificate from being
deleted if an application is using it.
There are two ways to generate certificates:
Self-signed certificate
Certificate signed by a Certificate Authority (CA)

Self-signed Certificate
To generate a self-signed certificate use the cert-create command. This command creates
a server certificate and self-signs it.

Certificate signed by a Certificate Authority (CA)

To generate a certificate signed by a CA, follow these steps:
1. Create a certificate signing request.
2. Export the certificate signing request and send it to the CA administrator.
3. Import the certificate received from CA administrator against the right certificate signing
request.
4. Import the intermediate root and root certificate to the switch, if not done already.

Pluribus Networks
www.pluribusnetworks.com
291
CLI Commands
These commands allow you to manage TLS certificates.
To create a server certificate that self-signs., use the cert-create command:

CLI network-admin@switch > cert-create country country-string state

state-string city city-string organization organization-string
organizational-unit organization-unit-string common-name common-name-string
name name-string [container/zone name]

cert-create Creates a server certificate and

self-sign.
country country-string Specify a country name (two letter
code).
state state-string Specify a state or province name.
city city-string Specify a city name.
organization organization-string Specify an organization name.
organizational-unit Specify an organizational unit name.
organizational-unit-string
common-name common-name-string Specify a common name.
name name-string Specify a certificate name.
any of the following options:
container zone name Specify a certificate zone name.

To delete a certificate, use the cert-delete command:

CLI network-admin@switch > cert-delete name name-string [container/zone name]

cert-delete Deletes a certificate.

name name-string Specify a country name (two letter
code).
any of the following options:
container zone name Specify a certificate zone name.

To import a CA certificate file, use the cert-import command:

CLI network-admin@switch > cert-import name name-string file-ca file-ca-string

[container zone name][file-inter file-inter-string]

cert-import Imports certificates from the SFTP

directory.
name name-string Specify a certificate name.
file-ca file-ca-string Specify the name of CA certificate
file.
file-server file-server-string Specify the name of server certificate
file.
any of the following options:
container zone name Specify a certificate zone name.
any of the following options:
file-inter file-inter-string Specify the name of intermediate CA
certificate file.

To import a server certificate file, use the cert-import command:

292
Pluribus Networks
www.pluribusnetworks.com
CLI network-admin@switch > cert-import name name-string file-server
file-server-string [container zone name][file-ca file-ca-string]file-inter
file-inter-string]

cert-import Imports certificates from SFTP

directory.
name name-string Specify the certificate name.
file-server file-server-string Specify the name of server certificate
file.
at least 0 of the following options:
container zone name Specify a certificate zone name.
any of the following options:
file-ca file-ca-string Specify the name of the CA certificate
file.
file-inter file-inter-string Specify the name of the intermediate
CA certificate file.

To create a certificate signing request, use the cert-request-create command:

CLI network-admin@switch > cert-request-create name name-string

[container/zone name]

cert-request-create Create a certificate signing request

from an existing server certificate.
name name-string Specify the certificate name.
at least 0 of the following options:
container zone name Specify a certificate zone container
name.

To display a certificate signing request, use the cert-request-show command:

CLI network-admin@switch > cert-request-show name name-string [container/zone

name]cert-request cert-request-name

----------------------------------------------------------------
-----BEGIN CERTIFICATE REQUEST-----
MIICnDCCAYQCAQEwVzELMAkGA1UEBhMCdXMxCzAJBgNVBAgMAmNhMQswCQYDVQQH
DAJtcDELMAkGA1UECgwCcGwxDTALBgNVBAsMBGVuZ2cxEjAQBgNVBAMMCXBsdXJp
YnVzMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMrE6Jowg0VKUw2M
NlL8vp1N8dYE/UL5pvu8FKYWgwG7tC2fjHunZCI0XmssFtZysQul/r9nk+edA5tt
0zIWRmqTB60wnWmzl6uGymeAsC9OSm0ZHFc9zZfUxKjRM/n1dOri3Pw/rODbCjM9
qwO5hsvZc/c1o3ajYFrj1yMlKDIiPW1td1VTpc5TL6wCwnDM697Yb9oQ0cbLKTDl
w5AjQSgJK29rLUl8ptAZXIUkeendpE4MCYrl6Hd+ziOJHXncj65MJyfANTZMrtGD
IJD3m+JsKZt882vMw3AZ3C9WEuE0OZrbabGBHqVKARik2qFhu2bGjlbuj/M6TOf5
Jj1WROUCAwEAAaAAMA0GCSqGSIb3DQEBBQUAA4IBAQCh1YhXRNwkwmw3FVH4H0Xi
rczy0FkyHkdSbIUIf+6n3qroRpBpcEdrx8fREyiw8hLUks9OcUlT+nSshsWIitI7
R5dcFlyo5HUVjqQQVMlSq3j4fM9XE8y8KRMZ3mfLXRTmuFPxbBuE3ZGjlBSLnBgK
ODqHF1gVa4u7l9mO3TRXczLQiAPaw38/kxEwkh4erJp4jjXf8K0h9JMGvYONYWeI
1PbiZpjIWDLNbg6sKqqrPAxEAjzGNMgNPIMXRepmEmnC/BaLVA04noZran8LRLNp
Id41o3TnlXiAodF/Mc7H5fI1hYf0YzWDSfz3PNufn6Dusu5M2ma7jtWlEdBW8huH
-----END CERTIFICATE REQUEST-----

To display certificates, use the cert-show command:

Pluribus Networks
www.pluribusnetworks.com
293
CLI network-admin@switch > cert-show [cert-type ca|intermediate|server]
[subject subject-string] [issuer issuer-string] [serial-number
serial-number-number] [valid-from valid-from-string] [valid-to
valid-to-string] [country country-string] [state state-string] [city
city-string] [organization organization-string] [organizational-unit
organizational-unit-string] [common-name common-name-string] [ name
name-string] [container/zone name]

name used-by cert-type container subject

----- ------- --------- --------- ------------------------------------------
cert3 ca /C=us/ST=ca/L=mp/O=pl/OU=engg/CN=pluribus1
cert3 server /C=us/ST=ca/L=mp/O=pl/OU=engg/CN=pluribus1
cert1 ovs ca /C=US/ST=CA/L=PA/O=ovs/OU=ou/CN=Pluribus
cert1 ovs server /C=US/ST=CA/L=PA/O=ovs/OU=ou/CN=Pluribus

Configuring OpenvSwitch for Certificates

The following openvswitch-create and openvswitch-modify command options allow you
to specify a certificate name when creating an OpenvSwitch configuration.

CLI network-admin@switch > openvswitch-create name name-string [cert-name

cert-name string[ [ca-cert-name ca-cert-name-string] [cert-location
none|global|container]

openvswitch create Create an OpenvSwitch

configuration.
any of the following options:
cert-name cert-name-string Specify the certificate name
for SSL connections.
ca-cert-name ca-cert-name-string Specify the CA Certificate
name for SSL connection.
cert-location none|global|container Specify the certificate
location - global or within the
container.

CLI network-admin@switch > openvswitch-modify name name-string [cert-name

cert-name string[ [ca-cert-name ca-cert-name-string] [cert-location
none|global|container]

cert-name cert-name-string Specify the certificate name

for SSL connections.
ca-cert-name ca-cert-name-string Specify the certificate name
for SSL connections.
cert-location none|global|container Specify the certificate
location - global or within the
container.

Open Virtual Switch Database (OVSDB) Error Reporting

Netvisor OS now supports an error-reporting mechanism for OVSDB and VTEPs. When an
error occurs, Netvisor OS sends a schema change to the OVSDB controller.
As more functionality is added to Netvisor for OVSDB, OVSDB error reporting adds new
errors to support the new functionality.

294
Pluribus Networks
www.pluribusnetworks.com
Appendix A - Acknowledgments for Open Source Software
The Pluribus Networks Netvisor Command Line Interface (CLI) used the following Open Source Software:

bean.js
(bean.js):
bean.js - copyright Jacob Thornton 2011
https://github.com/fat/bean
MIT License
special thanks to:
dean edwards: http://dean.edwards.name/
dperini: https://github.com/dperini/nwevents
the entire mootools team: github.com/mootools/mootools-core
Bonzo: DOM Utility (c) Dustin Diaz 2011
https://github.com/ded/bonzo
License MIT

d3v2
(d3v2):
Copyright (c) 2012, Michael Bostock
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following
disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided with the distribution.
The name Michael Bostock may not be used to endorse or promote products derived from this software without
specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL MICHAEL BOSTOCK BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

d3v2
(d3v2):
TERMS OF USE - EASING EQUATIONS

Pluribus Networks
www.pluribusnetworks.com
295
Open source under the BSD License.
Copyright 2001 Robert Penner
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the
following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following
disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
disclaimer in the documentation * and/or other materials provided with the distribution.
Neither the name of the author nor the names of contributors may be used to endorse or promote products derived
from this software without specific * prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"* AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

DataTables
(DataTables):
@summary DataTables
@description Paginate, search and sort HTML tables
@version 1.9.4
@file jquery.dataTables.js
@author Allan Jardine (www.sprymedia.co.uk)
@contact www.sprymedia.co.uk/contact
@copyright Copyright 2008-2012 Allan Jardine, all rights reserved.
This source file is free software, under either the GPL v2 license or a BSD style license, available at:
http://datatables.net/license_gpl2
http://datatables.net/license_bsd
This source file is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the
implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the license files for details.
For details please refer to: http://www.datatables.net

(Envision)
Envision.js
(c) 2012 Carl Sutherland, Humble Software
Distributed under the MIT License
Source: http://www.github.com/HumbleSoftware/envisionjs

296
Pluribus Networks
www.pluribusnetworks.com
Homepage: http://www.humblesoftware.com/envision

excanvas
(excanvas.js):
Filament Group modification note:
This version of excanvas is modified to support lazy loading of this file. More info here:
http://pipwerks.com/2009/03/12/lazy-loading-excanvasjs/
Copyright 2006 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and limitations under the License.

FloodLight
Copyright 2012, Big Switch Networks, Inc.
Originally created by David Erickson, Stanford University
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for
the specific language governing permissions and limitations under the License

Flotr2
Flotr2 (c) 2012 Carl Sutherland
MIT License
Special thanks to:
Flotr: http://code.google.com/p/flotr/ (fork)
Flot: https://github.com/flot/flot (original fork)

g.Raphael
g.Raphael 0.5 - Charting library, based on Raphaël
Copyright (c) 2009 Dmitry Baranovskiy (http://g.raphaeljs.com)
Licensed under the MIT (http://www.opensource.org/licenses/mit-license.php) license.

Pluribus Networks
www.pluribusnetworks.com
297
GRUB
GRUB -- GRand Unified Bootloader
Copyright (C) 1999,2000,2001,2002,2004 Free Software Foundation, Inc.
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public
License as published by the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the
implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free
Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

GSON
Copyright (C) 2008 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an
"AS IS" BASIS, ITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and limitations under the License.

DNS/DHCP
Copyright (c) 2004-2013 by Internet Systems Consortium, Inc. ("ISC")
Copyright (c) 1995-2003 by Internet Software Consortium
Permission to use, copy, modify, and distribute this software for any purpose with or without fee is hereby granted,
provided that the above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND ISC DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE
INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL ISC BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING
FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS
ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
Internet Systems Consortium, Inc.
950 Charter Street
Redwood City, CA 94063
<[email protected]>
https://www.isc.org

JIT
Copyright (c) 2011 Sencha Inc. - Author: Nicolas Garcia Belmonte (http://philogb.github.com/)

298
Pluribus Networks
www.pluribusnetworks.com
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

jquery.js
jQuery JavaScript Library v1.4.3
http://jquery.com/
Copyright 2010, John Resig
Dual licensed under the MIT or GPL Version 2 licenses.
http://jquery.org/license
Includes Sizzle.js
http://sizzlejs.com/
Copyright 2010, The Dojo Foundation
Released under the MIT, BSD, and GPL Licenses.
Date: Thu Oct 14 23:10:06 2010 -0400

jQuery UI

Copyright (c) 2011 Paul Bakaus, http://jqueryui.com/

This software consists of voluntary contributions made by many individuals (AUTHORS.txt,
http://jqueryui.com/about) For exact contribution history, see the revision history and logs, available at
http://jquery-ui.googlecode.com/svn/
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Pluribus Networks
www.pluribusnetworks.com
299
jquery.cookie.js
jQuery Cookie plugin
Copyright (c) 2010 Klaus Hartl (stilbuero.de)
Dual licensed under the MIT and GPL licenses:
http://www.opensource.org/licenses/mit-license.php
http://www.gnu.org/licenses/gpl.html

jquery.hotkeys.js
jQuery Hotkeys Plugin
Copyright 2010, John Resig
Dual licensed under the MIT or GPL Version 2 licenses.
Based upon the plugin by Tzury Bar Yochay:
http://github.com/tzuryby/hotkeys
Original idea by:
Binny V A, http://www.openjs.com/scripts/events/keyboard_shortcuts/

jquery.validate.min.js
Query Validation Plugin 1.8.1
http://bassistance.de/jquery-plugins/jquery-plugin-validation/
http://docs.jquery.com/Plugins/Validation
Copyright (c) 2006 - 2011 Jörn Zaefferer
Dual licensed under the MIT and GPL licenses:
http://www.opensource.org/licenses/mit-license.php
http://www.gnu.org/licenses/gpl.html

JSTL
DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
Copyright (c) 1997-2010 Oracle and/or its affiliates. All rights reserved.
The contents of this file are subject to the terms of either the GNU General Public License Version 2 only ("GPL") or
the Common Development and Distribution License("CDDL") (collectively, the "License"). You
may not use this file except in compliance with the License. You can obtain a copy of the License at
https://glassfish.dev.java.net/public/CDDL+GPL_1_1.html or packager/legal/LICENSE.txt. See the License for the
specific language governing permissions and limitations under the License.
When distributing the software, include this License Header Notice in each file and include the License file at
packager/legal/LICENSE.txt.
GPL Classpath Exception:
Oracle designates this particular file as subject to the "Classpath" exception as provided by Oracle in the GPL Version
2 section of the License file that accompanied this code.

300
Pluribus Networks
www.pluribusnetworks.com
Modifications:
If applicable, add the following below the License Header, with the fields enclosed by brackets [] replaced by your
own identifying information:
"Portions Copyright [year] [name of copyright owner]"
Contributor(s):
If you wish your version of this file to be governed by only the CDDL or only the GPL Version 2, indicate your decision
by adding "[Contributor] elects to include this software in this distribution under the [CDDL or GPL Version 2]
license." If you don't indicate a single choice of license, a recipient has the option to distribute your version of this
file under either the CDDL, the GPL Version 2 or to extend the choice of license to its licensees as provided above.
However, if you add GPL Version 2 code and therefore, elected the GPL Version 2 license, then the option applies
only if the new code is made subject to such option by the copyright holder.
This file incorporates work covered by the following copyright and permission notice:
Copyright 2004 The Apache Software Foundation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and limitations under the License.

jstree
jsTree 1.0-rc3
http://jstree.com/
Copyright (c) 2010 Ivan Bozhanov (vakata.com)
Licensed same as jquery - under the terms of either the MIT License or the GPL Version 2 License
http://www.opensource.org/licenses/mit-license.php
http://www.gnu.org/licenses/gpl.html
$Date: 2011-02-09 01:17:14 +0200 (ср, 09 февр 2011) $
$Revision: 236 $
Start Copyright text (libedit 3.0)
Copyright (c) 1992, 1993
The Regents of the University of California. All rights reserved.
This code is derived from software contributed to Berkeley by Christos Zoulas of Cornell University.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the
following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following
disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided with the distribution.

Pluribus Networks
www.pluribusnetworks.com
301
3. Neither the name of the University nor the names of its contributors may be used to endorse or promote products
derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

(log4j
Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the
NOTICE file distributed with this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file
except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and limitations under the License.
(pciutils-3.1.10):
The PCI Utilities -- Declarations
Copyright (c) 1997--2008 Martin Mares <[email protected]>
Can be freely distributed and used under the terms of the GNU GPL.

qtip 2.0
qTip2 - Pretty powerful tooltips - v2.0.0 - 2012-10-03
http://craigsworks.com/projects/qtip2/
Copyright (c) 2012 Craig Michael Thompson; Licensed MIT, GPL

raphael 2.1.0
Raphaël 2.1.0 - JavaScript Vector Library
Copyright © 2008-2012 Dmitry Baranovskiy (http://raphaeljs.com)
Copyright © 2008-2012 Sencha Labs (http://sencha.com)
Licensed under the MIT (http://raphaeljs.com/license.html) license.
Copyright (c) 2013 Adobe Systems Incorporated. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License.
You may obtain a copy of the License at //
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

302
Pluribus Networks
www.pluribusnetworks.com
See the License for the specific language governing permissions and limitations under the License.
Eve 0.4.2 - JavaScript Events Library
Author Dmitry Baranovskiy (http://dmitry.baranovskiy.com/)

Rickshaw v1.1.2
Adapted from https://github.com/Jakobo/PTClass */
Copyright (c) 2005-2010 Sam Stephenson
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
persons to whom the Software is furnished to do so, subject to the following conditions:
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Based on Alex Arnell's inheritance implementation.
section: Language
class Class
Manages Prototype's class-based OOP system.
Refer to Prototype's web site for a [tutorial on classes and
inheritance](http://prototypejs.org/learn/class-inheritance).

science.js 1.7.0
Copyright (c) 2011, Jason Davies
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the
following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following
disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided with the distribution.
The name Jason Davies may not be used to endorse or promote products derived from this software without specific
prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL JASON DAVIES BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Pluribus Networks
www.pluribusnetworks.com
303
sizzle
Copyright (c) 2009, John Resig
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the
following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following
disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of the <organization> nor the names of its contributors may be used to endorse or promote
products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY John Resig ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL <copyright holder> BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

tcl 8.5.9
This software is copyrighted by the Regents of the University of California, Sun Microsystems, Inc., Scriptics
Corporation, ActiveState Corporation and other parties. The following terms apply to all files associated with the
software unless explicitly disclaimed in individual files.
The authors hereby grant permission to use, copy, modify, distribute, and license this software and its
documentation for any purpose, provided that existing copyright notices are retained in all copies and that this
notice is included verbatim in any distributions. No written agreement, license, or royalty fee is required for any of
the authorized uses. Modifications to this software may be copyrighted by their authors and need not follow the
licensing terms described here, provided that the new terms are clearly indicated on the first page of each file where
they apply.
IN NO EVENT SHALL THE AUTHORS OR DISTRIBUTORS BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, SPECIAL,
INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OF THIS SOFTWARE, ITS DOCUMENTATION,
OR ANY DERIVATIVES THEREOF, EVEN IF THE AUTHORS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
THE AUTHORS AND DISTRIBUTORS SPECIFICALLY DISCLAIM ANY WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT.
THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, AND THE AUTHORS AND DISTRIBUTORS HAVE NO OBLIGATION
TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
GOVERNMENT USE: If you are acquiring this software on behalf of the U.S. government, the Government shall have
only "Restricted Rights" in the software and related documentation as defined in the Federal Acquisition Regulations
(FARs) in Clause 52.227.19 (c) (2). If you are acquiring the software on behalf of the Department of Defense, the
software shall be classified as "Commercial Computer Software" and the Government shall have only "Restricted
Rights" as defined in Clause 252.227-7013 (b) (3) of DFARs. Notwithstanding the foregoing, the authors grant the
U.S. Government and others acting in its behalf permission to use and distribute the software in accordance with
the terms specified in this license.

304
Pluribus Networks
www.pluribusnetworks.com
tcllib 1.13
This software is copyrighted by Ajuba Solutions and other parties.
The following terms apply to all files associated with the software unless explicitly disclaimed in individual files.
The authors hereby grant permission to use, copy, modify, distribute, and license this software and its
documentation for any purpose, provided that existing copyright notices are retained in all copies and that this
notice is included verbatim in any distributions. No written agreement, license, or royalty fee is required for any of
the authorized uses.
Modifications to this software may be copyrighted by their authors and need not follow the licensing terms
described here, provided that the new terms are clearly indicated on the first page of each file where they apply.
IN NO EVENT SHALL THE AUTHORS OR DISTRIBUTORS BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, SPECIAL,
INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OF THIS SOFTWARE, ITS DOCUMENTATION,
OR ANY DERIVATIVES THEREOF, EVEN IF THE AUTHORS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
THE AUTHORS AND DISTRIBUTORS SPECIFICALLY DISCLAIM ANY WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT.
THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, AND THE AUTHORS AND DISTRIBUTORS HAVE NO OBLIGATION
TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
GOVERNMENT USE: If you are acquiring this software on behalf of the U.S. government, the Government shall have
only "Restricted Rights" in the software and related documentation as defined in the Federal Acquisition Regulations
(FARs) in Clause 52.227.19 (c) (2). If you are acquiring the software on behalf of the Department of Defense, the
software shall be classified as "Commercial Computer Software" and the Government shall have only "Restricted
Rights" as defined in Clause 252.227-7013 (c) (1) of DFARs. Notwithstanding the foregoing, the authors grant the
U.S. Government and others acting in its behalf permission to use and distribute the software in accordance with
the terms specified in this license.

tclreadline 2.1.0
Copyright (c) 1998 - 2000, Johannes Zellner <[email protected]>
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the
following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following
disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of Johannes Zellner nor the names of contributors to this software may be used to endorse or
promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Pluribus Networks
www.pluribusnetworks.com
305
UI Widgets
The MIT License
Copyright (c) 2010 Filament Group, Inc
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following
conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Underscore.js v1.1.7
Underscore.js 1.1.7
(c) 2011 Jeremy Ashkenas, DocumentCloud Inc.
Underscore is freely distributable under the MIT license.
Portions of Underscore are inspired or borrowed from Prototype,
Oliver Steele's Functional, and John Resig's Micro-Templating.
For all details and documentation:
http://documentcloud.github.com/underscore

Underscore.js v1.1.7

Xelerated version (hxps 2.5.3 and earlier):

Copyright 2008-2012 (c) Xelerated AB.
This program may be used and/or copied only with the written permission from Xelerated AB, or in accordance with
the terms and conditions stipulated in the agreement/contract under which the program has been supplied.
All rights reserved.

Underscore.js v1.1.7
Marvell version (hxps 2.6 and above):
(c), Copyright 2008-2013, Marvell International Ltd. (Marvell)
This code contains confidential information of Marvell.
No rights are granted herein under any patent, mask work right or copyright of Marvell or any third party. Marvell
reserves the right at its sole discretion to request that this code be immediately returned to Marvell.

306
Pluribus Networks
www.pluribusnetworks.com
This code is provided "as is". Marvell makes no warranties, expressed, implied or otherwise, regarding its accuracy,
completeness or performance.

Oracle Solaris Open Source Projects

http://solaris.java.net/license.html

Pluribus Networks
www.pluribusnetworks.com
307