REVIEWS

Your network is growing up and getting out of control.

You might have physical hosts and Virtual Machines distributed across the entire
network, which you might want to get into the same network segment.

Stretching your VLANs could be a solution, as it can help you span your layer 2
across the physical network.

But the problem is that the number of VLANs runs out quickly and the solution is
VxLAN.

With this technology, you can layer 2 networks on top of your existing layer 3
network.

All those distributed hosts can be kept together using VxLAN.

In this article, we'll provide an overview of VXLAN and you'll learn what it is, how it
works, and where to use it.

You'll also learn about its benefits and deployment method.

Let's dive in!

1. What is VxLAN?
VxLAN (Virtual extensible Local Area Network) is an industry-standard overlay
network virtualization technology.
It was initially designed to address the issues related to scalability in large-scale
network deployments such as ISPs or cloud providers.

As the name implies, VxLAN virtually extends a layer 2 segment across the layer 3
network infrastructure. VxLAN encapsulates the layer 2 Ethernet frames inside a
VXLAN packet that includes an IP address.

VxLAN segments are identified by a 24-bit VNID (VxLAN IDentification) field which
can scale to 16 million segments.

The VxLAN is a standardized specification created by the collaboration of VMware,

Cisco, and Arista Network vendors. VxLAN is defined in the RFC 7348

a. VxLAN vs. VLAN

VxLAN is very similar to VLAN, which also encapsulates layer 2 frames and
segments networks.

The main difference is that VLAN uses the tag on the layer 2 frame for encapsulation
and can scale up to 4000 VLANs.

VXLAN, on the other hand, encapsulates the MAC in UDP and is capable of scaling
up to 16 million VxLAN segments.

b. What are VxLAN major advantages?

It is a fact that one of the significant benefits of VxLAN is scalability. But when you
can span layer 2 networks across IP network infrastructure, there are also many
more benefits.

1. Scalability and flexibility: VxLAN improves the scalability in a network or

virtualized data center, and it also makes its fabric more flexible. The number of
VLAN layer 2 identifiers are drastically increased from 4,000 to 16 million.
2. Segmentation and Multi-tenancy: VxLAN provides a high level of security by
segmenting the network. The VxLAN traffic is limited to VNI, so it is isolated. This
segmentation can also help in multi-tenant architectures, where a single
infrastructure must be shared.
3. Layer 2 Simplification. Simplify the network and reduce the need for layer 2
Spanning Trees, Trunking, and VLAN stretching.
4. Allow IP Mobility: VMs can be migrated from a host in a subnet to another host
in another subnet without having to change the IP address.
5. Layer 2 and layer 3 Connectivity. A virtual layer 2 running VNIs is built upon a
layer 3 infrastructure running IP. VxLAN switches encapsulate layer 2 frames into
layer 3 packets.
6. It is a Software-Defined Network (SDN). VxLAN decouples the central network
controller (virtual network) from the data plane (physical network). Having a
centralized controller simplifies network management, deployment, and
monitoring. An example of a software-based virtual network switch that supports
VxLAN overlays is Open vSwitch.
7. Hardware Support: Although it is more common to run VxLAN in software, some
platforms implement it in hardware through ASICs. An example is Cisco’s Nexus
9000-EX platform switches.
8. It is a standard: VXLAN is a technical standard. When you implement it, you are
not locked by any vendor.

2. VxLAN Overlay Network Design

VxLAN is an overlay encapsulation technology. It creates a virtual network overlaid
on top of the existing physical network infrastructure.

It uses the underlay IP network and builds a flexible layer 2 overlay logical network
on it.

With the overlay, any layer 2 connection can span across layer 3 network.

There are many advantages to using an overlay network.

The most obvious one is its segmentation.

The overlay and the underlay networks are totally independent, so if there is a
change in the underlay network topology, the overlay network will not be affected
(design-wise).

The overlay network can be re-designed without needing to add, remove, or update
network devices.

Of course, physical problems that affect performance or uptime of the underlay will
be reflected on the overlay.

For example, if there are not enough devices to provide enough bandwidth, the
overlay will also be affected.

How does the overlay VxLAN avoid being affected by underlay changes?

Using a switching fabric, referred to as the Spine-and-Leaf.

a. Spine-and-Leaf (Underlay) + VxLAN (Overlay)

The best way to guarantee performance, scalability, reliability, and flexibility on a
VxLAN overlay while allowing the underlay to change, is to make good use of a
switching fabric topology.

The best example of a switching fabric topology is the Spine-and-Leaf, which is

commonly used as an underlay network.

Spine-and-leaf is an independent architecture. It is not exclusive to VxLAN, but it is

often associated with it.

Usually the spine-and-leaf is the underlay of the VxLAN is the overlay.

Spine-and-Leaf uses two layers:

 Spine: The spine layer switches are only used to pass traffic through leaf
switches. They are not aware of VxLAN.
 Leaf: The Leaf layer of switches interconnect the spine and the end points. The
leaf layer switches create the VxLAN tunnels, encapsulation, and maps VLANs to
VNI. The leaf switches that perform VxLAN functions are known as VTPEs
(VxLAN Tunnel Endpoints
All the leaf switches have a link to every spine switch. Every link between leaf and
switch is routed through an IP address by an IGP routing protocol such as BGP or
OSPF.

This topology makes every destination only two hops away.

Leaf-and-switch may also use ECMP (Equal Cost Multi-Pathing) to recover when a
spine switch or link fails or to balance the traffic loads.

The spine-and-leaf fabric topology is highly relevant to VxLAN because, as the

overlay network scales, the supporting underlay can physically grow or decrease the
size without affecting the design of the overlay.

Adding VxLAN on top of the spine-and-leaf underlay allows IP mobility of east-west

traffic patterns, full scalability, and fault tolerance.

As your network scales, your design doesn’t need to change.

You just need to add more switches, IP addresses, and links to the underlay.

3. VXLAN Encapsulation
We know so far that VxLAN stretches the layer 2 subnets across the layer 3 network
limits. It builds a logical overlay network on top of a switching fabric like the Spine-
and-Leaf.

To make this work, VxLAN encapsulates layer 2 Ethernet frames in a VxLAN

packet that is also encapsulated in an IP UDP header.

The following picture shows the VxLAN packet format.

VxLAN adds the following fields to the original Layer 2 frame.

 Outer MAC header: This is the header that contains information for next-hop
transport. It includes the destination and the source MAC address of the VxLAN
endpoints, a VLAN ID (16 bits), and Type. The size of the outer MAC header is 14
bytes.
 Outer IP header: This header allows transport across the IP network. It includes
the destination and the source IP address of the VxLAN endpoints. The size of the
outer IP header is 20 bytes.
 Outer UDP header: This header identifies the packet as VxLAN. It contains the
UDP source port, VxLAN port, and UDP length. The size of the UDP header is 8
bytes.
 A VxLAN header. This header is also referred to as the VxLAN Network Identifier
(VNI). The VNID is used to identify the VxLAN segment. It is similar to the VLAN
ID tag (16 bits) found on the MAC header but with a size of 24 bits, which allows
up to 16 million different segments.
4. VxLAN Tunnel Endpoints (VTEPs)
Any endpoint like a host, switch, or router that supports VxLAN can be referred to as
a VTPE (VxLAN Tunnel Endpoint).

As the name implies, the job of VTEPs is to create and terminate tunnels between
each other. In other words, they encapsulate and decapsulate VxLAN traffic.

a. How does a VTEP work?

The VTPE is connected to the underlay network using a layer 3 IP address. VTPEs
may have one or more VNIs associated with it.

When a layer 2 frame with the same VNI arrives at the ingress VTEP, it
encapsulates the frame with a VxLAN and UDP/IP headers.

Then sends it over using the underlay IP network transport towards the egress VTPE
for decapsulation.

The egress VTPE removes the IP and UDP headers and delivers the original layer 2
frame.

A VTEP can be either a virtual or a physical switch port and is usually configured on
leaf switches.

VTEPs communicate using traditional underlay IP routing and forwarding

mechanisms such as OSPF and EIGRP.

b. VTEP + IP Transport: The Bud Node

In some cases, you might need a device to forward IP traffic and perform VxLAN
encapsulation/decapsulation.

A bud node is a switching device with two roles, a VTEP to perform VxLAN related
tasks and an IP transit device to forward VxLAN traffic.

In order to deliver traffic to other VTEPs, the bud node should be in the same
multicast group used by the VxLAN VNIs.

The below screenshot shows an example of the bud node. Let’s say that host 1
wants to communicate with host 3.

The VTEP-2 and VTEP-3 belong to a multicast group used for VXLAN VNIs. The
VTEP-2 checks the VxLAN ID from the packet and forwards it based on IP.

But if Host-1 wants to communicate with host-2, VTEP-2 can also decapsulate the
VxLAN packet.

Not all platforms can be a bud node. Running two roles at the same time requires
powerful processing power.

Platforms that run VxLAN on ASIC, such as the Cisco Nexus 9000 series, are
adequate for this type of topology.

c. VTEP Redundancy with vPC

VTEPs may fail, so you would need to consider a way to provide redundancy and
load sharing.
Although the design of the spine-and-leaf topology is already redundant and leaves
are not supposed to connect to each other, you can still introduce peering between
them to provide host-level redundancy.

Two leaf VTEPs gateways can act as one through peer-link and keep-alive links.
You can accomplish this with a Cisco-based feature, called the vPC (Virtual Port-
Channel).

vPC is a layer 2 feature found on the Cisco Nexus switches, which allows you to
connect a host to two switches at the same time.

The pair of virtual vPC switches can provide redundancy to the attached hosts.

These vPCs work as logical VTEP devices sharing an anycast address.

5. Deploying VxLAN
VxLAN is a standard-based technology, so it is not limited to a specific vendor and
can be supported by hardware or software.

You can deploy the technology through a VxLAN host or VxLAN gateway. You can
limit the deployment to one method or use a combination of both.

a. Host-based VxLAN
VxLAN doesn’t need to be deployed on a switch; it can also run on any host, as long
as it natively supports VxLAN.

An example is a hypervisor, which can be configured to run VxLAN on all of its virtual
machines.

The vSwitch running the functions of the VTEP, encapsulates traffic coming from the
VMs before they go out to physical switches.

Since the VxLAN encapsulation is happening at the host, the rest of the network
infrastructure only sees IP traffic.

The advantage of a host-based deployment is that the entire physical network can
be simplified.

Other examples of hosts that may support VxLAN are servers, firewalls, load
balancers, etc.
b. Gateway-based VxLAN
When hosts do not support VxLAN, the best way to deploy it is directly on a switch or
router.

When a switch performs the VTEP functions, it is referred to as a VxLAN Gateway.

The switches can perform VxLAN encapsulation/decapsulation and can also

translate the VLAN ID to VNI.

The VxLAN gateway creates the tunnel to the destination VTEP (either host or
gateway), so the hosts and IP infrastructure are not aware of the existence of
VxLAN.

A benefit of gateway-based deployments is that some platforms such as the Cisco

Nexus 9000-EX are capable of implementing VxLAN on hardware.

Running VxLAN right from the ASIC, rather than from software, can increase
performance dramatically.

c. Hybrid Deployment
You can also use a combination of both. A hybrid deployment is when you are using
VxLAN with some devices behind a VxLAN Gateway and also have some hosts
running native VxLAN.

6. Putting it all Together

Now that you know about Spine-and-leaf overlay topology, VxLAN encapsulation,
and VTEPs, let’s go through the process of how VxLAN traffic flows from host to host
in a simple VxLAN network.
1. Host A wants to communicate with Host Z on the other side of the network. Host A
(unaware of VxLAN) creates a regular layer 2 frame and sends it over to the
switch port. The receiving access switch port is configured with a specific VLAN.
So the switch assigns the VLAN ID to the frame coming from Host A.
2. The Switch A (VxLAN Gateway) with the role of VTEP, maps the (source) VLAN
ID with the (destination) VxLAN ID. The VTEP adds the VxLAN header, and
encapsulates the layer 2 frame into a layer 3 packet, and forwards it across the
layer 3 infrastructure.
3. Now, the in-between layer 3 infrastructure only sees IP traffic and is unaware of
any VxLAN information. The VxLAN traffic is tunneled. All routers on the underlay
network, only see an IP header, so they just forward it accordingly.
4. The receiving end (VxLAN Gateway), Switch B which is also a VTEP, opens up
the packet and finds the VxLAN information. Switch B also understands VxLAN
and knows the VxLAN ID to VLAN ID map. So it takes the packet, uncovers the
VLAN ID, and uses the destination MAC address to switch it to the respective
access switch port.
5. Host Z, also unaware of VxLAN, receives the regular layer 2 frame sent from Host
A.