#CiscoLiveAPJC

#CiscoLiveAPJC Session ID © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IOS XR Deep Dive for
Service Providers and Data
Center

Jiri Chaloupka - Principal Technical Marketing Engineer

BRKMPL-2253

#CiscoLiveAPJC
Cisco Webex App
https://ciscolive.ciscoevents.com/
ciscolivebot/#BRKMPL-2253

Questions?
Use Cisco Webex App to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App

2 Click “Join the Discussion”

3 Install the Webex App or go directly to the Webex space

4 Enter messages/questions in the Webex space

Webex spaces will be moderated

by the speaker until November 15, 2024.

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
 • EVPN Basic Principles
• EVPN L2 Multihomed Service
• All-Active / Single-Active / Port-Active
• EVPN Distributed L3 Anycast Gateway
• EVPN Centralized Gateway
• EVPN L3 Interconnect Options
• EVPN & VPNv4/6 Interconnect
• EVPN Routes – Summary

Agenda • EVPN-VPWS Multihomed Service

• All-Active / Single-Active
• VPLS to EVPN Seamless Migration
• Pseudowire(PW) to EVPN-VPWS
• Seamless Migration
• EVPN & VPLS Interconnect
• Summary
• Extra Offline Learning
#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
 Unified Control Plane and Data Plane
Common BGP Control Plane
EVPN, VPNv4/6 Overlay
Next Generation
Services Overlay &
Underlay
Data Plane Segment Routing (SR: MPLS, SRv6) SR, VXLAN SR, VXLAN

Data Center Network

Service Provider Network overlap

Leaf
VM

PE1 DCI1

Spine Spine

Leaf
VM
A1 Access WAN/Core

Leaf
PE2 DCI2 VM

BGP: VPNv4/6 VPLS Overlay

Fabric-Path (Trill)
Legacy Solution: LDP: VPLS, PW Fabric-Path

MPLS: LDP, RSVP-TE MPLS, L2 L2, IP Underlay

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Service Provider Network - Simplification Journey

Provisioning NETCONF NETCONF

YANG YANG

Programmability
PCE

L2/L3VPN Services LDP BGP LDP BGP BGP

Inter-Domain CP BGP-LU BGP-LU

RSVP IGP with
FRR or TE
SR-MPLS or
LDP IGP with SR SRv6
Intra-Domain CP
IGP

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Next-Generation Solutions for L2VPN
Solving VPLS challenges for per-flow Redundancy

M1 M2
CE1 PE1 PE3 CE2
• Existing VPLS solutions do not offer an Echo !
All-Active per-flow redundancy PE2 PE4

• Looping of Traffic Flooded from PE

M1 Duplicate !
M2
• Duplicate Frames from Floods from the
CE1 PE1 PE3 CE2
Core
• MAC Flip-Flopping over Pseudowire PE2 PE4

• E.g. Port-Channel Load-Balancing does

not produce a consistent hash-value for a M1 M2
frame with the same source MAC (e.g. non CE1 PE1
MAC
Flip-Flop PE3 CE2
MAC based
Hash-Schemes)
PE2 PE4

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
 MPLS Transport & BGP Service
BGP L3VPN/ L3 EVPN BGP L2VPN EVPN
BGP Signaling BGP Signaling BGP Signaling BGP Signaling

PE2 PE4 PE2 PE4

CE1 MPLS CE2 CE1 MPLS CE2

PE1 PE3 PE1 PE3

Data Plane Data Plane

IP Packet Transport IP Packet L2 Frame Transport L2 Frame

MPLS Label MPLS Label
Service Service
BGP Label BGP Label
IP Packet L2 Frame

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
EVPN – Basic
Principles
EVPN Advantages:

Integrated • Integrated Layer 2 and Layer 3 VPN services

Services • L3VPN-like principles and operational experience for scalability and control
• All-active Multi-homing & PE load-balancing (ECMP)

• Fast convergence (link, node, MAC moves)

Network
• Control-Place (BGP) learning. PWs are no longer used.
Efficiency
• Optimized Broadcast, Unknown-unicast, Multicast traffic delivery

• Choice of MPLS, VxLAN or SRv6 data plane encapsulation

Service • Support existing and new services types (E-LAN, E-Line, E-TREE)
Flexibility • Peer PE auto-discovery. Redundancy group auto-sensing

Fully support IPv4 and IPv6 in the data plane and control plane
Investment •

Protection • Open-Standard and Multi-vendor support

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
L2VPN Services Overlay Encapsulation
MPLS Data Plane
Transport Label
Encapsulation
Service Label

Ethernet Header

IP Header Original Packet

Application Data

IP Data Plane
VXLAN IP Header UDP Header VXLAN Header Ethernet Header IP Header Application Data

Encapsulation Original Packet

SRv6 IPv6 Header Ethernet Header IP Header Application Data

Encapsulation Original Packet

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
L3VPN Services Overlay Encapsulation
MPLS Data Plane
Transport Label
Encapsulation
Service Label

IP Header
Original Packet
Application Data

VXLAN RFC7348 - Requires Inner Ethernet

IP Data Plane • Additional overhead for L3VPN / IP Forwarding

VXLAN IP Header UDP Header VXLAN Header Ethernet Header IP Header Application Data

Encapsulation Original Packet

SRv6 IPv6 Header IP Header Application Data

Encapsulation Original Packet

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Concepts
EVPN Instance (EVI) Ethernet Segment BGP Routes BGP Route Attributes

SHD CE1
Route Types Extended Communities

BD EVI ESI1 PE1 [1] Ethernet Auto-Discovery (AD) Route ESI MPLS Label

[2] MAC/IP Advertisement Route ES-Import

MHD CE2
[3] Inclusive Multicast Route MAC Mobility
BD
EVI

PE2
ESI2 [4] Ethernet Segment Route Default Gateway
PE
[5] IP Prefix Advertisement Route Encapsulation

• EVI identifies a VPN in the • Represents a ‘site’ • New SAFI [70] • New BGP extended
network connected to one or more • Routes serve control plane communities defined
• Encompass one or more PEs purposes, including: • Expand information carried
bridge-domains, • Uniquely identified by a MAC address reachability in BGP routes, including:
depending on service 10-byte global Ethernet MAC mass withdrawal MAC address moves
interface type Segment Identifier (ESI) Redundancy mode
Split-Horizon label adv.
Port-based • Could be a single device or
Aliasing MAC / IP bindings of a GW
VLAN-based (shown above) an entire network
Multicast endpoint discovery Split-horizon label encoding
VLAN-bundling Single-Homed Device (SHD)
Redundancy group discovery Data plane Encapsulation
Multi-Homed Device (MHD)
Designated forwarder election
Single-Homed Network (SHN)
IP address reachability
Multi-Homed Network (MHN)
L2/L3 Integration

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
 EVPN - Load-Balancing Modes
All-Active Single-Active Port-Active
(per flow) (per VLAN) (per port)

PE1 PE2 PE1 PE2 PE1 PE2

V1 V1 V1 V2 V1, V2

CE1 CE2 CE3

Single LAG at the CE Multiple LAGs at the CE Single LAGs at the CE

VLAN goes to both PE VLAN active on single PE Port active on single PE
Traffic hashed per flow Traffic hashed per VLAN Traffic hashed per port
Benefits: Bandwidth, Convergence Benefits: Billing, Policing Benefits: Protocol Simplification

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
EVPN - Ethernet VPN
• Concepts are same!!! Pick your side!

Pick your side!

SP1 SP2
PE2 PE4
CE1
PE1 PE3
L1 L2 L3 L4

C1 C2
VM VM VM VM

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
EVPN - Ethernet-Segment for Multi-Homing
L1 and L2 (L3 and L4) have to know if they multi-home same broadcast domain

The bundle on the Leafs SP1 SP2

connecting to a node should
have Identical ES identifier
(ESI)

L1 L2 L3 L4

Unique 10-byte global

identifier per Ethernet Segment Ethernet Segment represents a
C1 C2 node connected multiple Leaves

VM VM VM VM

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
 EVPN - Ethernet VPN
MAC address advertisement and MAC address table synchronization
Leaves run Multi-Protocol BGP to advertise & learn MAC addresses over the Network
MAC addresses are advertised to rest of Leaves
L3/4 – Learn MAC address advertised by L1
L2 – uses MAC address advertised by L1 to synchronize MAC address table
-> L2 forwards MAC via local ETH interface represented by same Ethernet Segment between L1 and L2

SP1 SP2

MAC advertisement &

learning/synchronization via BGP EVPN
NLRI

L1 L2 L3 L4
Data Plane learning
from the hosts
All Active multi-homing
C1 C2 Ethernet Segment
VM VM VM VM

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
EVPN L2 All-Active
Multihomed Service
EVPN - Testbed

RR103 RR104

LACP R39

H2
R38 R35

LACP R37 R34

H1
R36

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
EVPN Configuration
CE has to receive same lacp system
MAC

lacp system mac 3637.3637.3637 l2vpn

bridge group 100
interface Bundle-Ether100 bridge-domain 100
l2transport interface Bundle-Ether100
! !
! evi 100
!
evpn RT-2 MAC advertise !
evi 100 !
advertise-mac !
!
interface Bundle-Ether100
ethernet-segment
identifier type 0 36.37.00.00.00.00.00.11.00
!
!

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
EVPN Configuration - BGP

router bgp 1
bgp router-id 3.3.3.36
address-family l2vpn evpn
!
neighbor-group rr
remote-as 1
update-source Loopback0 BGP EVPN CP
address-family l2vpn evpn
!
neighbor 3.3.3.103
use neighbor-group rr
!
neighbor 3.3.3.104
use neighbor-group rr
!
!

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
 EVPN – Designated Forwarder (DF)
Challenge:
How to prevent duplicate copies of flooded traffic from being delivered to a multi-homed Ethernet Segment?
If (L3 and L4) Multi-Homing access via same Ethernet Segment -> only one of them can forward traffic to access
Same for (L1 and L2)

Why extra BUM Label?

What if Unicast Traffic is sent to L3 or L4 (not flooded)? -> DF Election applies only to BUM (from Core to Access)
DF, Redirect, Fast Re-Route (FRR), etc.
Service Label informs egress Leaf if traffic is BUM or Unicast

SP1 SP2

L1 L2 L3 L4
NDF DF
C1 Duplicate C2
#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
DF Election per EVI/ESI - Algorithm
Service Carving

Nodes Position EVIs

R36

R37
0

1
+ 100

EVI-ID modulo Number of Nodes = Position

100 modulo 2 = 0

R36 is DF for EVI-100

Who will be DF for EVI-101?

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Ethernet Segment R36#show evpn ethernet-segment esi 0036.3700.0000.0000.1100 carving detail
……

- DF Election
Ethernet Segment Id Interface Nexthops
------------------------ ---------------------------------- -------------------
-
0036.3700.0000.0000.1100 BE100 3.3.3.36
3.3.3.37
ES to BGP Gates : Ready
ES to L2FIB Gates : Ready
Main port :
Interface name : Bundle-Ether100
Interface MAC : 008a.9644.d8dd
IfHandle : 0x0800001c
State : Up
Redundancy : Not Defined
ESI type : 0
Value : 36.3700.0000.0000.1100
ES Import RT : 3637.0000.0000 (from ESI)
Source MAC : 0000.0000.0000 (N/A)
Topology :
Operational : MH, All-active
Configured : All-active (AApF) (default)
Service Carving : Auto-selection
Peering Details : 3.3.3.36[MOD:P:00] 3.3.3.37[MOD:P:00]
Service Carving Results:
Forwarders : 1
Permanent : 0
Elected : 1
EVI E : 100
Not Elected : 0
MAC Flushing mode : STP-TCN
Peering timer : 3 sec [not running]
Recovery timer : 30 sec [not running]
Carving timer : 0 sec [not running]
Local SHG label : 64005
Remote SHG labels : 1
64005 : nexthop 3.3.3.37

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 R36: RT-4 Ethernet Segment Router

R36#show bgp l2vpn evpn rd 3.3.3.36:0 [4][0036.3700.0000.0000.1100][32][3.3.3.36]/128

Mon Oct 15 03:24:50.736 UTC
BGP routing table entry for [4][0036.3700.0000.0000.1100][32][3.3.3.36]/128, Route Distinguisher: 3.3.3.36:0
Versions:
Process bRIB/RIB SendTblVer
RT-4 Ethernet Segment Identifier (ESI)
Speaker 82835 82835
Last Modified: Oct 14 21:32:13.399 for 05:52:37
Paths: (1 available, best #1)
Advertised to update-groups (with more than one peer):
0.2
Path #1: Received by speaker 0
Advertised to update-groups (with more than one peer):
0.2
Local
0.0.0.0 from 0.0.0.0 (3.3.3.36)
Origin IGP, localpref 100, valid, redistributed, best, group-best, import-candidate, rib-install
Received Path ID 0, Local Path ID 1, version 82835
Extended community: EVPN ES Import:3637.0000.0000 DF Election:00:0:00

Nodes which share same ESI import this route

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
EVPN – BUM Ingress Replication
Two service labels per EVPN instance
BUM Label – to forward Broadcast, Unknown Unicast and Multicast
Unicast Label – to forward Unicast

SP1 SP2

L1 L2 L3 L4

C1 C2
VM VM VM VM

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 R36: RT-3 Inclusive Multicast
R36#show bgp l2vpn evpn rd 3.3.3.36:100 [3][0][32][3.3.3.36]/80
Mon Oct 15 13:10:17.010 UTC
BGP routing table entry for [3][0][32][3.3.3.36]/80, Route Distinguisher: 3.3.3.36:100
Versions:
Process bRIB/RIB SendTblVer RT-3
Speaker 39774 39774
Last Modified: Aug 31 01:37:02.399 for 6w3d
Paths: (1 available, best #1)
Advertised to update-groups (with more than one peer):
0.2
Path #1: Received by speaker 0
Advertised to update-groups (with more than one peer):
0.2
Local
0.0.0.0 from 0.0.0.0 (3.3.3.36)
Origin IGP, localpref 100, valid, redistributed, best, group-best, import-candidate
Received Path ID 0, Local Path ID 1, version 39774
Extended community: RT:1:100 EVI 100 Route-Target
PMSI: flags 0x00, type 6, label 64120, ID 0x03030324

Ingress Replication Multicast (BUM) Label

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
EVPN – Split Horizon
Challenge:
How to prevent flooded traffic from echoing back to a multi-homed Ethernet
Segment?
Transport
BUM Label Label

SP1 SP2
SH Label

L1 L2

C1 Echo !

VM VM

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
EVPN – Split Horizon

Challenge:
How to prevent flooded traffic from echoing back to a multi-homed Ethernet
Segment?
Transport
BUM Label Label

SP1 SP2
SH Label

L1 L2

C1 Echo !
C11
VM VM VM VM

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
EVPN – MAC Mass-Withdraw

Challenge:
How to inform other Leafs of a failure affecting many MAC addresses quickly while
the control-plane re-converges?

SP2 MAC1 → ESI1 → Leaf1 + Leaf2

MAC1 can be SP1
reached via ESI1

L1 L2 L3 L4
MAC1 can NOT be
reached via ESI1
C1 C2
VM VM VM VM

ESI1 MAC1
#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
 R36: RT-1 Per ESI Ethernet Auto-Discovery

R36#show bgp l2vpn evpn rd 3.3.3.36:0 [1][3.3.3.36:1][0036.3700.0000.0000.1100][4294967295]/184

Sun Oct 14 20:56:59.687 UTC
BGP routing table entry for [1][3.3.3.36:1][0036.3700.0000.0000.1100][4294967295]/184, Route Distinguisher: 3.3.3.36:0
Versions:
Process bRIB/RIB SendTblVer RD - unique per advertising Ethernet Segment Identifier (ESI)
Speaker 76372 76372 RT-1 node (R36 unique)
Local Label: 0
Last Modified: Sep 18 23:02:40.399 for 3w4d
Paths: (1 available, best #1)
Advertised to update-groups (with more than one peer):
0.2
Path #1: Received by speaker 0
Advertised to update-groups (with more than one peer):
0.2
Local
0.0.0.0 from 0.0.0.0 (3.3.3.36)
Origin IGP, localpref 100, valid, redistributed, best, group-best, import-candidate, rib-install
Received Path ID 0, Local Path ID 1, version 76372
Extended community: EVPN ESI Label:0x00:64005 RT:1:100 EVI(s) Route-Target
All EVI(s) which use this ESI
Redundancy mode Split-Horizon Label
All-Active: 0x00
Single-Active: 0x01

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
 R36, R37, R38, R39 - EVPN Startup
R36 - Example
1. RT4: DF Election & Multi-Homed Ethernet
Segment Auto-Discovery
LACP R39
Service Carving: 100 modulo 2 = 0 H2
R36 is DF for EVI-100
R38 R35

RT-4 - DF Election

LACP R37 R34 RD: 1.1.1.36:1

H1 ESI: 0036.3700.0000.0000.1100

R36 Ext-Com: 3637.0000.0000 (RT)

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
 R36, R37, R38, R39 - EVPN Startup
R36 - Example
1. RT4: DF Election & Multi-Homed Ethernet
Segment Auto-Discovery
LACP R39
2. RT1: Per ESI Ethernet Auto-Discovery
(Split-Horizon, Mass-Withdraw) H2
RT-1 - Per ESI Ethernet AD
R38 R35
RD: 1.1.1.36:1

ESI: 0036.3700.0000.0000.1100

LACP R37 R34 Flag:0x00 All-Active

Ext-Com:
Split-Horizon Label: 64005
H1 Ext-Com: 1:100 (RT)

R36

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 R36, R37, R38, R39 - EVPN Startup
R36 - Example
1. RT4: DF Election & Multi-Homed Ethernet
Segment Auto-Discovery
LACP R39
2. RT1: Per ESI Ethernet Auto-Discovery
(Split-Horizon, Mass-Withdraw) H2
3. RT3: Inclusive Multicast RT-3 - Inclusive Multicast
R38 R35
RD: 1.1.1.36:100
Ext-Com: Type 6 Ingress-Replication
Multicast(BUM) Label: 64120
LACP R37 R34
Ext-Com: 1:100 (RT)

H1
R36

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
 BUM Forwarding

1. RT4: DF Election & Multi-Homed Ethernet

Segment Auto-Discovery
LACP R39
2. RT1: Per ESI Ethernet Auto-Discovery
(Split-Horizon, Mass-Withdraw) H2
3. RT3: Inclusive Multicast
R38 R35

LACP R37 R34

H1 Transport Label R38-9

R36 BUM Label R38-9/EVI100

BUM - Traffic
IR BUM - Traffic

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
 BUM Forwarding

1. RT4: DF Election & Multi-Homed Ethernet

Segment Auto-Discovery
LACP R39
2. RT1: Per ESI Ethernet Auto-Discovery
(Split-Horizon, Mass-Withdraw) H2
3. RT3: Inclusive Multicast
R38 R35

X
LACP R37 R34
Transport Label R37
H1 BUM Label R37/EVI100

R36 SH Label R37/ESIx

BUM - Traffic
IR BUM - Traffic

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
 R36: RT-2 MAC Advertisement
R36#show bgp l2vpn evpn rd 3.3.3.36:100 [2][0][48][0062.ec71.fbd7][0]/104
Mon Oct 15 04:33:39.527 UTC
BGP routing table entry for [2][0][48][0062.ec71.fbd7][0]/104, Route Distinguisher: 3.3.3.36:100
Versions:
Process bRIB/RIB SendTblVer
Speaker 83317 83317 RT-2 Advertised MAC
Local Label: 64004
3.3.3.37 (metric 30) from 3.3.3.103 (3.3.3.37)
Received Label 64004
Origin IGP, localpref 100, valid, internal, import-candidate, imported, rib-install
Received Path ID 0, Local Path ID 0, version 0
Extended community: SoO:3.3.3.37:100 RT:1:100
R37 MAC DP Learned and
Originator: 3.3.3.37, Cluster list: 3.3.3.103 Advertised
EVPN ESI: 0036.3700.0000.0000.1100
Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 3.3.3.37:100

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
 R36: RT-2 MAC Advertisement
R36#show evpn evi mac
Mon Oct 15 20:57:14.505 UTC

VPN-ID Encap MAC address IP address Nexthop Label

---------- ------ -------------- ---------------------------------------- --------------------------------------- --------
100 MPLS 0062.ec71.1000 :: 3.3.3.38 64006
100 MPLS 0062.ec71.1000 :: 3.3.3.39 64006
100 MPLS 0062.ec71.fbd7 :: 3.3.3.37 64004
100 MPLS 0062.ec71.fbd8 :: Bundle-Ether100 64004
100 MPLS 0062.ec71.fbd9 :: 3.3.3.37 64004
100 MPLS 0062.ec71.fbe0 :: 3.3.3.38 64006
100 MPLS 0062.ec71.fbe0 :: Learned and Advertised 3.3.3.39 64006
100 MPLS 0062.ec71.fbe1 :: MAC 3.3.3.38 64006
100 MPLS 0062.ec71.fbe1 :: 3.3.3.39 64006

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
 R36, R37, R38, R39 - EVPN Startup
R36 - Example
1. RT4: DF Election & Multi-Homed Ethernet
Segment Auto-Discovery
LACP R39
2. RT1: Per ESI Ethernet Auto-Discovery
(Split-Horizon, Mass-Withdraw) H2
3. RT3: Inclusive Multicast RT-2 - MAC Advertisement

4. RT2: MAC Advertisement R38 R35

RD: 1.1.1.36:100

ESI: 0036.3700.0000.0000.1100

LACP R37 R34

MAC: 0062.ec71.fbd7

H1 Label: 64004

R36 Ext-Com: 1:100 (RT)

L2 Frame SMAC:
0062.ec71.fbd7

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
 Unicast Forwarding

L2 Frame Flow1
Transport Label R36
1. RT4: DF Election & Multi-Homed Ethernet DMAC: H1

Segment Auto-Discovery RT-2 MAC Label/EVI

LACP R39
2. RT1: Per ESI Ethernet Auto-Discovery L2 Frame Flow1
DMAC: H1
(Split-Horizon, Mass-Withdraw) H2
3. RT3: Inclusive Multicast
4. RT2: MAC Advertisement R38 R35

LACP R37 R34

H1
R36
L2 Frame Flow1
DMAC: H1

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
EVPN – Aliasing

Challenge:
How to load-balance traffic towards a multi-homed device across multiple Leafs
when MAC addresses are learnt by only a single Leaf?

MAC1 can also be

SP2 MAC1 → ESI1 → Leaf1 + Leaf2
reached via ESI1 SP1

L1 L2 L3 L4
MAC1 can be
reached via ESI1
C1 C2
VM VM VM VM

ESI1 MAC1
#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
R36: RT-1 Per EVI Ethernet Auto-Discovery
RP/0/RP0/CPU0:R36#show bgp l2vpn evpn rd 3.3.3.36:100 [1][0036.3700.0000.0000.1100][0]/120
Mon Oct 15 03:35:13.604 UTC
BGP routing table entry for [1][0036.3700.0000.0000.1100][0]/120, Route Distinguisher: 3.3.3.36:100
Versions:
Ethernet Segment Identifier (ESI)
Process bRIB/RIB SendTblVer RT-1
Speaker 79640 7964
Last Modified: Oct 12 17:40:06.399 for 2d09h
Paths: (2 available, best #1)
Advertised to update-groups (with more than one peer):
0.2
Path #1: Received by speaker 0
Advertised to update-groups (with more than one peer):
0.2
Local
0.0.0.0 from 0.0.0.0 (3.3.3.36)
Origin IGP, localpref 100, valid, redistributed, best, group-best, import-candidate, rib-install
Received Path ID 0, Local Path ID 1, version 39769
Path #2: Received by speaker 0
Not advertised to any peer
Local
3.3.3.37 (metric 30) from 3.3.3.103 (3.3.3.37)
Received Label 64004 Aliasing Label allocated by R37 for EVI 100
Origin IGP, localpref 100, valid, internal, import-candidate, imported, rib-install
Received Path ID 0, Local Path ID 0, version 0
Extended community: RT:1:100
Originator: 3.3.3.37, Cluster list: 3.3.3.103
EVI 100 Route-Target
Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 3.3.3.37:100

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
 R36, R37, R38, R39 - EVPN Startup
R36 - Example
1. RT4: DF Election & Multi-Homed Ethernet
Segment Auto-Discovery
LACP R39
2. RT1: Per ESI Ethernet Auto-Discovery
(Split-Horizon, Mass-Withdraw) H2
3. RT3: Inclusive Multicast RT-1 - Per EVI Ethernet AD

4. RT2: MAC Advertisement R38 R35

RD: 1.1.1.36:100
5. RT1: Per EVI Ethernet Auto-Discovery
ESI: 0036.3700.0000.0000.1100

LACP R37 R34

Aliasing-Label: 64004

H1 Ext-Com: 1:100 (RT)

R36

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
 Unicast Forwarding
L2 Frame Flow1
Transport Label R36
1. RT4: DF Election & Multi-Homed Ethernet DMAC: H1

Segment Auto-Discovery RT-2 MAC Label/EVI100

LACP R39
2. RT1: Per ESI Ethernet Auto-Discovery L2 Frame Flow1
DMAC: H1
(Split-Horizon, Mass-Withdraw) H2
3. RT3: Inclusive Multicast
4. RT2: MAC Advertisement R38 R35
5. RT1: Per EVI Ethernet Auto-Discovery

LACP R37 R34

H1
R36
L2 Frame Flow1
DMAC: H1

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
 Unicast Forwarding
L2 Frame Flow2
DMAC: H1

L2 Frame Flow1
Transport Label R36
1. RT4: DF Election & Multi-Homed Ethernet DMAC: H1

Segment Auto-Discovery RT-2 MAC Label/EVI100

LACP R39
2. RT1: Per ESI Ethernet Auto-Discovery L2 Frame Flow1
DMAC: H1
(Split-Horizon, Mass-Withdraw) H2
3. RT3: Inclusive Multicast Transport Label R37
4. RT2: MAC Advertisement R38 R35
5. RT1: Per EVI Ethernet Auto-Discovery L2 Frame Flow2
RT1 Label/EVI100

DMAC: H1 L2 Frame Flow2

DMAC: H1

LACP R37 R34

H1
R36
L2 Frame Flow1
DMAC: H1

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
 Unicast Forwarding
L2 Frame Flow2
DMAC: H1

L2 Frame Flow1
Transport Label R36
1. RT4: DF Election & Multi-Homed Ethernet DMAC: H1

Segment Auto-Discovery RT-2 MAC Label/EVI100

LACP R39
2. RT1: Per ESI Ethernet Auto-Discovery L2 Frame Flow1
DMAC: H1
(Split-Horizon, Mass-Withdraw) H2
3. RT3: Inclusive Multicast Transport Label R37
4. RT2: MAC Advertisement R38 R35
5. RT1: Per EVI Ethernet Auto-Discovery L2 Frame Flow2
RT1 Label/EVI100

DMAC: H1 L2 Frame Flow2

Per Flow Balancing via R36 and R37 - Aliasing DMAC: H1

LACP R37 R34

H1
Per Flow Balancing via R36 and R37 - Aliasing
R36
L2 Frame Flow1
DMAC: H1

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 EVPN – MAC Mobility

Challenge:
How to detect the correct location of MAC after the movement of host from one Ethernet
Segment to another also called “MAC move”?

Sequence number and Next-Hop MAC IP ESI Seq. Next-

value will be changed after the host Hop
SP1 SP
move MAC-1 IP-1 0 1 Leaf-3

MAC IP ESI Seq. Next-

Hop
MAC-1 IP-1 0 0 Leaf-1
L1 L2 L3 L4 Sequence number is incremented
and Next-hop is changed to Leaf-3

C1 C2
VM
Host move

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
EVPN Single-Active
 EVPN - Load-Balancing Modes
All-Active Single-Active Port-Active
(per flow) (per VLAN) (per port)

PE1 PE2 PE1 PE2 PE1 PE2

V1 V1 V1 V2 V1, V2

CE1 CE2 CE3

Single LAG at the CE Multiple LAGs at the CE Single LAGs at the CE

VLAN goes to both PE VLAN active on single PE Port active on single PE
Traffic hashed per flow Traffic hashed per VLAN Traffic hashed per port
Benefits: Bandwidth, Convergence Benefits: Billing, Policing Benefits: Protocol Simplification

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
EVPN - Testbed

RR103 RR104

Single-Active
R39

H2
R38 R35

LACP R37 R34

H1
R36

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
All-Active R36#show evpn internal-label

Example
VPN-ID Encap Ethernet Segment Id EtherTag Label
---------- ------ --------------------------- -------- --------
100 MPLS 0038.3900.0000.0000.1100 0 68103
Summary pathlist:
0x02000001 3.3.3.38 68096
0x02000002 3.3.3.39 68096

R36#show mpls forwarding labels 68103 detail

Local Outgoing Prefix Outgoing Next Hop Bytes

Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
68103 68096 EVPN:100 3.3.3.38 0
Updated: Jan 27 07:50:05.582
Version: 42, Priority: 3
Label Stack (Top -> Bottom): { 68096 }
NHID: 0x0, Encap-ID: 0x1386f00000002, Path idx: 0, Backup path idx: 0, Weight: 0
MAC/Encaps: 0/4, MTU: 0
Packets Switched: 0

68096 EVPN:100 3.3.3.39 0

Updated: Jan 27 07:50:05.582
Version: 42, Priority: 3
Label Stack (Top -> Bottom): { 68096 }
NHID: 0x0, Encap-ID: 0x1387100000002, Path idx: 1, Backup path idx: 0, Weight: 0
MAC/Encaps: 0/4, MTU: 0
Packets Switched: 0

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Single-Active – Configuration and Verification
Remote R38/R39
R36#show evpn internal-label

VPN-ID Encap Ethernet Segment Id EtherTag Label

---------- ------ --------------------------- -------- -------- evpn
100 MPLS 0038.3900.0000.0000.1100 0 68103 interface Bundle-Ether100
Summary pathlist:
ethernet-segment
0x02000001 3.3.3.38 68096
0x00000000 3.3.3.39 (B) 68096
load-balancing-mode single-active
!
R36#show mpls forwarding labels 68103 detail !
Sun Jan 27 07:52:03.877 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
68103 68096 EVPN:100 3.3.3.38 0
Updated: Jan 27 07:51:14.370
Path Flags: 0x400 [ BKUP-IDX:1 (0x0) ]
Version: 47, Priority: 3
Label Stack (Top -> Bottom): { 68096 }
NHID: 0x0, Encap-ID: 0x1386f00000002, Path idx: 0, Backup path idx: 1, Weight: 0
MAC/Encaps: 0/4, MTU: 0
Packets Switched: 0

68096 EVPN:100 3.3.3.39 0 (!)

Updated: Jan 27 07:51:14.370
Path Flags: 0x300 [ IDX:1 BKUP, NoFwd ]
Version: 47, Priority: 3
Label Stack (Top -> Bottom): { 68096 }
NHID: 0x0, Encap-ID: 0x1387100000002, Path idx: 1, Backup path idx: 0, Weight: 0
MAC/Encaps: 0/4, MTU: 0
Packets Switched: 0
(!): FRR pure backup

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Single-Active ethernet-segment carving detail
R38#show evpn ethernet-segment esi 0038.3900.0000.0000.1100 carving detail

Ethernet Segment Id Interface Nexthops

------------------------ ---------------------------------- --------------------
0038.3900.0000.0000.1100 BE100 3.3.3.38
3.3.3.39
ES to BGP Gates : Ready
ES to L2FIB Gates : Ready
Main port :
Interface name : Bundle-Ether100
Interface MAC : 008a.967f.30dd
IfHandle : 0x0800002c
State : Up
Redundancy : Not Defined
ESI type : 0
Value : 38.3900.0000.0000.1100
ES Import RT : 3839.0000.0000 (from ESI)
Source MAC : 0000.0000.0000 (N/A)
Topology :
Operational : MH, Single-active
Configured : Single-active (AApS)
Service Carving : Auto-selection
Peering Details : 3.3.3.38[MOD:P:00] 3.3.3.39[MOD:P:00]
Service Carving Results:
Forwarders : 1
Permanent : 0
Elected : 1
EVI E : 100
Not Elected : 0
MAC Flushing mode : STP-TCN
Peering timer : 3 sec [not running]
Recovery timer : 30 sec [not running]
Carving timer : 0 sec [not running]
Local SHG label : 68098
Remote SHG labels : 1
68098 : nexthop 3.3.3.39

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
EVPN Port-Active
 EVPN - Load-Balancing Modes
All-Active Single-Active Port-Active
(per flow) (per VLAN) (per port)

PE1 PE2 PE1 PE2 PE1 PE2

V1 V1 V1 V2 V1, V2

CE1 CE2 CE3

Single LAG at the CE Multiple LAGs at the CE Single LAGs at the CE

VLAN goes to both PE VLAN active on single PE Port active on single PE
Traffic hashed per flow Traffic hashed per VLAN Traffic hashed per port
Benefits: Bandwidth, Convergence Benefits: Billing, Policing Benefits: Protocol Simplification

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
EVPN - Testbed

RR103 RR104

LACP R39

H2
R38 R35

R36/R37
LACP R37 R34
evpn
interface Bundle-Ether100
ethernet-segment H1
load-balancing-mode port-active

!
!
R36

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
 Port-Active –Verification
R36#show bundle R37#show bundle
Bundle-Ether100 Bundle-Ether100
Status: Up Status: LACP OOS (out of service)
Local links <active/standby/configured>: 1 / 0 / 1 Local links <active/standby/configured>: 0 / 1 / 1
Local bandwidth <effective/available>: 10000000 (10000000) kbps Local bandwidth <effective/available>: 0 (0) kbps
MAC address (source): 008a.9644.d8de (Chassis pool) MAC address (source): 008a.9644.08de (Chassis pool)
Inter-chassis link: No Inter-chassis link: No
Minimum active links / bandwidth: 1 / 1 kbps Minimum active links / bandwidth: 1 / 1 kbps
Maximum active links: 64 Maximum active links: 64
Wait while timer: 2000 ms Wait while timer: 2000 ms
Load balancing: Load balancing:
Link order signaling: Not configured Link order signaling: Not configured
Hash type: Default Hash type: Default
Locality threshold: None Locality threshold: None
LACP: Operational LACP: Operational
Flap suppression timer: Off Flap suppression timer: Off
Cisco extensions: Disabled Cisco extensions: Disabled
Non-revertive: Disabled Non-revertive: Disabled
mLACP: Not configured mLACP: Not configured
IPv4 BFD: Not configured IPv4 BFD: Not configured
IPv6 BFD: Not configured IPv6 BFD: Not configured

Port Device State Port ID B/W, kbps

Port Device State Port ID B/W, kbps -------------------- --------------- ----------- -------------- ----------
-------------------- --------------- ----------- -------------- ---------- Te0/0/0/0 Local Standby 0x8000, 0x0001 10000000
Te0/0/0/0 Local Active 0x8000, 0x0001 10000000 Link is in standby due to bundle out of service state
Link is Active

R37#show int bundle-ether 100

Bundle-Ether100 is down, line protocol is down

R37#show int tenGigE 0/0/0/0

TenGigE0/0/0/0 is up, line protocol is up

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
EVPN L2 &
L3 Integration
Distributed vs Centralized Routing
Layer2 Bridging mandatory between Leaves only Layer2 Bridging mandatory between Leaves and DCI

IRB
L4 X.X.X.H2/24 L4 X.X.X.H2/24

H2 H2
IRB

DCI2 SP2 L3 IRB

DCI2 SP2 L3

CO IRB
CO
IRB
L2 L2
DCI1 SP1 DCI1 SP1
H1 H1

X.X.X.H1/24 X.X.X.H1/24
L1 IRB L1

• Optimized forwarding of east-west traffic • All east<->west routed traffic traverses to centralized gateways
• ARP/MAC state localized to Leafs • Centralized gateways have full ARP/MAC state in the DCI
• Helps with horizontal scaling of DC • Scale challenge

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
EVPN Distributed
L3 Anycast Gateway
EVPN – Distributed Symmetric Anycast Gateway
Leaves run Multi-Protocol BGP to advertise & learn MAC + HOST IP addresses over the Network
MAC + IP addresses are advertised to rest of Leaves
L3/4 – Learn MAC + IP HOST address advertised by L1
-> L2/L3 update MAC address table + IP Forwarding table
L2 – uses MAC address advertised by L1 to synchronize MAC address table
-> L2 forwards MAC via local ETH interface represented by same Ethernet Segment between L1 and L2
L2 – uses MAC + IP HOST address advertised by L1 to synchronize ARP/ND information
-> L2 forwards IP via local ETH interface
Identical Anycast Gateway Virtual IP
Distributed Anycast Gateway serves and MAC address are configured
as the gateway for connected hosts SP1 SP2 on all the Leafs

BVI BVI BVI BVI

GW GW GW GW

L1 L2 L3 L4
All the BVIs perform active forwarding
in contrast to active/standby like First-
hop routing protocol
C1 C2
VM VM VM VM

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
EVPN – IRB in Network Fabric

Intra-subnet
Forwarding

Inter-subnet
Forwarding

SP1 SP2

BVI BVI BVI BVI

GW GW GW GW

L1 L2 L3 L4

C1 C2 C3 C4
VM VM VM VM

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
EVPN Distributed L3 Anycast GW - Symmetric IRB
Anycast IRB 192.168.2.1/24

RR103 RR104
IRB

LACP R39

H2: 192.168.2.20/24 H2 IRB

R38 R35

IRB

LACP R37 R34

H1: 192.168.1.10/24 H1
IRB
R36

Anycast IRB 192.168.1.1/24

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
EVPN Configuration - IRB
evpn
no evi 100
no advertise-mac
!

vrf a Not needed! We need MAC/IP RT-2

address-family ipv4 unicast
import route-target
100:100
!
export route-target
100:100
! VRF configuration
!
!

interface BVI100
host-routing MAC/IP RT2
vrf a
ipv4 address 192.168.1.1 255.255.255.0
mac-address 3637.3637.3637 Anycast Distributed IRB: Same IP and MAC
! R36,R37

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
EVPN Configuration - BGP VRF
router bgp 1
bgp router-id 3.3.3.36
address-family vpnv4 unicast
!
address-family l2vpn evpn
!
neighbor-group rr
remote-as 1
update-source Loopback0
address-family l2vpn evpn
!
neighbor 3.3.3.103
use neighbor-group rr
!
neighbor 3.3.3.104
use neighbor-group rr
!
vrf a
rd auto
address-family ipv4 unicast
additional-paths receive
maximum-paths ibgp 2 BGP Multi-Path for Inter-subnet forwarding
redistribute connected
!
!

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
 R36: RT-2 MAC/IP Advertisement
R36#show bgp l2vpn evpn rd 3.3.3.36:100 [2][0][48][0062.ec71.fbd7][32][19$
Tue Oct 16 02:47:45.576 UTC
BGP routing table entry for [2][0][48][0062.ec71.fbd7][32][192.168.1.10]/136, Route Distinguisher: 3.3.3.36:100
Versions:
Process bRIB/RIB SendTblVer
Speaker 84847 84847
Last Modified: Oct 15 23:14:52.399 for 03:32:53 RT-2 Advertised MAC IP
Paths: (2 available, best #1)
Advertised to update-groups (with more than one peer):
0.2
Path #1: Received by speaker 0
Advertised to update-groups (with more than one peer):
0.2
Local
0.0.0.0 from 0.0.0.0 (3.3.3.36)
Second Label 64008
Origin IGP, localpref 100, valid, redistributed, best, group-best, import-candidate, rib-install
Received Path ID 0, Local Path ID 1, version 84838
Extended community: SoO:3.3.3.37:100 RT:1:100 RT:100:100
EVPN ESI: 0036.3700.0000.0000.1100
Path #2: Received by speaker 0
RT EVI 100 and RT VRF A
Not advertised to any peer
Local
RT-2 per-BD label
3.3.3.37 (metric 30) from 3.3.3.103 (3.3.3.37)
Received Label 64004, Second Label 64008
Origin IGP, localpref 100, valid, internal, import-candidate, imported, rib-install
Received Path ID 0, Local Path ID 0, version 0 VRF Agg label
Extended community: SoO:3.3.3.37:100 RT:1:100 RT:100:100 RT EVI 100 and RT VRF A
Originator: 3.3.3.37, Cluster list: 3.3.3.103
EVPN ESI: 0036.3700.0000.0000.1100
Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 3.3.3.37:100
RP/0/RP0/CPU0:R36#

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
 R36: RT-2 MAC/IP
R36#show evpn evi mac
Tue Oct 16 02:52:22.437 UTC

VPN-ID Encap MAC address IP address Nexthop Label

---------- ------ -------------- ---------------------------------------- --------------------------------------- --------
100 MPLS 0062.ec71.fbd7 192.168.1.10 3.3.3.37 64004
65535 N/A 008a.9644.d8d8 :: Local 0

Learned and Advertised RT-2 per-BD label

MAC and IP

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
 R36: VRF Routes
R36#show route vrf a
Tue Oct 16 02:46:34.463 UTC

Codes: C - connected, S - static, R - RIP, B - BGP, (>) - Diversion path

D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - ISIS, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, su - IS-IS summary null, * - candidate default
U - per-user static route, o - ODR, L - local, G - DAGR, l - LISP
A - access/subscriber, a - Application route
M - mobile route, r - RPL, t - Traffic Engineering, (!) - FRR Backup path

Gateway of last resort is not set

C 192.168.1.0/24 is directly connected, 03:37:59, BVI100

L 192.168.1.1/32 is directly connected, 03:37:59, BVI100
B 192.168.1.10/32 [200/0] via 3.3.3.37 (nexthop in vrf default)
B 192.168.2.20/32 [200/0] via 3.3.3.38 (nexthop in vrf default), 03:28:28
[200/0] via 3.3.3.39 (nexthop in vrf default), 03:28:28

EVPN Learned Route BGP Multi Path to H2 connected to R38 and R39

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
 R36, R37, R38, R39 - EVPN Startup
R36 - Example Anycast IRB 192.168.2.1/24

1. RT4: DF Election & Multi-Homed Ethernet IRB

Segment Auto-Discovery
LACP R39
2. RT1: Per ESI Ethernet Auto-Discovery
(Split-Horizon, Mass-Withdraw) H2 IRB
3. RT3: Inclusive Multicast RT-2 - MAC Advertisement

4. RT2: MAC/IP Advertisement R38 R35

RD: 1.1.1.36:100

IRB ESI: 0036.3700.0000.0000.1100

LACP R37 R34

MAC: 0062.ec71.fbd7

H1 Label: 64004(BD) + 64008(VRF)

R36 IP: 192.168.1.10

L2 Frame SMAC: Ext-Com: 1:100 (RT) + VRF RT

0062.ec71.fbd7
IP Header SurceIP:
192.168.1.10 Anycast IRB 192.168.1.1/24

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
EVPN
Centralized GW
CGW
 EVPN Centralized Gateway (CGW)
CGW - Configuration
evpn
virtual access-evi
ethernet-segment
A1 identifier type 0 77.77.77.77.77.77.77.77.77

l2vpn
bridge group test
CE1 bridge-domain test
access-evi 300
routed interface BVI300

A2 CGW1

L2 EVPN L3 VPN
Core

A3 CGW2 Access - Configuration

evpn
evi 300
CE2
advertise-mac

l2vpn
A4 bridge group test
bridge-domain test
interface Bundle-Ether100
!
evi 300

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
 EVPN Centralized Gateway (CGW)
R28#show evpn ethernet-segment

Ethernet Segment Id Interface Nexthops

------------------------ ---------------------------------- --------------------
0077.7777.7777.7777.7777 Access-EVI:all 1.1.1.26
A1 1.1.1.28

RP/0/RSP0/CPU0:R28#show arp vrf a

CE1 -------------------------------------------------------------------------------
0/0/CPU0
-------------------------------------------------------------------------------
Address Age Hardware Addr State Type Interface
A2 CGW1 192.168.250.1 - a011.1111.1111 Interface ARPA BVI300
192.168.250.10 - 28ac.9ea7.d41b EVPN_SYNC ARPA BVI300
L2 EVPN L3 VPN
Core
CGW in Single-Active mode from Access-to-CGW (South->North)
Based on Access-EVI DF election NDF CGW BVI is added to Core SHG
A3 CGW2
prevents traffic from access-EVI go to BVI
allows traffic from BVI to Access-EVI
CE2
Single-Active South->North
All-Active North->South
A4

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Distributed vs Centralized Gateway
• Distributed Anycast Gateway is our priority!
• Best Scalable solution
• Optimal L2/L3 forwarding

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
BGP Layer3
Interconnect
 BGP Layer3 Interconnect
Principles

• DCI/BL provides Layer3 Interconnect Layer2 Bridging Required over Leaves

• DCI/BL participates in L3 Routing, but not in Layer2 Bridging
• DCI/BL summarization is required/recommended
IRB
L4 X.X.X.H2/24

H2

Z.Z.Z.CE1/24 DCI2 SP2 L3 IRB

CE1
PE/DCI
3 CORE CO
IRB
L2
DCI1 SP1
H1

X.X.X.H1/24
L1 IRB

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
BGP Layer3 Interconnect
DCI/BL Summarization
Host-Routes are not required outside CO/DC
L3/4 VRF FIB:
PE/DCI3 VRF FIB: DCI1/2 VRF FIB: X.X.X.H1 -> L1, L2
X.X.X.H1 -> DC1, DCI2 X.X.X.H1 -> L1, L2 X.X.X.H2 -> IRB(local)
X.X.X.H2 -> DCI1, DCI2 X.X.X.H2 -> L3, L4 X.X.X.0/24 -> IRB(local)
X.X.X.0/24 -> DC1, DCI2 X.X.X.0/24 -> L1, L2, L3, L4 Z.Z.Z.0/24 -> DCI1, DCI2
Z.Z.Z.0/24 -> CE1 Z.Z.Z.0/24 -> PE3
IRB
L4 X.X.X.H2/24

H2

Z.Z.Z.CE1/24 DCI2 SP2 L3 IRB L1/2 VRF FIB:

X.X.X.H1 -> IRB(local)
CE1
PE/DCI
3 CORE CO X.X.X.H2 -> L3, L4
X.X.X.0/24 -> IRB(local)
IRB Z.Z.Z.0/24 -> DCI1, DCI2
L2
DCI1 SP1
H1

X.X.X.H1/24
L1 IRB
#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
BGP Layer3 Interconnect
Control Plane
BGP - L3VPN VPNv4/6 BGP - L3VPN VPNv4/6 Option #1 – VPNv4/6 & VPNv4/6
BGP – EVPN L3 BGP – EVPN L3
Option #2 – EVPN & EVPN
BGP - L3VPN VPNv4/6 BGP – EVPN L3
Option #3 – VPNv4/6 & EVPN

IRB
L4 X.X.X.H2/24

H2

Z.Z.Z.CE1/24 DCI2 SP2 L3 IRB

CE1
PE/DCI
3 CORE CO
IRB
L2
DCI1 SP1
H1

L1 IRB
X.X.X.H1/24

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
BGP Layer3 Interconnect
Option #1 – VPNv4/6 & VPNv4/6
BGP - L3VPN VPNv4/6 BGP - L3VPN VPNv4/6
VPNv4: Z.Z.Z.0/24 VPNv4: Z.Z.Z.0/24

VPNv4: X.X.X.0/24 VPNv4: X.X.X.0/24

VPNv4: X.X.X.H1, X.X.X.H2

X
IRB
L4 X.X.X.H2/24

H2

Z.Z.Z.CE1/24 DCI2 SP2 L3 IRB

CE1
PE/DCI
3 CORE CO
IRB
L2
DCI1 SP1
H1

L1 IRB
X.X.X.H1/24

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
BGP Layer3 Interconnect
Option #2 – EVPN & EVPN
BGP – EVPN L3 BGP – EVPN L3
RT5: Z.Z.Z.0/24 RT5 Prefix: Z.Z.Z.0/24

RT5: X.X.X.0/24 RT5 Prefix: X.X.X.0/24

RT2 MAC/IP: X.X.X.H1, X.X.X.H2

X
IRB
L4 X.X.X.H2/24

H2

Z.Z.Z.CE1/24 DCI2 SP2 L3 IRB

CE1
PE/DCI
3 CORE CO
IRB
L2
DCI1 SP1
H1

X.X.X.H1/24
L1 IRB
#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
BGP Layer3 Interconnect
Option #3 – VPNv4/6 & EVPN
BGP - L3VPN VPNv4/6 BGP – EVPN L3
VPNv4: Z.Z.Z.0/24 RT5 Prefix: Z.Z.Z.0/24

VPNv4: X.X.X.0/24 RT5 Prefix: X.X.X.0/24

RT2 MAC/IP: X.X.X.H1, X.X.X.H2

X
IRB
L4 X.X.X.H2/24

H2

Z.Z.Z.CE1/24 DCI2 SP2 L3 IRB

CE1
PE/DCI
3 CORE CO
IRB
L2
DCI1 SP1
H1

X.X.X.H1/24
L1 IRB
#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
BGP Layer3 Interconnect
Control Plane Options Highlight
• Option #1 – VPNv4/6 & VPNv4/6
+ VPNv4/6 Industry proved solution for Layer3 VPN
+ DCI doesn’t need to understand BGP EVPN AF
- Leaf has to peer with Route-Reflector via both BGP EVPN and VPNv4/6 AF
EVPN AF to support L2 stretch (MAC advertisement) across DC/CO between Leaves
EVPN AF to sync ARP/ND for Multi-Homed All-Active
- DC/CO Route-Reflector has to support both BGP EVPN and VPNv4/6 AF
- Leaf has to advertise VM Host-Routes via VPNv4/6

• Option #2 – EVPN & EVPN

+ Single BGP Address Family End-To-End in Network
- Existing L3 VPNv4/6 services has to to migrated to L3 EVPN
No technical benefit to migrate existing L3 VPNv4/6 to L3 EVPN

• Option #3 – VPNv4/6 & EVPN

+ Recommended solution which benefits from both Options #1 and #2
+ New DC/CO - Leaf, Route-Reflector use single BGP AF EVPN
+ Existing L3 VPNv4/6 services stay untouched

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
R36: BGP Configuration - RT-5
router bgp 1
bgp router-id 3.3.3.36
address-family vpnv4 unicast
!
address-family l2vpn evpn
!
neighbor-group rr
remote-as 1
update-source Loopback0
address-family l2vpn evpn
advertise vpnv4 unicast
!
vrf a
rd auto
RT-5
address-family ipv4 unicast
additional-paths receive
maximum-paths ibgp 2
!

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
 R36: RT-5 Route
R36#show bgp l2vpn evpn rd 3.3.3.37:0 [5][0][24][192.168.1.0]/80
Tue Oct 16 03:35:06.480 UTC
BGP routing table entry for [5][0][24][192.168.1.0]/80, Route Distinguisher: 3.3.3.37:0
Versions:
Process bRIB/RIB SendTblVer
Speaker 84912 84912
Last Modified: Oct 16 03:23:18.399 for 00:11:48
Paths: (2 available, best #1) RT-5 prefi VRF A R37 RD
Not advertised to any peer x
Path #1: Received by speaker 0
Not advertised to any peer
Local
3.3.3.37 (metric 30) from 3.3.3.103 (3.3.3.37) VRF Agg label
Received Label 64008
Origin incomplete, metric 0, localpref 100, valid, internal, best, group-best, import-candidate, not-in-vrf
Received Path ID 0, Local Path ID 1, version 84912
Extended community: Flags 0x6: RT:100:100
Originator: 3.3.3.37, Cluster list: 3.3.3.103
EVPN ESI: 0000.0000.0000.0000.0000, Gateway Address : 0.0.0.0
Path #2: Received by speaker 0
VRF A Route-Target
Not advertised to any peer
Local
3.3.3.37 (metric 30) from 3.3.3.104 (3.3.3.37) VRF Agg label
Received Label 64008
Origin incomplete, metric 0, localpref 100, valid, internal, not-in-vrf
Received Path ID 0, Local Path ID 0, version 0
Extended community: Flags 0x6: RT:100:100
Originator: 3.3.3.37, Cluster list: 3.3.3.104 VRF A Route-Target
EVPN ESI: 0000.0000.0000.0000.0000, Gateway Address : 0.0.0.0
RP/0/RP0/CPU0:R36#

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
 R36: VRF A - Routing Table
R36#show route vrf a

C 192.168.1.0/24 is directly connected, 04:55:09, BVI100

L 192.168.1.1/32 is directly connected, 04:55:09, BVI100
B 192.168.1.10/32 [200/0] via 3.3.3.37 (nexthop in vrf default)
B 192.168.2.0/24 [200/0] via 3.3.3.38 (nexthop in vrf default), 00:40:26
[200/0] via 3.3.3.39 (nexthop in vrf default), 00:40:26
B 192.168.2.20/32 [200/0] via 3.3.3.38 (nexthop in vrf default), 00:40:26
[200/0] via 3.3.3.39 (nexthop in vrf default), 00:40:26
RP/0/RP0/CPU0:R36

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
EVPN Routes
Summary
EVPN Routes – Cheat Sheet
BGP Signaling PE1 – Advertises:
PE2 PE4
PE1 RT-4 Ethernet Segment Route
CE1 MPLS CE2 • I have ESI1 in case when someone needs this information for
EVI1-L Designated Forwarder(DF) Election
PE1 PE3
BE1-SHL
Data Plane
EVI1-BUML RT-1 Per ESI Ethernet Auto-Discovery (AD) Route
L2 Frame Transport L2 Frame
MPLS Label • I have ESI1
Service
• ESI1 is All-Active
BGP Label
L2 Frame
BD1 EVI1 • AC with ESI1 is connected to EVI1 and EVI2
BD1 MAC • My Split Horizon Label for ESI1 is BE1-SHL
MAC-A

BVI1
MAC-A -> BE1.1
IP-A VRF1 ARP RT-1 Per EVI Ethernet Auto-Discovery (AD) Route(s)
BE1 - ESI1
Vlan1 IP-A MAC-A -> BVI1 • EVI1 per-EVI (Aliasing) Label is EVI1-L
VRF1 IP-B MAC-B -> BVI2 • EVI2 per-EVI (Aliasing) Label is EVI2-L
Vlan2 VRF1-AGGL

BVI2 RT-3 Inclusive Multicast Route(s)

MAC-B BD2 MAC • EVI1 Label for BUM traffic is EVI1-BUML
IP-B MAC-B -> BE1.2 • EVI2 Label for BUM traffic is EVI2-BUML
BD2 EVI2
RT-2 MAC/IP Advertisement Route(s)
• MAC-A in EVI1 via label EVI1-L and IP-A in VRF1 via label VRF1-AGGL
EVI2-L
• MAC-B in EVI2 via label EVI2-L and IP-B in VRF1 via label VRF1-AGGL
BE1-SHL
EVI2-BUML RT-5 Prefix Advertisement Route(s)
VRF1-AGGL
• IPv4/6 prefix of BVI1 in VRF1 via label
• IPv4/6 prefix of BVI2 in VRF1 via label VRF1-AGGL

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
EVPN-VPWS
Multihomed Service
EVPN vs EVPN-VPWS - Balancing Mode

Single-Active
• Both EVPN and EVPN-VPWS advertise RT1(per-ESI) PE2 PE4
• Signal All-Active or Single-Active CE1 MPLS CE2
PE1 PE3

• Remote node performs per-flow load-balancing -> All-Active mode

• How remote node knows who is Active in Single-Active mode?

• EVPN
• Remote node follows MAC (RT2) advertisement -> node advertising MAC is active
• EVPN-VPWS
• Additional signaling per-service is required to inform remote node who is Active

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
EVPN-VPWS Layer 2 Attributes
Extended Community
+-------------------------------------------+

| Type (0x06) / Sub-type (0x04) (2 octets) | 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5

+-------------------------------------------+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| Control Flags (2 octets) | | MBZ |C|P|B| (MBZ = MUST Be Zero)

+-------------------------------------------+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| L2 MTU (2 octets) |

+-------------------------------------------+ Control-Word(C) = 4
| Reserved (2 octets) | Primary(P) = 2
Backup(B) = 1
+-------------------------------------------+

L2 MTU is a 2-octet value indicating the MTU in bytes

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
EVPN-VPWS
All-Active
 EVPN-VPWS - Testbed
Startup Sequence is almost identical with EVPN except:
RT3 and RT2 are not required

RR103 RR104 R38/R39

l2vpn
LACP R39 xconnect group 500
p2p 500
interface Bundle-Ether100
H2 neighbor evpn evi 500 service 333
!
!
R38 R35 !

R36
R37 R34 l2vpn
xconnect group 500
p2p 500
interface Bundle-Ether100
neighbor evpn evi 500 service 333

H1 R36 !
!

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
R36: L2vpn xconnect status & Data Plane verification
R36#show l2vpn xconnect
Legend: ST = State, UP = Up, DN = Down, AD = Admin Down, UR = Unresolved,
SB = Standby, SR = Standby Ready, (PP) = Partially Programmed

XConnect Segment 1 Segment 2

Group Name ST Description ST Description ST
------------------------ ----------------------------- -----------------------------
500 500 UP BE100 UP EVPN 500,3839,68106 UP
----------------------------------------------------------------------------------------

R36#show mpls forwarding labels 68106

Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
68106 68107 EVPN:500 3.3.3.38 0
68107 EVPN:500 3.3.3.39 0

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
R36: RT-1 Per EVI Ethernet Auto-Discovery
R36#show bgp l2vpn evpn rd 3.3.3.36:500 [1][0038.3900.0000.0000.1100][3839]/120
Control-Word(C) = 4
BGP routing table entry for [1][0038.3900.0000.0000.1100][3839]/120, Route Distinguisher: 3.3.3.36:500 Primary(P) = 2
Versions:
Process bRIB/RIB SendTblVer RT-1
Backup(B) = 1
Speaker 316 316 ESI R38/R39 AC-ID
Last Modified: Jan 27 08:24:37.527 for 00:01:42
Paths: (2 available, best #1)
Not advertised to any peer
Path #1: Received by speaker 0
Not advertised to any peer Control-Word + Primary
Local
3.3.3.38 (metric 30) from 3.3.3.103 (3.3.3.38) MTU 1500B
Received Label 68107
Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported, rib-install
Received Path ID 0, Local Path ID 1, version 314
Extended community: EVPN L2 ATTRS:0x06:1500 RT:1:500
Originator: 3.3.3.38, Cluster list: 3.3.3.103
Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 3.3.3.38:500
Path #2: Received by speaker 0
Not advertised to any peer Control-Word + Primary
Local
3.3.3.39 (metric 30) from 3.3.3.103 (3.3.3.39) MTU 1500B
Received Label 68107
Origin IGP, localpref 100, valid, internal, import-candidate, imported, rib-install
Received Path ID 0, Local Path ID 0, version 0
Extended community: EVPN L2 ATTRS:0x06:1500 RT:1:500
Originator: 3.3.3.39, Cluster list: 3.3.3.103
Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 3.3.3.39:500

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
R36: EVPN-VPWS Instance View
R36#show evpn evi vpn-id 500 detail

VPN-ID Encap Bridge Domain Type

---------- ------ ---------------------------- -------------------
500 MPLS VPWS:500 VPWS (vlan-unaware)
Stitching: Regular
Unicast Label : 0
Multicast Label: 0
Flow Label: N
Control-Word: Enabled EVPN-VPWS
Forward-class: 0 • No RT2 – MAC
Advertise MACs: No • No RT3 - BUM
Advertise BVI MACs: No
Aliasing: Enabled
UUF: Enabled
Re-origination: Enabled
Multicast source connected: No

Statistics:
Packets Sent Received
Total : 0 0
Unicast : 0 0
BUM : 0 0
Bytes Sent Received
Total : 0 0
Unicast : 0 0
BUM : 0 0
RD Config: none
RD Auto : (auto) 3.3.3.36:500
RT Auto : 1:500
Route Targets in Use Type
------------------------------ ---------------------
1:500 Import
1:500 Export

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
EVPN-VPWS
Single-Active
EVPN-VPWS - Testbed
Startup Sequence is almost identical with EVPN except:
RT3 and RT2 are not required
RR103 RR104

Single-Active
R39

H2
R38 R35

LACP R37 R34

H1
R36

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Config: EVPN-VPWS
R36 R38/R39
l2vpn l2vpn
xconnect group 500 xconnect group 500
p2p 500 p2p 500
interface Bundle-Ether100 interface Bundle-Ether100
neighbor evpn evi 500 target 3839 source 3637 neighbor evpn evi 500 target 3637 source 3839
! !
! !
! !

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
R36: L2vpn xconnect status & Data Plane verification
R36#show l2vpn xconnect

XConnect Segment 1 Segment 2

Group Name ST Description ST Description ST
------------------------ ----------------------------- -----------------------------
500 500 UP BE100 UP EVPN 500,3839,24004 UP
----------------------------------------------------------------------------------------

R36#show mpls forwarding labels 24004

Local Outgoing Prefix Outgoing Next Hop Bytes

Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------ Active
24004 28127 EVPN:500 3.3.3.39 0
28127 EVPN:500 3.3.3.38 0 (!)
Standby

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
R36: RT-1 Per EVI Ethernet Auto-Discovery
R36#show bgp l2vpn evpn rd 3.3.3.36:500 [1][0038.3900.0000.0000.1100][3839]/120
Tue Apr 14 07:47:20.033 UTC Control-Word(C) = 4
BGP routing table entry for [1][0038.3900.0000.0000.1100][3839]/120, Route Distinguisher: 3.3.3.36:500 Primary(P) = 2
Versions:
Process bRIB/RIB SendTblVer RT-1
Backup(B) = 1
Speaker 430 430 ESI R38/R39 AC-ID
Last Modified: Apr 14 07:47:09.651 for 00:00:10
Paths: (2 available, best #1)
Not advertised to any peer
Path #1: Received by speaker 0
Not advertised to any peer Control-Word + Backup
Local
3.3.3.38 (metric 30) from 3.3.3.103 (3.3.3.38) MTU 1500B
Received Label 28127
Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported, rib-install
Received Path ID 0, Local Path ID 1, version 428
Extended community: EVPN L2 ATTRS:0x05:1500 RT:1:500
Originator: 3.3.3.38, Cluster list: 3.3.3.103
Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 3.3.3.38:500
Path #2: Received by speaker 0
Not advertised to any peer Control-Word + Primary
Local
3.3.3.39 (metric 30) from 3.3.3.103 (3.3.3.39) MTU 1500B
Received Label 28127
Origin IGP, localpref 100, valid, internal, import-candidate, imported, rib-install
Received Path ID 0, Local Path ID 0, version 0
Extended community: EVPN L2 ATTRS:0x06:1500 RT:1:500
Originator: 3.3.3.39, Cluster list: 3.3.3.103
Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 3.3.3.39:500

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
VPLS to EVPN
Seamless Migration
 VPLS & EVPN Seamless Integration - Migration
VFI1 is by default in Split Horizon Group 1
CE2 R37 R39 CE4
• SHG1 protects loops in MPLS Core
• Full Mesh of pseudowires(PW) is required
for Any-to-Any forwarding
R36 MPLS
VFI1
PW_R37 UP
BD1 PW_R38 UP
PW_R39 UP
CE1
R38 CE3

l2vpn
bridge group 100
bridge-domain 100
vfi 1
neighbor x.x.x.37 pw-id 37
!
neighbor x.x.x.38 pw-id 38
!
neighbor x.x.x.39 pw-id 39
!
!

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
 VPLS & EVPN Seamless Integration - Migration
VFI1 is by default in Split Horizon Group 1
CE2 R37 R39 CE4
• SHG1 protects loops in MPLS Core
• Full Mesh of pseudowires(PW) is required
for Any-to-Any forwarding
R36 MPLS
VFI1
PW_R37 UP
EVI100 is also by default in Split Horizon Group 1
BD1 PW_R38 UP • R36 doesn’t forward data between VFI1 and EVI100
CE1 X PW_R39 UP

EVI100 R38 CE3

l2vpn
bridge group 100
bridge-domain 100
vfi 1
neighbor x.x.x.37 pw-id 37
!
neighbor x.x.x.38 pw-id 38
!
neighbor x.x.x.39 pw-id 39
!
evi 100
!

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
 VPLS & EVPN Seamless Integration - Migration
VFI1 is by default in Split Horizon Group 1
CE2 R37 R39 CE4
• SHG1 protects loops in MPLS Core
• Full Mesh of pseudowires(PW) is required
for Any-to-Any forwarding
R36 MPLS
VFI1
PW_R37 UP
EVI1 is also by default in Split Horizon Group 1
BD1 PW_R38 DOWN • R36 doesn’t forward data between VFI1 and EVI100
CE1 X PW_R39 UP
BGP EVPN
EVI100 R38 CE3 R36&R38 run BGP EVPN
• PW_R38 goes DOWN
• Data Forwarding between R36 and R38 via EVI100
l2vpn
bridge group 100
bridge-domain 100
vfi 1
neighbor x.x.x.37 pw-id 37
!
neighbor x.x.x.38 pw-id 38
!
neighbor x.x.x.39 pw-id 39
!
evi 100
!

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
PW to EVPN-VPWS
Seamless Migration
 EVPN-VPWS/Legacy-PW Seamless Migration
Supported Modes
CE1 PE38 MPLS PE39 CE2 Discovery: Static/BGP-AD
Signaling: LDP, BGP
LDP based PW
R38 Configuration
l2vpn
xconnect group test
p2p test
interface TenGigE0/0/0/0
neighbor ipv4 3.3.3.39 pw-id 10

R38#show l2vpn xconnect

XConnect Segment 1 Segment 2
Group Name ST Description ST Description ST
------------------------ ----------------------------- -----------------------------
test test UP Te0/0/0/0 UP 3.3.3.39 10 UP
----------------------------------------------------------------------------------------

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
EVPN-VPWS/Legacy-PW Seamless Migration
CE1 PE38 MPLS PE39 CE2

LDP based PW
R38 Configuration
l2vpn
xconnect group test Allows Tengig0/0/0/0 to be migrated
p2p test
vpws-seamless-integration Existing LDP based PW is UP and forwarding data
interface TenGigE0/0/0/0
neighbor ipv4 3.3.3.39 pw-id 10
New EVPN-VPWS service is ready and is signaled via BGP EVPN AF

p2p test-new
interface TenGigE0/0/0/0
neighbor evpn evi 1000 service 10
R38#show l2vpn xconnect

XConnect Segment 1 Segment 2

Group Name ST Description ST Description ST
------------------------ ----------------------------- -----------------------------
test test UP Te0/0/0/0 UP 3.3.3.39 10 UP
----------------------------------------------------------------------------------------
test test-new DN Te0/0/0/0 UP EVPN 1000,10,None DN
----------------------------------------------------------------------------------------

R38#show bgp l2vpn evpn rd 3.3.3.38:1000

Route Distinguisher: 3.3.3.38:1000 (default for vrf VPWS:1000)
*> [1][0000.0000.0000.0000.0000][10]/120
0.0.0.0 0 i

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
 EVPN-VPWS/Legacy-PW Seamless Migration
CE1 PE38 MPLS PE39 CE2

LDP based PW - DOWN

EVPN-VPWS - UP
R38 Configuration R39 Configuration
EVPN-VPWS is UP
l2vpn
xconnect group test
l2vpn LDP PW is Down and service is in “Seamless Inactive” mode
p2p test
xconnect group test p2p test can be removed
p2p test
vpws-seamless-integration
vpws-seamless-integration
interface TenGigE0/0/0/0
interface TenGigE0/0/0/0
neighbor ipv4 3.3.3.39 pw-id 10
neighbor ipv4 3.3.3.38 pw-id 10
p2p test-new
p2p test-new
interface TenGigE0/0/0/0
interface TenGigE0/0/0/0
neighbor evpn evi 1000 service 10
neighbor evpn evi 1000 service 10

R38#show l2vpn xconnect

XConnect Segment 1 Segment 2
Group Name ST Description ST Description ST
------------------------ ----------------------------- -----------------------------
test test DN Te0/0/0/0 SB(SI) 3.3.3.39 10 UP
----------------------------------------------------------------------------------------
test test-new UP Te0/0/0/0 UP EVPN 1000,10,3.3.3.39 UP
----------------------------------------------------------------------------------------

R38#show bgp l2vpn evpn rd 3.3.3.38:1000

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 3.3.3.38:1000 (default for vrf VPWS:1000)
*> [1][0000.0000.0000.0000.0000][10]/120
0.0.0.0 0 i
* i 3.3.3.39 100 0 i

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
EVPN & VPLS
Interconnect
 EVPN & VPLS Interconnect

CE2 A2 R37 PE2

LACP

MPLS Core/Access MPLS Core CE3

R36 PE1
CE1 A1
VPLS EVPN

R36/R37 Configuration R36 Configuration R37 Configuration

evpn l2vpn l2vpn
evi 100 bridge group 100 bridge group 100
advertise-mac bridge-domain 100 bridge-domain 100
! access-vfi 1 access-vfi 1
virtual vfi 1 neighbor x.x.x.A1 pw-id 1 neighbor x.x.x.A1 pw-id 10
ethernet-segment ! !
identifier type 0 11.11.11.11.11.11.11.11.11 neighbor x.x.x.A2 pw-id 2 neighbor x.x.x.A2 pw-id 20
! !
! !
Virtual Ethernet Segment (vES) evi 100 evi 100
• VPLS is Single-Active Access to EVPN

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
Summary
Summary
• Extra Offline Learning:
• ETREE
• EVPN Fast Re-Route (FRR)

• EVPN is an very important complement to BGP based services

• BGP is Unified Services Control Plane across Network

• EVPN All-Active Multihomed Service with Distributed Anycast Gateway & Integration
to L3VPN simplifies SPDC/NextGen-CO/WAN Integration
• EVPN is not strictly a replacement of “traditional” VPNv4/6
• EVPN and VPNv4/6 can coexist
• Service Layer is Data Plane independent, but the right Data Plane (encapsulation)
selection decreases complexity and provides additional capabilities
• Stay up to date https://e-vpn.io/

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
#CiscoLiveAPJC © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
Complete Your Session Evaluations

Complete a minimum of 4 session surveys and the Overall Event Survey to

claim a Cisco Live T-Shirt.

Complete your surveys in the Cisco Live mobile app.

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 115
 • Visit the Cisco Showcase
for related demos

• Book your one-on-one

Meet the Engineer meeting
Continue Attend the interactive education
your education
•
with DevNet, Capture the Flag,
and Walk-in Labs

• Visit the On-Demand Library

for more sessions at
www.CiscoLive.com/on-demand

BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 116
Thank you

#CiscoLiveAPJC
Extra Offline
Learning
EVPN ETREE
 EVPN ETREE – RT Constrains (Scenario 1a)
• Host connected to Leaf can talk ONLY to device connected to Root

• H1, H2, H3 can talk to H4

Leaf Additional Configuration
• H1, H2, H3 CANNOT talk to each other Prevents H1 and H2 to talk locally
Root Configuration
evpn l2vpn
evi 100 bridge group evpn
Leaf4 bgp
route-target export 1:1000
bridge-domain evpn100
interface TenGigE0/0/0/0
route-target import 1:1000 split-horizon group
route-target import 1:100 !
H3 ! interface Bundle-Ether100
split-horizon group
!

Leaf3
MPLS Root1 H4
H2
Leaf2 Leaf Configuration
evpn
evi 100
H1 bgp
route-target export 1:100
route-target import 1:1000
!
Leaf1 etree
rt-leaf <- MAC Synchronization
!

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
 EVPN ETREE Leaf Label (Scenario 1b)
Root Configuration
No specific Root Configuration
l2vpn
bridge group test • ASR9k/NCS add Leaf ACs to SHG2 automatically
Leaf4 bridge-domain test => Prevents local Leaf to Leaf AC forwarding
interface Bundle-Ether100
!
evi 300
H3

Leaf3
MPLS Root1 H4
H2
Leaf2
Leaf Configuration
l2vpn
H1 bridge group test
bridge-domain test
etree
leaf
Leaf1 !
interface Bundle-Ether100
!
evi 300

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
 EVPN ETREE Leaf Label (Scenario 1b) - BUM
Leaf Configuration
l2vpn Each Leaf (device with at least one Leaf AC) advertises RT1 per-ESI
bridge group test with ESI 0 with ETREE extended community to distribute ETREE Label
bridge-domain test
etree R28#show bgp l2vpn evpn rd 1.1.1.28:0 [1][1.1.1.28:1][0000.0000.0000.0000.0000][4294967295]/184
leaf Wed Mar 23 03:41:36.734 UTC
! BGP routing table entry for [1][1.1.1.28:1][0000.0000.0000.0000.0000][4294967295]/184, Route Distinguisher: 1.1.1.28:0
Versions:
interface Bundle-Ether100 Process bRIB/RIB SendTblVer
! Speaker 1481327 1481327
evi 300 Local Label: 0
Last Modified: Mar 23 03:21:20.580 for 00:20:17
Paths: (1 available, best #1)
Advertised to update-groups (with more than one peer):
0.2
Path #1: Received by speaker 0
Leaf4 Advertised to update-groups (with more than one peer):
0.2
Local
0.0.0.0 from 0.0.0.0 (1.1.1.28)
Origin IGP, localpref 100, valid, redistributed, best, group-best, import-candidate, rib-install
H3 Received Path ID 0, Local Path ID 1, version 1481327
Extended community: EVPN E-TREE:0x00:24010 RT:1:3000

Leaf3 ETREE Label works same as Split-Horizon Label (SHL)

SHL prevents BUM forwarding between two ACs with the same ESI
H2 ETREE Label prevents forwarding between Leaves ACs

Leaf2 Leaf to Leaf BUM traffic has ETREE Label

If Traffic with ETREE label is received cannot be forwarded to Leaf AC
H1 Root to Leaf or Leaf to Root BUM traffic doesn’t have ETREE label
BUM between Root <-> Leaf is allowed

Leaf1
#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 122
 EVPN ETREE Leaf Label (Scenario 1b) - Unicast
Leaf Configuration
l2vpn Leaf Advertises local MAC with ETREE extended community
bridge group test Same extended community was used to distribute ETREE Label
bridge-domain test
etree RP/0/RSP0/CPU0:R28#show bgp l2vpn evpn bridge-domain test [2][0][48][682c.7b24.c63d][0]/104
Wed Mar 23 04:13:10.244 UTC
leaf BGP routing table entry for [2][0][48][682c.7b24.c63d][0]/104, Route Distinguisher: 1.1.1.28:300
! Versions:
interface Bundle-Ether100 Process bRIB/RIB SendTblVer
Speaker 1481349 1481349
! Local Label: 24012
evi 300 Last Modified: Mar 23 03:21:48.580 for 00:51:22
Paths: (1 available, best #1)
Advertised to update-groups (with more than one peer):
0.2
Path #1: Received by speaker 0

Leaf4 Advertised to update-groups (with more than one peer):

0.2
Local
0.0.0.0 from 0.0.0.0 (1.1.1.28)
Origin IGP, localpref 100, valid, redistributed, best, group-best, import-candidate, rib-install

H3 Received Path ID 0, Local Path ID 1, version 1481349

Extended community: SoO:1.1.1.28:300 EVPN E-TREE:0x01:0 RT:1:300
EVPN ESI: 0026.2826.2826.2826.2802

Leaf3 ETREE Label is set to 0, but Leaf Flag is set to 1

H2 Unicast traffic is filtered by ingress node

If traffic is originated from Leaf AC and destination is local/remote Leaf AC frame is dropped
Leaf2

H1

Leaf1
#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 123
 EVPN ETREE Leaf Label (Scenario 2) per-AC
Root/Leaf Configuration
l2vpn
bridge group test
bridge-domain test
interface Bundle-Ether100 <- interface to H4
Leaf4 interface Bundle-Ether200 <- interface to H5
etree
leaf
!
H3 !
evi 300
H5
Leaf3
MPLS
H2 Root
Leaf2 Leaf H4 Leaf Configuration
Same as Scenario 1b
H1
l2vpn
bridge group test
bridge-domain test
Leaf1 etree
leaf
!
interface Bundle-Ether100
!
evi 300

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 124
EVPN ETREE Summary
Scenario 1a: RT Constrains is simple and HW “friendly”
Unicast/BUM filtering by ingress node => scale benefit

Scenario 1b: Simple configuration, but additional ETREE label must be imposed for BUM
BUM filtered by egress node
Support IRB

Scenario 2: Same principle as Scenario 1b also compatible with Scenario 1b

ASR9k allows to combine Root/Leaf ACs in the same Bridge-Domain
Support IRB

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 125
EVPN Fast Re Route (FRR)
Fast Convergence (FRR Data Plane) - Core
Core Failure (Link/Node) – PIC Core
Technology: RSVP-TE/LFA/rLFA/TI-LFA
Transport: IGP -> MPLS, SRv6
Overlay Service: Service Independent
Device: P-Router, Spine

PE2 PE4 L2
P2 S2

X
P1 X
S1
PE1 PE3 L1

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 127
Fast Convergence (FRR Control Plane) – DC Leaf/TOR
MAC Mobility
VM/MAC Move
Technology: EVPN Mac Mobility (EVPN RT-2)
Transport: Transport Independent
Overlay Service: EVPN MAC IP ESI Seq. Next-
Device: Leaf/TOR Hop
MAC-1 IP-1 0 1 Leaf-3/4
Sequence number is incremented and
L4 Next-hop is changed to Leaf-3/4

VM1

Sequence number and Next-Hop L3 S2

Move
value will be changed after the host
move
L2 S1

VM1
MAC IP ESI Seq. Next-
Hop
L1
MAC-1 IP-1 0 0 Leaf-1/2

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 128
Fast Convergence (FRR CP/DP) – Edge/Leaf/TOR
Leaf/TOR Failure (Link) – EVPN Mass Withdraw
Technology: EVPN RT1 Mass Withdraw
Transport: Transport Independent
Overlay Service: EVPN
Device: Leaf/TOR/Access/Edge

PE2
MAC-CE1 -> ESI1 -> PE1
PE4 X
-> PE2
P2
CE1
P1
PE1 PE3
RT1 ESI1 Mass Withdraw

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
Fast Convergence (FRR Data Plane) – Edge L3VPN
Edge Failure (Link) – BGP PIC Edge
Technology: BGP PIC Edge
Transport: MPLS, SRv6 (Transport Independent)
Overlay Service: L3VPN
Device: Access/PE
BGP CE-PE is mandatory!!!

PE2
P2

CE1 L3VPN
P1
PE1

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 130
Fast Convergence (FRR Data Plane) – Edge L2VPN
Edge Failure (Link) – EVPN FRR
Technology: EVPN FRR
Transport: Transport Independent
Overlay Service: EVPN
Device: Access/PE/Leaf/TOR

All-Active Single-Active

PE2 PE2
P2 P2

CE1 EVPN CE1 EVPN

P1 P1
PE1 PE1

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 131
Fast Convergence (EVPN FRR Data Plane) – Edge

• Single-Active NDF filter traffic in both directions

• Re-Directed traffic will be re-directed back to PE1 (L3 Loop) or dropped

• Solution is to bypass NDF => Only redirected packet can bypass NDF!
• Extra FRR label is used to bypass NDF
• FRR Label is used for both All-Active and Single-Active access

All-Active Single-Active

PE2 PE2
P2 P2

CE1 EVPN CE1 EVPN

P1 P1
PE1 PE1

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 132
Solving the EVPN DF Election Problem on Recovery

• NTP based solution

• Clocks are synchronized, stratum 3
• Clock timestamping exchange between peering PE
• Service Carving Synchronization

Failure Recovery

PE2 PE2
P2 P2

CE1 EVPN CE1 EVPN

P1 P1
PE1 PE1

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 133
EVPN FRR - Configuration
All-Active Single-Active
evpn evpn
interface Bundle-Ether100 interface Bundle-Ether100
ethernet-segment ethernet-segment
identifier type 0 36.37.36.37.36.37.36.37.01 identifier type 0 36.37.36.37.36.37.36.37.01
convergence load-balancing-mode single-active
reroute convergence
reroute

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 134
 DF Election Convergence Improvements
evpn
interface Bundle-Ether100
ethernet-segment
identifier type 0 36.37.36.37.36.37.36.37.01 BGP Next-Hop Tracking for RT4
load-balancing-mode single-active
convergence
Node Failure Convergence
nexthop-tracking Improvement
reroute
NTP Timestamping for RT4

R37#show evpn ethernet-segment carving detail

Service Carving Synchronization:
Mode : NTP_SCT
Peer Updates :
3.3.3.36 [SCT: 2020-10-28 12:57:47:456146]
3.3.3.37 [SCT: 2020-10-28 12:57:47:451599] NTP Timestamping for RT4

R37#show ntp status

Clock is synchronized, stratum 3, reference is 10.255.11.1

R37#show bgp l2vpn evpn rd 3.3.3.36:0 [4][0036.3736.3736.3736.3701][32]

3.3.3.36 (metric 30) from 3.3.3.103 (3.3.3.36)
Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, not-in-vrf
Received Path ID 0, Local Path ID 1, version 1359
Extended community: EVPN ES Import:3637.3637.3637 DF Election:0:0x0008:0 EVPN NTP: 3812880149.4488
Originator: 3.3.3.36, Cluster list: 3.3.3.103

#CiscoLiveAPJC BRKMPL-2253 © 2024 Cisco and/or its affiliates. All rights reserved. Cisco Public 135
Thank you

#CiscoLiveAPJC
#CiscoLiveAPJC