Troubleshooting BGP

Vignesh Rajendran Praveen - High Touch Engineer, Cisco Services

CCIE# 34503
Internet
The Two Napkin Protocol
Why Should You Care?

• Common BGP issues

Problem

• BGP for Enterprises

Troubleshooting
TAC Example
Steps

• BGP for Service Providers

Solution
Agenda
• BGP peering issues
• Session not coming up, Session flapping

• Platform issues due to BGP

• High CPU, Memory utilisation

• BGP Slow-peer issue

• BGP for Enterprises
• Multi-homing, Load-sharing

• BGP for Service Providers

• MPLS VPN, BGP PIC
Scenario 1 - Failed BGP Peering
Problem Description

• iBGP / eBGP is not establishing

• Newly configured BGP session not coming up
• Session was up before, but not coming up now

OSPF OSPF
R1 R2 R3 R1 R2 R3
iBGP eBGP iBGP eBGP
AS65535 AS65534 AS65535 AS65534

Physical Topology Logical Topology

Failed BGP Peering
Configuration
Check
OSPF ✓ AS Numbers

R1 R2 R3
✓ Peering IP

iBGP eBGP
AS65535 AS65534  eBGP Multihop?

router bgp 65535 router bgp 65534

bgp router-id 2.2.2.2 bgp router-id 3.3.3.3
bgp log-neighbor-changes bgp log-neighbor-changes
neighbor 1.1.1.1 remote-as 65535 neighbor 10.23.23.2 remote-as 65535
neighbor 1.1.1.1 update-source Loopback0 . . .
neighbor 10.23.23.5 remote-as 65534
. . .
Failed BGP Peering
Reachability

router bgp 65535

Lo0 = 1.1.1.1/32 AS65535 Lo0 = 2.2.2.2/32
bgp router-id 1.1.1.1
bgp log-neighbor-changes R1 Rn R2
neighbor 2.2.2.2 remote-as 65535
neighbor 2.2.2.2 update-source Loopback0
...

R1# ping 2.2.2.2 source loopback0

Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
.....
Success rate is 0 percent (0/5)
Failed BGP Peering
Verify any Firewall / ACL in path for TCP port 179
ASA_FW# sh run access-list AS65535 AS65534

access-list OUT extended permit icmp any any

R2 R3
access-list OUT extended permit ospf any any
access-list OUT extended permit tcp any any eq telnet
. . . .

Rn# sh ip access-list R1_R2

Lo0 = 1.1.1.1/32 AS65535 Lo0 = 2.2.2.2/32
permit icmp any any
permit ospf any any R1 Rn R2

permit tcp host 10.12.12.1 eq bgp 2.2.2.2

permit tcp host 10.12.12.1 2.2.2.2 eq bgp
ACL
. . . .
Failed BGP Peering
Verify TCP session
R2#sh tcp brief
TCB Local Address Foreign Address (state)
65F19834 2.2.2.2.179 1.1.1.1.46523 ESTAB

Quick test when BGP is down

R1#telnet 2.2.2.2 179 /source-interface loopback 0
Trying 2.2.2.2 ...
% Destination unreachable; gateway or host down

R1#
Failed BGP Peering
Blocked process in XR
RP/0/RSP0/CPU0:ASR9010-B# show process bgp
Mon Jun 3 09:47:12.646 EST
• Ensure BGP process is in Run Job Id: 1040
state. PID: 307494
Executable path: /disk0/iosxr-routing-
4.2.3/bin/bgp
• Check for blocked BGP or Instance #: 1
TCP process on the RP / LC Version ID: 00.00.0000
Respawn: ON
using show process blocked Respawn count: 1
command. Max. spawns per minute: 12
Last started: Tue May 28 14:35:50
2013
Process state: Run
Package state: Normal
Started on config: default
. . . .
Failed BGP Peering
Show process bgp (contd. Output)
RP/0/RSP0/CPU0:ASR9010-B# show process bgp
<snip>
10 2 488K 10 Nanosleep 0:00:02:0004 0:00:00:0847 bgp
1049 13 3 488K 10 Receive 0:00:00:0811 6:36:52:0264 bgp
1049 14 3 488K 10 Condvar 14:56:55:0236 9:07:49:0890 bgp
1049 15 0 488K 10 Condvar 14:56:55:0240 25:09:49:0542 bgp
1049 16 3 488K 10 Running 0:00:00:0000 57:53:33:0110 bgp
1049 17 1 488K 10 Receive 0:00:28:0379 0:00:00:0066 bgp
1049 18 1 488K 10 Mutex 13:15:50:0870 3:31:49:0712 bgp
<snip>

• You can also use “show process blocked” to check the blocked
processes
Failed BGP Peering
Packet Capture
Use SPAN on Cisco Switches to get traffic to your sniffer

And EPC on Cisco Routers using IOS & IOS-XE

IOS-XR
- Only supported on ASR-9000
- Use ACLs to control what packets to SPAN
TAC Case Example - 1
BGP Peering down

• Customer reported new iBGP peer not coming up

between two routers using different IOS codes.
• Configuration verified
AS100

• “show ip bgp summary” shows BGP R1 R2

state changes from Idle to Active and

then to Closing state
• TCP session goes to Established but then
immediately moves to CloseWait
TAC Case Example - 1
Show log | in BGP
R2#
*Jun 5 18:18:04.667: %BGP-3-NOTIFICATION: sent to neighbor
10.1.12.1 active 2/7 (unsupported/disjoint capability) 0 bytes
R2#
*Jun 5 18:18:04.671: %BGP-4-MSGDUMP: unsupported or mal-
formatted message received from 10.1.12.1:
FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 002D 0104 0064 00B4
0101 0101 1002 0601 0400 0100 0102 0280 0002 0202 00
TAC Case Example - 1
Resolution
• Analysis
- Capability 131 is set when BGP is trying to establish multisession
- The other side did not understand this capability i.e. Single-session

• Resolution
- Configure both sessions to use same capability i.e. Single-session / Multi-session
- neighbor <ip-addr> transport single-session | multi-session
Scenario 2 - BGP Peer Flapping
Problem Description

• Multiple BGP Sessions Flapping

• Keeps oscillating between two states (Idle/Established)
• Common reasons AS 65534

iBGP

 MTU Mismatch eBGP

 High CPU
 Interface Input-Queue Drops iBGP

AS 65535
BGP Peer Flapping
Notifications
R2#
*Mar 24 20:25:47.262: %BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down BGP
Notification sent
*Mar 24 20:25:47.262: %BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 4/0
(hold time expired) 0 bytes

• BGP NOTIFICATIONs consist of an error code, sub-code and data

- All Error Codes and Sub-codes can be found here
- http://www.iana.org/assignments/bgp-parameters/bgp-parameters.xml
- http://tinyurl.com/bgp-notification-codes
- Data portion may contain what triggered the notification
- Example: corrupt part of the UPDATE
Notifications Contd…
%BGP-3-NOTIFICATION: sent to neighbor 2.2.2.2 2/2 (peer in wrong AS) 2 bytes 00C8
FFFF FFFF FFFF FFFF FFFF FFFF FFFF FFFF 002D 0104 00C8 00B4 0202 0202 1002 0601 0400
0100 0102 0280 0002 0202 00

Value Name Reference

1 Message Header Error RFC 4271

2 OPEN Message Error RFC 4271

3 UPDATE Message Error RFC 4271

4 Hold Timer Expired RFC 4271

5 Finite State Machine Error RFC 4271

6 Cease RFC 4271

The first 2 in “2/2” is the Error Code….so “OPEN Message Error”

Notifications Contd…
Subcode # Subcode Name Subcode Description

1 Unsupported BGP version The version of BGP the peer is running isn’t compatible with
the local version of BGP
2 Bad Peer AS The AS this peer is locally configured for doesn’t match the
AS the peer is advertising
3 Bad BGP Identifier The BGP router ID is the same as the local BGP router ID

4 Unsupported Optional Parameter There is an option in the packet which the local BGP
speaker doesn’t recognise
6 Unacceptable Hold Time The remote BGP peer has requested a BGP hold time
which is not allowed (too low)
7 Unsupported Capability The peer has asked for support for a feature which the local
router does not support

OPEN Message Subcodes shown above

The second 2 in “2/2” is the Error Subcode….so “Bad Peer AS”
BGP Peer Flapping
MTU Mismatch
R2#show ip bgp neighbors 1.1.1.1 | in max data segment
Datagrams (max data segment is 9176 bytes):
Lo0 = 1.1.1.1/32 AS65535 Lo0 = 2.2.2.2/32

R2# ping 1.1.1.1 source 2.2.2.2 df-bit size 1500

Type escape sequence to abort. R1 R2
Sending 5, 1500-byte ICMP Echos to 1.1.1.1, timeout is
2 seconds:
Packet sent with a source address of 2.2.2.2
Packet sent with the DF bit set
1500
!!!!!
R2# ping 1.1.1.1 source 2.2.2.2 df-bit size 9216
Type escape sequence to abort.  BGP OPENs and Keepalives are
Sending 5, 9216-byte ICMP Echos to 1.1.1.1, timeout is
2 seconds: small
Packet sent with a source address of 2.2.2.2
Packet sent with the DF bit set  UPDATEs can be much larger
.....
 Maybe small packets work but
larger packets do not?
MTU – 20byte IP header – 20 byte TCP header = MSS
BGP Peer Flapping
High CPU – Victim BGP
• High CPU can cause data and control
plane packet loss
• High CPU can be caused due to
process or interrupt (traffic hitting CPU)
• Example – Packets with TTL set to 1
are punted to CPU

PE1_RTR# sh proc cpu sorted | ex 0.00

CPU utilization for five seconds: 92%/91%; one minute: 14%; five minutes: 8%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
82 7490 929 8062 0.29% 2.96% 1.52% 0 Exec
4 6308484 908039 6947 0.05% 0.08% 0.06% 0 Check heaps
BGP Peer Flapping
High CPU - Embedded Event Manager (EEM)
• Serves as a powerful tool for high CPU troubleshooting
• Triggered based on event and thresholds
• Multiple actions can be set based on events
event manager applet HIGHCPU
event snmp oid "1.3.6.1.4.1.9.9.109.1.1.1.1.3.1" get-type exact entry-op gt entry-val "90"
exit-op lt exit-val "70" poll-interval 5 maxrun 200
action 1.0 syslog msg "START of TAC-EEM: High CPU"
action 1.1 cli command "enable”
action 1.4 cli command "debug netdr capture“
action 2.0 cli command "sh clock | append disk0:proc_CPU"
action 2.1 cli command "show process cpu sorted | append disk0:proc_CPU“
action 2.2 cli command "show proc cpu history | append disk0:proc_CPU"
action 2.3 cli command "show netdr | append disk0:proc_CPU"
action 3.1 cli command "show log | append disk0:proc_CPU"
action 4.0 syslog msg "END of TAC-EEM: High CPU"
BGP Peer Flapping
Interface Input-Queue Drops
PE_RTR# sh int gi 0/1 | in drop
Input queue: 5/75/4351/0 (size/max/drops/flushes); Total output drops: 0
Input-Queue drops can
PE_RTR# show buffer input-interface gig0/1 header also lead to multiple
Control-Plane packet loss
Buffer information for Big buffer at 0x2A5BEBF4 data_area 0xC49ED84, refcount 1, next
0x402CBA18, flags 0xA80 along
linktype 7 (IP), enctype 1 (ARPA), encsize 14,with data loss
rxtype
1 if_input 0x401620E0 (GigabitEthernet0/1), if_output 0x0 (None) inputtime 16w2d
(elapsed 00:00:00.004) outputtime 00:00:00.000 (elapsed never), oqnumber
65535 datagramstart 0xC49EDCA, datagramsize 1514, maximum size 1692 mac_start
0xC49EDCA, addr_start 0xC49EDCA, info_start 0x0 network_start 0xC49EDD8,
transport_start 0xC49EDEC, caller_pc 0x211295A4 source: 33.33.33.1, destination:
33.33.33.2, id: 0x370F, ttl: 255, prot: 1
TAC Case Example - 2
BGP Session Flapping

Lo0 = 1.1.1.1/32 AS65535 Lo0 = 5.5.5.5/32

R1 R5

R2 4000 R3 2000 R4

• Customer reported that the BGP session between R1 & R5 flapped once
in every 3 minutes.
• Issue started after the number of prefixes exchanged was increased from
100 to 100,000.
• Requested “show ip bgp neighbor <nei-ip>”
TAC Case Example - 2
PMTUD

Lo0 = 1.1.1.1/32 AS65535 Lo0 = 5.5.5.5/32

R1 R5

R2 4000 R3 2000 R4

R1#show ip bgp neighbor 5.5.5.5 | i

ACL blocking ICMP tcp|segment

Transport(tcp) path-mtu-discovery is
enabled
• BGP Path MTU Discovery failure.
Datagrams (max data segment is 3960
bytes):

R1#
TAC Case Example - 2
Resolution
• Globally change the TCP MSS negotiation value
- ip tcp mss <mss-value>

• Modify the way ICMP messages are filtered by the ACL

- permit icmp any any unreachable

• Disable BGP path mtu discovery on the BGP peers under router config mode
- no bgp transport path-mtu-discovery
Agenda
• BGP peering issues
• Session not coming up, Session flapping

• Platform issues due to BGP

• High CPU, Memory utilisation

• BGP slow-peer issue

• BGP for Enterprises
• Multi-homing, Load-sharing

• BGP for Service Providers

• MPLS VPN, BGP PIC
Scenario 3 – High CPU due to BGP
Problem Description
• High CPU noticed due to BGP
- BGP Scanner
- BGP Router
• Symptoms
- Router unstable
- Traffic loss
- Loss of Manageability
High CPU due to BGP
BGP Scanner
• Can be expected for short durations carrying large Internet routing table
• High cpu condition varies with the number of neighbors and the number of
routes learned per neighbor
• Make use of maximum-prefix knob where required

------------------ show process cpu ------------------

CPU utilization for five seconds: 99%/0%; one minute: 28%; five minutes: 17%

143 393745836 2319493 169755 71.49% 15.34% 12.38% 0 BGP Scanner

. . . .
High CPU due to BGP
BGP Router

Router#show process cpu

CPU utilization for five seconds: 100%/0%; one minute: 99%; five minutes: 81%
....
139 6795740 1020252 6660 88.34% 91.63% 74.01% 0 BGP Router

• Look at the scenario

• Is BGP going through “Initial Convergence”?
• Are there any route churns?
• The high cpu on the device could also be due to the instability of the BGP table.
(Receiving two copies of routing table – one from iBGP and one from eBGP)
High CPU due to BGP
Route Churn (Flapping Routes)
• How to identify route churn?
- Do “sh ip bgp summary | in table”, note the table version
- Wait 4-5 seconds
- Do “sh ip bgp summary | in table”, compare the table version from 4-5 seconds ago
• You have 150k routes and see the table version increase by 300
- This is probably normal route churn
- Know how many bestpath changes you normally see per minute
• You have 150k routes and see the table version increase by 5k
- This is bad and is the cause of your high CPU
High CPU due to BGP
Route Churn

Router#Show ip bgp all sum | in tab

BGP table version is 936574954, main routing table version 936574954
BGP table version is 429591477, main routing table version 429591477
Router#

Over 1800 prefix flaps

Router#Show ip bgp all sum | in tab
BGP table version is 936576768, main routing table version 936575068
BGP table version is 429591526, main routing table version 429591526
Router#

Router#show ip route | in 00:00:0

B 10.1.0.0 [200/0] via 10.185.80.140, 00:00:00
B 10.2.0.0 [200/0] via 10.185.80.140, 00:00:00
B 10.3.0.0 [200/0] via 10.185.80.140, 00:00:00
B 10.4.0.0 [200/0] via 10.185.80.140, 00:00:00
B 10.5.0.0 [200/0] via 10.185.80.140, 00:00:00
. . . .
TAC Case Example - 3
High CPU due to BGP Router RR1

PE1
• Customer reported one of the router had PE2
AS65535
become inaccessible RR2

• Found CPU spiking to 100% due to BGP

Router process

PE1#show ip route | in 00:00:0

B 10.1.1.0 [200/0] via 10.1.12.2, 00:00:00

B 10.2.1.0 [200/0] via 10.1.12.2, 00:00:00

<snip>

B 10.100.10.0 [200/0] via 10.1.12.2, 00:00:00

TAC Case Example - 3
High CPU due to BGP Router
RR1
• To investigate further, we followed the path
PE1 PE2
through till router PE2 AS65535
RR2
• On PE2 we noticed several flapping links
• Captured syslog for last 2 days
<SNIP>

1367861: Sep 11 2012 17:33:14: %LINK-SW1_SP-3-UPDOWN: Interface GigabitEthernet1/4/36, changed state

to up

1367862: Sep 11 2012 17:33:14: %LINEPROTO-SW1_SP-5-UPDOWN: Line protocol on Interface

GigabitEthernet1/4/36, changed state to up

1367919: Sep 11 2012 17:34:43: %LINK-3-UPDOWN: Interface GigabitEthernet1/4/36, changed state to down
TAC Case Example - 3
Resolution

• Configured IP event dampening and BGP Dampening

• Work on stabilising the flapping links.
Scenario 4 – High Memory / Memory Leak
Problem Description
• High Memory consumption by BGP
• Caused due to:
- Memory Leak

• Symptoms
- Malloc Failures
- Slow performance
- Router reloads
High Memory / Memory Leak
Malloc Failures
Sep 20 22:43:01.831 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 65556 bytes
failed from 0x400E04EC, alignment 16
Pool: Processor Free: 8952 Cause: Not enough free memory
Alternate Pool: Reclaimed Free: 25520 Cause: Not enough free memory
-Process= "BGP Router", ipl= 0, pid= 156
-Traceback= 40348B24 403FB928 403FDFA0 403F7238 40F5AC5C 4026B690 406B794C 40682DB0
406833FC 40884688 40884DF4 40885BB4 40F9B160 40885C68 40843E68

• If Malloc error show the process as BGP – doesn’t mean BGP is the culprit
• This error log can be the consequence of insufficient memory or a memory leak
condition
• Run “show memory debug leak [chunk]” to identify a memory leak
High Memory / Memory Leak
Memory Leak in IOS XR
• Use IOS XR memory comparator tool to track any incremental memory leak
• Simple 3 step process
- Show memory compare start 4-5 min gap
- Show memory compare end
- Show memory compare report
RP/0/RP0/CPU0:XR_RTR# show memory compare report
Sun Apr 12 22:28:21.715 PDT
JID name mem before mem after difference mallocs
restart/exit/new
--- ---- ---------- --------- ---------- -------
1088 mibd_interface 232432236 232957764 525528 33753
1086 mibd_entity 1046476 1528332 481856 11
1044 bgp 1037562644 1037827636 264992 425
0 malloc_dump 0 22144 22144 355
<snip>
TAC Case Example - 4
Memory Leak
• Customer reported “show memory” loses about 3-10 Mb of Free memory per
day
• It was noticed that BGP Router process accumulated most of the holding
memory
• Same behavior was noticed even after the reload.
Allocated Freed Holding Process

458001556 61617656 240877764 BGP Router

459588300 61623540 241919840 BGP Router

The holding memory keeps on increasing
463207104 61699296 244022952 BGP Router

467475524 61706168 246534736 BGP Router

TAC Case Example - 4
Memory Leak
• Asked customer to run “show memory debug leak chunk”
- Customer denied
• Setup a lab to reproduce the problem

Configured 400 BGP

RR1
neighbors and 100,000
PE1 TGN routes
AS65535
RR2

• Still not able to reproduce the problem

TAC Case Example - 4
Memory Leak

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down

State/PfxRcd
80.81.192.33 4 1273 0 0 0 0 0 never Idle (Admin)
80.81.192.45 4 20646 0 0 0 0 0 never Idle (Admin)
80.81.192.117 4 3209 0 0 0 0 0 never Idle (Admin)
80.81.193.61 4 8220 0 0 0 0 0 never Idle (Admin)
80.81.193.217 4 3209 0 0 0 0 0 never Idle (Admin)
195.30.0.18 4 5539 0 0 0 0 0 never Idle (Admin)
. . . . .

• Resolution
- Removing the Idle / Admin down neighbors helped overcome the memory leak
Agenda
• BGP peering issues
• Session not coming up, Session flapping

• Platform issues due to BGP

• High CPU, Memory utilisation

• BGP slow-peer issue

• BGP for Enterprises
Multi-homing, Load-sharing

• BGP for Service Providers

MPLS VPN, BGP PIC
Scenario 5 – BGP Slow Peer
Problem Description
• Customer reports updates not getting across all PE routers

• Symptoms
• Slow convergence RR1
• Updates not replicated to all peers

PE1 PE3 PE2

I’m Slow
BGP Slow-peer
BGP Cache Size & OutQ
• Should be reaching the maximum cache size for that update-group
• OutQ column should show very high OutQ value
RR# show ip bgp vpnv4 all replication
Current Next
Index Members Leader MsgFmt MsgRepl Csize Version Version
1 348 12.123.67.97 1726595727 1938155978 999/1000 1012333000/1012351142
2 2 12.122.78.19 79434677 79398843 0/200 1012351503/1012351503
3 1 199.37.187.24 0 0 0/100 0/0
4 2 12.122.78.249 79219618 97412908 0/200 1012351504/1012351504

RR# show ip bgp vpnv4 all summary

..
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
12.123.67.97 4 109 42 87065 0 0 1000 00:10:00 0
12.122.78.19 4 109 42 87391 0 0 674 00:10:00 0
BGP Slow-peer
Other parameters to check
#show ip bgp neighbor 10.1.0.1
..
Last read 00:00:43, last write 00:00:17, hold time is 180, keepalive interval
is 60 seconds
Keepalives are temporarily in throttle due to closed TCP window
Neighbor capabilities:
..
BGP table version 250001, neighbor version 200001/250001
Output queue size : 410
..
Enqueued packets for retransmit: 15, input: 0 mis-ordered: 0 (0 bytes)
..
iss: 130287252 snduna: 131516888 sndnxt: 131532233 sndwnd: 0
..
Datagrams (max data segment is 1460 bytes):
Rcvd: 922 (out of order: 0), with data&colon; 65, total data bytes: 1261
Sent: 1463 (retransmit: 29 fastretransmit: 1),with data&colon; 1391, total
data bytes: 1245129
BGP Slow-peer
Static Slow peer
• The manual knob to flag a peer as slow will create a separate update group for
the peer.
• The advantage - there is a limit to the overhead that this feature will create.
• The drawback - slow member update group will have to progress at the pace of
the slowest of the slow peers.
neighbor {<nbr-addr>/<peer-grp-name>} slow-peer split-update-group static

• This command will manually mark a neighbor as slow peer.

• The peer will be part of slow update group.
BGP Slow-peer
Dynamic Slow peer
• IOS BGP will monitor the transmission speeds of the peers.
• A peer will have to be exhibiting slowness for several minutes to be flagged.
• Log message for when a slow peer is detected/recovered

bgp slow-peer detection [threshold <seconds>]

neighbor {<nbr-addr>/<peer-grp-name>} slow-peer detection [threshold < seconds >]

• The threshold defines “the threshold time in seconds” to detect a peer as slow
peer.
• The range is 120 seconds to 3600 seconds. Default is 300 seconds.
BGP Slow-peer
Slow peer protection
• Depends on Dynamic Slow Peer feature
• When a slow peer recovery is detected (the peer has converged), the peer will
be moved back to its original group
bgp slow-peer split-update-group dynamic [permanent]

neighbor {<nbr-addr>/<peer-grp-name>} slow-peer split-update-group dynamic [permanent]

• When “permanent” is not configured, the “slow peer” will be moved to its regular
original update group, after it becomes regular peer (converges).
• If “permanent” is configured, the peer will not be moved to its original update
group automatically
BGP Slow-peer
Syslog Messages
• The below log message will be generated when a peer is detected as dynamic
slow peer.
"bgp neighbor %s in af %d is detected as slow-peer"

• The below log message will be generated when a “slow-peer” recovers.

"slow bgp peer %s in af %d has recovered"
BGP Slow-peer
Show / Clear Commands
• Show Commands
show ip bgp [AF/scope/topo] update-group summary slow

show ip bgp [AF/scope/topo] summary slow

show ip bgp [AF/scope/topo] neighbor slow

• Clear Commands
Clear [ip] bgp <nbr-addr> slow

Clear [ip] bgp peer-group <group-name> slow

Clear [ip] bgp af * slow

Clear ip bgp * slow

TAC Case Example – 5 RR1

BGP Slow Peer

PE1 PE3 PE2

• Customer reported routes were stuck in BGP RR.

• Their end-customer moved service from one of their locations to another but the
old routes are still seen on their RR and other locations.

RR1# sh ip bgp vpnv4 all replication

Current Next
Index Members Leader MsgFmt MsgRepl Csize Version Version
1 150 216.156.3.10 274950548 650809652 2000/2000 421492656/421493582
2 5 65.106.7.100 41049479 204232170 0/500 421493582/0
5 1 66.239.189.212 16143960 16143960 0/100 421491282/421493582
TAC Case Example – 5
Resolution
• Two neighbors were identified to be showing slow peer symptoms
• Customer’s RR router did not have the slow peer capability in the IOS they were
running
• Two workarounds / solutions:

• Create a separate outbound policy for slow peers.

• You can use the "neighbor <ip> advertise-interval <interval>".

• Default for internal neighbors is 5 sec and for external is 30 seconds.
Agenda
• BGP peering issues
• Session not coming up, Session flapping

• Platform issues due to BGP

• High CPU, Memory utilisation

• BGP slow-peer issue

• BGP for Enterprises
• Multi-homing, Load-sharing

• BGP for Service Providers

• MPLS VPN, BGP PIC
Scenario 6 – Multi-Homing
Problem Description
• In a multi-homed network, load sharing is not working as expected
• Symptoms:
- Under or over utilised link / resource

B 90% D
AS200
AS65000 F
A 5% Provider
C E
BGP Multi-homing
Overview
• More than one link external to the local network
- Usually two external facing routers
• Two design scenarios :
- Multi-homing to the same ISP
- Multi-homing to different ISPs
• Each design categorised into two modes :
- Fail-over mode
- Load-sharing mode
Following examples assume that the multi-homed site has a /19 address block
 For your
Best Path Algorithm reference only

1 Weight Highest wins Scope is router only

2 LOCAL_PREFERENCE Highest wins Scope is AS only
3 Locally Originated Redistribution or network statement favored over aggregate-
address
4 AS_PATH Shortest wins Skipped if “bgp bestpath as-path ignore” configured
AS_SET counts as 1
CONFED parts do not count
5 ORIGIN Lowest wins IGP < EGP < Incomplete
6 MED Lowest wins MEDs are compared only if the first AS in the AS_SEQUENCE
is the same
7 eBGP over iBGP
8 Metric to Next Hop Lowest wins IGP cost to the BGP NEXTHOP
9 Multiple Paths in RIB Flag path as “multipath” is max-paths is configured
10 Oldest External Wins Unless BGP best path compare router-id configured
11 BGP Router ID Lowest
12 CLUSTER_LIST Smallest Shorter CLUSTER_LIST wins
13 Neighbor Address Lowest Lowest neighbor address
Multi-Homing with same ISP
Failover Mode
• Applies when end-site has bought a large primary WAN link to their upstream a
small secondary WAN link as the backup
• Primary link:
B D
- Outbound – announce /19 unaltered primary
AS200
AS100 F
- Inbound – receive default route A backup
C E

• Backup link:
- Outbound – announce /19 with increased metric
- Inbound – received default, and reduce local preference
Multi-Homing with same ISP
Load-Sharing Mode
• This example assumes equal capacity circuits - Unequal capacity circuits
requires more refinement
• Simple - Just receive the default route
- Assumes multi-path feature is enabled on internal routers of receiving AS
• Split /19 and announce as two /20s
- Advertise one with a lower MED and the other with a higher MED on each link

B D
Link-1
AS100 AS200
A F
Link-2
C E
Multi-Homing with two different ISPs
Failover Mode
• One link primary, the other link backup only

• Announce /19 aggregate on each link

- Primary link makes standard announcement
- Backup link lengthens the AS PATH by using AS PATH prepend

D
AS200
B
Internet
AS100
A

C
E
AS300
Multi-Homing with two different ISPs
Load-Sharing Mode
• Use both default from both ISPs for outbound
• For inbound :
Split /19 and announce as two /20s
- one is announced with lengthened AS by prepending and other is announced ‘as is’
on each link

D
AS200
B
Internet
AS100
A
C
E
AS300
TAC Case Example - 6
BGP Multi-Homing
• It was required to prefer router B as the primary router for all outgoing and
incoming traffic.
• In case of any failure, the traffic must automatically be routed to C.

D
AS200
AS100 B
Internet
A

C
E
AS300
TAC Case Example - 6
Resolution
• Making router B the primary HSRP gateway for A will solve the outbound
requirement
• Use of “set as-path prepend” will help solve this problem of inbound
requirement
Router C:
=========
access-list 100 permit <specific_traffic>
route-map foo permit 10
match ip address 1
set as-path prepend 100
!
router bgp 100
neighbor <AS_300_nei> route-map foo out
Agenda
• BGP peering issues
• Session not coming up, Session flapping

• Platform issues due to BGP

• High CPU, Memory utilisation

• BGP Slow-peer issue

• BGP for Enterprises
• Multi-homing, Load-sharing

• BGP for Service Providers

• MPLS VPN, PIC
Scenario 7a – BGP in MPLS VPN
Problem Description
• Customer reports reachability issues for a Customer VRF – Customer A
connected to PE1 is unable to reach its other site connected to PE2
• Caused due to:
• Wrong IGP/VPN Label propagation
• Symptoms PE1 RR1 PE2

• No reachability
• Packet loss

CE1 CE2
BGP in MPLS VPN
Troubleshooting steps
• Verify VRF configuration on both PE routers
- IOS - show run vrf A
• Verify local PE-CE Reachability (PE1 – CE1) & (PE2 – CE2)
• Verify PE to PE loopback reachability
PE1 RR1 PE2
- PE1# ping <PE2_loopback> source lo0
• Verify LSP path between PE routers MPLS VPN
- ping mpls ipv4 <dst> <subnet> CE1 CE2

- ping mpls traffic-eng tunnel <tunnel_num>

• Verify PE to PE reachability for VRF
- PE1# ping vrf A <vrf_ip_on_PE2>
BGP in MPLS VPN show mpls for 1.1.1.1
Labeling & Packet Flow IGP: POP IGP: 20
VPN:100 VPN:100 VPN:100

Verify VPN Label: PE1 RR1 PE2

Show ip bgp vpnv4 vrf A
100.1.1.1 Lo0=1.1.1.1/32 Lo0=3.3.3.3/32 Show ip bgp vpnv4 vrf A 100.1.1.1
should have 100 as the Out label
advertised by PE1
MPLS VPN
Data flow
CE1 CE2
Control Plane
Lo0=100.1.1.1/32 Lo0=200.1.1.1/32

PE1#sh ip bgp vpnv4 vrf A 100.1.1.1 PE2#sh ip bgp vpnv4 vrf A 100.1.1.1
BGP routing table entry for 1:1:100.1.1.1/32 BGP routing table entry for 1:1:100.1.1.1/32
192.168.10.2 from 0.0.0.0 (1.1.1.1) 1.1.1.1 (metric 21) from 2.2.2.2 (2.2.2.2)
<snip> <snip>
OSPF RT:0.0.0.0:2:0 OSPF ROUTER OSPF RT:0.0.0.0:2:0 OSPF ROUTER
ID:192.168.10.1:0 ID:192.168.10.1:0
mpls labels in/out 100/nolabel Originator: 1.1.1.1, Cluster list: 2.2.2.2
mpls labels in/out nolabel/100
Scenario 7b – RT Constraint Filtering in MPLS VPN
Problem Description
• Customer reported that their resource utilisation on edge routers have increased
with the growth of their SP network where as no extra services has been
deployed on those edge devices
• Symptoms
• Unwanted resource utilisation
PEn
• Performance issues
PE(n-1)
PE2 CE2

CE1 PE1 RR

AS100 PE3 CE3

RT Constraint Filtering in MPLS VPN
Overview
• PE routers use Route Target (RT) extended communities to control the
distribution of routes into VRFs.
• Ideally PE routers need to hold only routes marked with Route Targets
pertaining to VRFs that have local CE attachments.
• When the distribution of VPNs is sparse, there is wastage of resources in
maintaining the unwanted routes at the PE routers and unnecessary distribution
of routes by the route-reflectors.
• Route Target (RT)-constraint is a mechanism to prevent the propagation of VPN
NLRI to a PE that is not interested in the VPN.
BGP RT Constraint Filtering in MPLS VPN
Un-wanted Routes at RR & PE

VRF- Blue VRF- Green

VRF- Red VRF- Purple
PE-1 PE-3
Unwanted VRF routes for PE

RR-1 RR-2
VRF- Red VRF- Purple
VRF- Green VRF- Blue
PE-2 Unwanted VRF routes for RR PE-4

• PE-1 and PE-2 advertise VRF Blue, red and green VPN routes to RR-1
• RR-1 send ALL its VPN routes to RR-2.
• VRF-Red routes are really ‘unwanted’ on RR-2 since PE-3 and PE-4 does not have VRF
Red.
RT Constraint Filtering in MPLS VPN
Concept
• By having BGP speakers exchanging the ‘wanted’ Route Targets, this allows
BGP speaker to eliminate advertising ‘unwanted’ VPN routes to its peer.
• The ‘wanted’ Route Targets are called RT membership.
• MP-BGP UPDATE message to propagate RT membership information.
- Peers to advertise their RT membership.
- Restrict advertisement of VPN route based on received RT membership information.

• Perform Constraint Route Distribution on VPN v4 and v6 Route advertisements

only
RT Constraint Filtering in MPLS VPN
Enable RT Constraint on RR
RR-1#sh run | b router bgp 20.0.0.2 20.0.0.1
router bgp 65000
address-family vpnv4
PE-2 RR-1
neighbor 20.0.0.2 activate
neighbor 20.0.0.2 send-community both
neighbor 20.0.0.2 route-reflector-client
!
address-family rtfilter
neighbor 20.0.0.2 activate
neighbor 20.0.0.2 send-community both
neighbor 20.0.0.2 route-reflector-client

exit-address-family
RT Constraint Filtering in MPLS VPN
Enable RT Constraint on PE
PE-2#sh run | b router bgp 20.0.0.2 20.0.0.1
router bgp 65000
address-family vpnv4 PE-2 RR-1
neighbor 20.0.0.1 activate
neighbor 20.0.0.1 send-community both
exit-address-family
!
address-family rtfilter
neighbor 20.0.0.1 activate
neighbor 20.0.0.1 send-community both
exit-address-family
RT Constraint Capability Exchange
20.0.0.2 20.0.0.1

PE-2 RR-1

*Apr 29 01:18:34.658: BGP: 20.0.0.1 *Apr 29 01:16:24.896: BGP: 20.0.0.2

active OPEN has CAPABILITY code: 1, passive OPEN has CAPABILITY code: 1,
length 4 length 4
*Apr 29 01:18:34.658: BGP: 20.0.0.1 *Apr 29 01:16:24.896: BGP: 20.0.0.2
active OPEN has MP_EXT CAP for passive OPEN has MP_EXT CAP for
afi/safi: 1/132 afi/safi: 1/132
*Apr 29 01:18:34.658: BGP: 20.0.0.1 *Apr 29 01:16:24.897: BGP: 20.0.0.2
accept RTC SAFI accept RTC SAFI
BGP RT Constraint Filtering in MPLS VPN
Exchanging RTC between PE and RR
RT-Constraint: RT-Constraint:
NLRI = {RT 1, RT 2} NLRI = {RT 2, RT 3}

VRF- Blue RT 1 VRF- Red RT 2

VRF- Red RT 2 VRF- Green RT 3
PE-1 RR-1 PE-2
Outbound Filter: Outbound Filter:
Permit RT 1 Permit RT 3
Permit RT 2 Permit RT 2
Deny all Deny all

1. PE-1 send RTC NLRI {RT 1, RT 2} to RR-1

2. PE-2 sends RTC NLRI {RT 2, RT 3} to RR-1
3. RR-1 install an outbound Filter (Permit RT 1, RT 2) for PE-1
4. RR-1 installs an outbound Filter (Permit RT 2, RT 3) for PE-2
TAC Case Example - 7
MPLS VPN
• A Service Provider customer reported that a newly configured MPLS VPN
Services for their end customer was not working. CE2 was unable to see the
routes of CE1.
• Customer didn’t have the access to AS1PE2 router initially as it was in a distant
location and managed by a different group.

AS1RR1

AS100
AS1PE1 AS1PE2

CE1 CE2
100.1.1.0/24 200.1.1.0/24
TAC Case Example - 7
Troubleshooting Done
• Debugging done to see if AS1RR1 was sending the update or not
debug ip bgp vpnv4 unicast update <neighbor> <acl> in IOS
access-list 10 per 100.1.1.0 0.0.0.255

route-policy DEBUG_BGP
if destination in BGP_PREFIX then
pass
else
drop
endif IOS XR
end-policy

prefix-set BGP_PREFIX
100.1.1.0/24
end-set

debug bgp update vpnv4 unicast [in | out] route-policy DEBUG_BGP

TAC Case Example - 7
Resolution
• Further access was requested to troubleshoot AS1PE2 router

ip vrf Cust_A ip vrf Cust_A

rd 15956:3772958 rd 15832:3772958
route-target import 15956:3772958 route-target import 15832:3772958
route-target export 15956:3772968 route-target export 15956:3772958
route-target import 15956:3772958
route-target import 12956:3772953
route-target import 13956:3692952
route-target import 14956:3832957
route-target import 17956:3773959

• It was noticed that there was a missing entry on the vrf import statement
Scenario 8 – BGP PIC
Problem Description
• Customer reported they have a multi-homed Customer. But when the primary
BGP session goes down, it takes time to converge and the customer
experiences a traffic loss
• Symptoms
• Traffic loss

PE2

PE1
AS100
CE1 CE2

PE3
 What is PIC and BGP FRR?
• Prefix Independent Convergence (PIC) in CEF and platform whereby cutover to any
backup path happens within sub-seconds and independent of the number of prefixes.

• BGP Fast Re-Route (BGP FRR) – enables BGP to use alternate paths within sub-
seconds after a failure of the primary or active paths.

• Without backup paths available to CEF, convergence is driven from the routing protocols
updating the RIB and CEF one prefix at a time, leading to convergence times directly
proportional to the number of affected prefixes.

• When backup paths are available, CEF can use these to provide constant time and prefix
independent convergence, when a failure affecting a shared path-list occurs.
BGP PIC
PIC edge vs. PIC core

1 2
PE2
3

CE1 PE1 CE2

Customer Customer
Site A PE3 Site B

1. PIC core – when IGP path changes.

2. PIC edge – when remote PE node or it’s reach ability fails.
3. PIC edge – when PE-CE link fails.
BGP PIC
Configuration
• BGP PIC Core is enabled by default. If disabled, re-enable by:
R1(config)cef table output-chain build favor convergence-speed

• BGP PIC Edge

R1(config-bgp-vrf)bgp additional-path install

R1(config-bgp-vrf)bgp advertise-best-external
(on backup PE for PE-CE link protection)
BGP PIC
BGP Output
ASR-1K# sh ip bgp vpnv4 vrf site-111111 2.0.0.0
BGP routing table entry for 65300:111111:2.0.0.0/24, version 150035
Paths: (3 available, best #1, table sie-111111)
Additional-path-install
Advertised to update-groups:
105
Refresh Epoch 1
20570 20570
10.200.1.2 from 10.200.1.2 (10.200.1.2)
Origin incomplete, localpref 100, valid, external, best
Extended Community: RT:64300:111111 , recursive-via-connected
rx pathid: 0, tx pathid: 0x0
<snip>
Refresh Epoch 1
20570 20570
10.10.10.2 from 10.10.10.2 (5.5.5.5)
Origin incomplete, localpref 100, valid, internal, backup/repair
Extended Community: RT:64300:111111 , recursive-via-host
rx pathid: 0, tx pathid: 0
BGP PIC
CEF Output

ASR-1K# sh ip cef vrf site-111111 2.0.0.0 255.255.255.0 detail

2.0.0.0/24, epoch 0, flags rib only nolabel, rib defined all labels
recursive via 10.200.1.2
attached to GigabitEthernet0/1/1.200
recursive via 10.10.10.2, repair
attached to GigabitEthernet0/1/0.451
BGP PIC
Show Commands
• BGP
 sh ip bgp ipv4 unicast <x.x.x.x>
 sh ip bgp vpnv4 unicast vrf FOO <x.x.x.x> (To check if backup/repair is set on a prefix)
 sh ip bgp vpnv4 vrf FOO neighbor <neighbor_ip> (shows if PIC is enabled)

• RIB
 Show ip route vrf FOO <x.x.x.x>

• CEF
 show (ip | ipv6) cef [vrf XX] prefix/mask internal
 show monitor event cef (ipv4 | ipv6 | bfd) all
 show cef bfd
 debug cef bfd
 debug cef loadinfo map
 debug cef path
 debug cef filter fib (ipv4 | ipv6) prefix/mask
TAC Case Example - 8
BGP PIC
• Customer has implemented BGP PIC Edge feature and is also using BFD for
faster failover.
• When the best path fails, the customer does not see the fast convergence
happening

P2 PE4

CE1 PE1 CE2

P3 PE5
TAC Case Example - 8
CEF Output
ASR-1K#sh ip cef vrf site-111111 2.0.0.0 internal
2.0.0.0/30, epoch 0, flags rib only nolabel, rib defined all labels, RIB[B], refcount 6, per-
destination sharing
sources: RIB
<snip>
GigabitEthernet0/1/1.350(21): 10.200.1.2
path 7FA7EB87AB00, path list 7FA7EB611F90, share 1/1, type recursive, for IPv4, flags recursive-
via-connected
recursive via 10.200.1.2[IPv4:sie-111111], fib 7FA7ECBE2558, 1 terminal fib, v4:sie-
111111:10.200.1.2/32
path 7FA7EB87A630, path list 7FA7EB612C10, share 1/1, type adjacency prefix, for IPv4
attached to GigabitEthernet0/1/1.350, adjacency IP adj out of GigabitEthernet0/1/1.350, addr
10.200.1.2 7FA7EB643568
path 7FA7EB879DF0, path list 7FA7EB611F90, share 1/1, type recursive, for IPv4, flags repair,
recursive-via-host, unuseable
recursive via 10.10.10.2[IPv4:sie-111111], repair, fib 7FA7E2E01068, 1 terminal fib, v4:sie-
111111:10.10.10.2/32
<snip>
TAC Case Example - 8
Resolution

router bgp 65300

!
address-family ipv4 vrf site-111111
bgp additional-paths install
bgp recursion host Removing this configuration, resolved the
network 3.3.3.0 mask 255.255.255.0 issue. BGP Recursion host is useful when
<snip>
neighbor 10.200.1.2 remote-as 20570
performing node protection
neighbor 10.200.1.2 version 4
neighbor 10.200.1.2 fall-over bfd
neighbor 10.200.1.2 activate
neighbor 10.200.1.2 send-community
neighbor 10.200.1.2 next-hop-self
neighbor 10.200.1.2 soft inbound
exit-address-family
The Two Napkin Protocol
Q&A
Complete Your Online Session Evaluation
Give us your feedback and receive a
Cisco 2016 T-Shirt by completing the
Overall Event Survey and 5 Session
Evaluations.
– Directly from your mobile device on the Cisco Live
Mobile App
– By visiting the Cisco Live Mobile Site
http://showcase.genie-connect.com/ciscolivemelbourne2016/
– Visit any Cisco Live Internet Station located
throughout the venue
Learn online with Cisco Live!
T-Shirts can be collected Friday 11 March Visit us online after the conference
for full access to session videos and
at Registration presentations.
www.CiscoLiveAPAC.com
Thank you