Networkers 2001, Australia

March 28-30, Brisbane

2000, Cisco Systems, Inc.

Troubleshooting OSPF
Jono Ladmore

Presentation_ID

2000, Cisco Systems, Inc. 2001, Cisco Systems, Inc.

Agenda

LSA Details Troubleshooting Commands Common Issues Troubleshooting Tips

2000, Cisco Systems, Inc.

LSA Details

2000, Cisco Systems, Inc.

2206 1209_05_2000_c3

2000, Cisco Systems, Inc.

LSA Type Review

Type
1 2 3 4 5 7

LSA
Router Network Summary Network Summary ASBR External NSSA Opaque
5

9,10,11
2000, Cisco Systems, Inc.

Common LSA Header

Common LSA 20 byte header Options LS Type Link State ID Advertising Router LS Sequence Number LS Checksum Length LS Age

2000, Cisco Systems, Inc.

Router LSA Details

Router LSA (Type 1)
Describes the state and cost of the routers links to the area All of the routers links in an area must be described in a single LSA Flooded throughout the particular area and no more Router indicates whether it is an ASBR, ABR, or end point of virtual link
2000, Cisco Systems, Inc.

Router LSA of R3 for Area 1

show ip ospf database router 192.1.1.3
LS age = 0 Options = (E -bit) (ELS type = 1 Link State ID = 192.1.1.3 Advertising Router = 192.1.1.3 bit E = 0 bit B = 1 # links = 2 Link ID = 192.1.1.4 Link Data = 192.1.1.3 Type = 2 # TOS metrics = 0 metric = 1 Link ID = 192.1.4.0 Link Data = 255.255.255.0 Type = 3 # TOS metrics = 0 metric = 2
2000, Cisco Systems, Inc.

Always 0 at origination This is a router LSA Router ID of R3 Router ID of R3 Not an ASBR This is an ABR IP address of the DR Interface address of this router This is a transit network DR Cost to reach the interface IP network number Subnet mask of the interface Stub network
192.1.1.4 1 192.1.1.3

R4

Area 0
R3
192.1.4.0/24 2
8

Route LSA Details

Type Description Link ID Link Data
1 1 2 3 4
Point-to-Point numbered Point-to-Point Unnumbered Transit Stub Virtual Link Neighbors RID Neighbors RID IP Address of the DR IP Network Number Neighbors RID Interface IP Address MIB-II Ifindex Value Interface IP Address Subnet Mask Interface IP Address

2000, Cisco Systems, Inc.

Router LSA of R3 for Area 0

show ip ospf database router 192.1.1.3
LS age = 0 Options = (E -bit) (ELS type = 1 Link State ID = 192.1.1.3 Advertising Router = 192.1.1.3 bit E = 0 bit B = 1 # links = 1 Link ID = 195.12.1.1 Link Data = 18.10.0.7 Type = 1 # TOS metrics = 0 metric = 8 Link ID = 18.10.0.0 Link Data = 255.255.255.0 Type = 3 # TOS metrics = 0 metric = 8
2000, Cisco Systems, Inc.

Router id of the neighbor IP interface address of the router This is a point-to-point link DR IP subnet address Subnet mask This is a stub link
192.1.1.4 1 192.1.1.3

R4

Area 0
R3
192.1.4.0/24 2
10

8 18.10.0.7/24 195.12.1.1

Network LSA
Network LSA (Type 2)
Generated for every transit broadcast and NBMA network Describes all the routers attached to the network Only the designated router originates this LSA Flooded throughout the area and no more
2000, Cisco Systems, Inc.

11

Network LSA for 192.1.1.0

show ip ospf database network 192.1.1.4
LS age = 0 Options = (E -bit) (ELS type = 2 Link State ID = 192.1.1.4 IP interface address of DR Advertising Router = 192.1.1.4 RID of DR Network Mask = 255.255.255.0 Attached Router = 192.1.1.4 Attached Router = 192.1.1.3 RID of attached routers FULL with the DR Attached Router = 192.1.1.2 DR Attached Router = 192.1.1.1

R1 R2

192.1.1.4 192.1.1.1 /24 1 192.1.1.3 192.1.1.2 2

R4

Area 0
R3
8 18.10.0.7 195.12.1.1

192.1.4.0/24

2000, Cisco Systems, Inc.

12

Summary LSA
Describes the destination outside the area but still in the AS Flooded throughout a single area Originated by an ABR Only intra-area routes are advertised into the backbone (to avoid loops) Type 4 is the information about the ASBR
2000, Cisco Systems, Inc.

13

Type 3 and 4

Summary LSA
In Stub area: O IA* 0.0.0.0/0
Link State ID: 0.0.0.0 Network Mask: 0.0.0.0

Network mask field is always 0.0.0.0 for summary LSA Type 4

2000, Cisco Systems, Inc.

14

Type 3 Details
show ip ospf database summary 192.1.2.0
LS age = 0 Options = (E -bit) (ELS type = 3 Link State ID = 192.1.2.0 IP network number Advertising Router = 192.1.1.4 RID of ABR Network Mask = 255.255.255.0 metric = 4

O IA
ABR
DR
192.1.1.4 192.1.2.0/24 metric 4 8

192.1.2.0/24 3

R4

R1 R2

192.1.1.1 /24 1 192.1.1.3 192.1.1.2 2

Area 0
R3
8 18.10.0.7 195.12.1.1

192.1.4.0/24

2000, Cisco Systems, Inc.

15

Type 4 Details
show ip ospf database summary-asbr 192.1.2.0
LS age = 0 Options = (E -bit) (ELS type = 4 Link State ID = 131.108.1.1 RID of ASBR Advertising Router = 192.1.1.4 RID of ABR Network Mask = 0.0.0.0 metric = 16

ABR
192.1.2.0/24 3

Type 4 summary
8 8

External Route
140.10.0.0

DR

192.1.1.4 R1 192.1.1.1 /24 1 192.1.1.3

R4 R3 R3
2

Area 0
8 18.10.0.7 195.12.1.1

R2

RID ASBR 131.108.1.1 R7

192.1.1.2

192.1.4.0/24

2000, Cisco Systems, Inc.

16

External LSA
External LSA (Type 5)
Defines routes to destination external to the AS Default route is also sent as external Two types of external LSA: E1: Consider the total cost up to the external destination E2: Considers only the cost of the outgoing interface to the external destination
2000, Cisco Systems, Inc.

17

External LSA
External Type 1
To N1 External Cost = 1 R3 Cost = 10 R1 Cost = 8 R2 Network N1 N1
2000, Cisco Systems, Inc.

To N1 External Cost = 2

Type 1 11 10

Next Hop R1 R2

Selected Route
18

External LSA
External Type 2
To N1 External Cost = 1 R3 Cost = 10 R1 Cost = 8 R2 Network N1 N1
2000, Cisco Systems, Inc.

To N1 External Cost = 2

Type 1 1 2

Next Hop R1 R2

Selected Route

19

Type 5 Details
show ip ospf database external 140.10.0.0
LS age = 0 Options = (E-bit) (ELS type = 5 Link State ID = 140.10.0.0 Advertising Router = 131.108.1.1 Network Mask = 255.255.0.0 bit E = 1 metric = 4 Forwarding address = 0.0.0.0

IP network number Router ID of R7 E = 1 -> O E2 (Default) Traffic should be forwarded to the ASBR

External Route
192.1.2.0/24 3 192.1.1.4 R4 R1 192.1.1.1 /24 1 192.1.1.3

DR

140.10.0.0
8 8

Area 0
195.12.1.1

R2
2000, Cisco Systems, Inc.

192.1.1.2

R3 R7

RID ASBR 131.108.1.1

18.10.0.7 192.1.4.0/24

External Type 5
20

Type 5 Details
Forwarding address
Must be known via intra or inter area route
router ospf 1 redistribute rip subnets

RIP Between the Routers

ASBR
Rest of the OSPF Network

OSPF Is Enabled on this Interface

2000, Cisco Systems, Inc.

21

Type 7 Details
show ip ospf database nssa-external 140.10.0.0
LS age = 0 Options = (No TOS-capability, No Type 7/5 translation, DC) TOSLS type = 7 Link State ID = 140.10.0.0 IP network number Advertising Router = 131.108.1.1 Router ID of R7(NSSA ASBR) Network Mask = 255.255.0.0 bit P = 0 P = 0 -> This router is an NSSA ABR+ASBR P = 1 -> This router is an NSSA ASBR metric = 4 Forwarding address = 0.0.0.0 Traffic should be forwarded to the ASBR
DR

External Route
8 8

192.1.2.0/24 3

192.1.1.4 R4 R1 192.1.1.1 /24 1 192.1.1.3

140.10.0.0
4

Area 0
R7
195.12.1.1

R2
2000, Cisco Systems, Inc.

192.1.1.2

R3

RID ASBR 131.108.1.1

18.10.0.7 192.1.4.0/24

Type 7 NSSA
22

Troubleshooting Commands

2000, Cisco Systems, Inc.

2206 1209_05_2000_c3

2000, Cisco Systems, Inc.

23

Show IP OSPF
GSRGSR-3#sh ip ospf
Routing Process "ospf 100" with ID 10.10.128.3 Supports only single TOS(TOS0) routes Supports opaque LSA It is an area border router SPF schedule delay 5 secs, Hold time between two SPFs 10 secs Minimum LSA interval 5 secs. Minimum LSA arrival 1 secs Number of external LSA 0. Checksum Sum 0x0 Number of opaque AS LSA 0. Checksum Sum 0x0 Number of DCbitless external and opaque AS LSA 0 Number of DoNotAge external and opaque AS LSA 0 Number of areas in this router is 2. 2 normal 0 stub 0 nssa External flood list length 0 Area BACKBONE(0) Number of interfaces in this area is 5 Area has no authentication SPF algorithm executed 2773 times Area ranges are Number of LSA 97. Checksum Sum 0x2B19E3 Number of opaque link LSA 0. Checksum Sum 0x0 Number of DCbitless LSA 0 Number of indication LSA 0 Number of DoNotAge LSA 0 Flood list length 0
2000, Cisco Systems, Inc.

24

Show IP OSPF (Cont.)

Area 8 Number of interfaces in this area is 2 It is a stub area, no summary LSA in this area Area has no authentication SPF algorithm executed 11 times Area ranges are Number of LSA 5. Checksum Sum 0x22812 Number of DCbitless LSA 0 Number of indication LSA 0 Number of DoNotAge LSA 0 Flood list length 0

2000, Cisco Systems, Inc.

25

OSPF Database

36003600-g1#sh ip ospf data

OSPF Router with ID (30.8.1.1) (Process ID 1) Router Link States (Area 0) Link ID 30.8.1.1 ADV Router 30.8.1.1 Age 1592 Seq# Checksum Link count 0x800001D0 0xA180 0

Router Link States (Area 8) Link ID ADV Router Age Seq# Checksum Link count 30.8.1.1 30.8.1.1 1592 0x8000023F 0xC782 1 30.8.1.2 30.8.1.2 298 0x800003D1 0x2967 2 30.8.3.2 30.8.3.2 666 0x800002B8 0xE52B 1 Net Link States (Area 8) Link ID ADV Router Age 30.8.1.2 30.8.1.2 299 30.100.1.2 30.8.3.2 666
2000, Cisco Systems, Inc.

Seq# Checksum 0x80000203 0x4153 0x8000027A 0x10AB

26

OSPF Database

GSRGSR-3#show ip ospf database database-summary databaseOSPF Router with ID (10.10.128.3) (Process ID 100) Area 0 database summary LSA Type Count Router 63 Network 4 Summary Net 30 Summary ASBR 0 Type-7 Ext 0 Opaque Link 0 Opaque Area 0 Subtotal 97

Delete 0 0 0 0 0 0 0 0

Maxage 0 0 0 0 0 0 0 0

2000, Cisco Systems, Inc.

27

OSPF Neighbor
GSRGSR-3#sh ip ospf neighbor Neighbor ID 10.64.1.2 10.10.128.1 10.10.128.2 10.10.128.4 GSR-3# Pri State Dead Time Address 1 FULL/DR 00:00:38 10.128.16.2 1 2WAY/DROTHER 00:00:31 10.128.1.1 1 FULL/BDR 00:00:36 10.128.1.2 1 FULL/DR 00:00:38 10.128.1.4 Interface GigabitEthernet3/0 Ethernet 6/0 Ethernet 6/0 Ethernet 6/0

2000, Cisco Systems, Inc.

28 22

OSPF Adjacency Changes

r4-1# router ospf 1 ospf log-adjacency-changes log-adjacency%OSPF-5-ADJCHG: Process 1, Nbr 172.16.16.2 on Serial0.1 from FULL to DOWN, Neighbor Down %OSPF-5-ADJCHG: Process 1, Nbr 0.0.0.0 on Serial0.1 from DOWN to ATTEMPT, NBMA Start

2000, Cisco Systems, Inc.

29 22

Neighbor Details
r4r4-1#sh ip ospf nei det
Neighbor 172.16.133.5, interface address 172.16.133.5 In the area 0 via interface Ethernet0 Neighbor priority is 1, State is FULL, 6 state changes DR is 172.16.133.29 BDR is 172.16.133.6 Options is 0x2 Dead timer due in 00:00:37 Neighbor is up for 00:01:24 Index 2/2, retransmission queue length 0, number of retransmission 1 First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0) Last retransmission scan length is 1, maximum is 1 Last retransmission scan time is 4 msec, maximum is 4 msec Neighbor 172.16.133.29, interface address 172.16.133.29 In the area 0 via interface Ethernet0 Neighbor priority is 1, State is FULL, 6 state changes DR is 172.16.133.29 BDR is 172.16.133.6 Options is 0x2 Dead timer due in 00:00:31 Neighbor is up for 00:01:58 Index 1/1, retransmission queue length 0, number of retransmission 2 First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0) Last retransmission scan length is 1, maximum is 1 Last retransmission scan time is 0 msec, maximum is 0 msec

r4-1#
2000, Cisco Systems, Inc.

30

Show IP OSPF Interface

r4r4-1#show ip ospf int
Serial0.1 is up, line protocol is up Internet Address 172.16.7.1/24, Area 0 Process ID 1, Router ID 172.16.14.1, Network Type NON_BROADCAST, Cost: 64 Transmit Delay is 1 sec, State DR, Priority 100 Designated Router (ID) 172.16.14.1, Interface address 172.16.7.1 No backup designated router on this network Timer intervals configured, Hello 30, Dead 120, Wait 120, Retransmit 5 Hello due in 00:00:12 Neighbor Count is 2, Adjacent neighbor count is 2 Adjacent with neighbor 172.16.30.1 Adjacent with neighbor 172.16.16.2 Suppress hello for 0 neighbor(s) Serial0.2 is up, line protocol is up Internet Address 172.16.14.1/24, Area 33 Process ID 1, Router ID 172.16.14.1, Network Type NON_BROADCAST, Cost: 64 Transmit Delay is 1 sec, State DR, Priority 100 Designated Router (ID) 172.16.14.1, Interface address 172.16.14.1 No backup designated router on this network Timer intervals configured, Hello 30, Dead 120, Wait 120, Retransmit 5 Hello due in 00:00:11 Neighbor Count is 1, Adjacent neighbor count is 1 Adjacent with neighbor 172.16.100.2 Suppress hello for 0 neighbor(s)
2000, Cisco Systems, Inc.

31

Virtual Links

r4r4-1#sh ip ospf virtual- links virtual-

Virtual Link OSPF_VL0 to router 172.16.100.2 is up Run as demand circuit DoNotAge LSA allowed. Transit area 33, via interface Serial0.2, Cost of using 64 Transmit Delay is 1 sec, State POINT_TO_POINT, Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 Hello due in 00:00:09 Adjacency State FULL (Hello suppressed) r4-1#

2000, Cisco Systems, Inc.

32 22

Other show Commands

R6R6-2500#sh ip ospf database self -originate (shows the LSAs this router is generating) selfOSPF Router with ID (192.168.6.1) (Process ID 100) Router Link States (Area 0) Link ID 192.168.6.1 ADV Router 192.168.6.1 Age 773 Seq# Checksum Link count 0x8000005B 0xC608 1

Net Link States (Area 0) Link ID 135.4.1.2 ADV Router 192.168.6.1 Age 773 Seq# 0x80000056 Checksum 0x4D14

Router Link States (Area 5) Link ID 192.168.6.1 ADV Router 192.168.6.1 Age 264 Seq# 0x80000053 Checksum Link count 0xC745 0

Summary Net Link States (Area 5) Link ID 135.4.1.0 ADV Router 192.168.6.1 Age 774 Seq# 0x8000005C Checksum 0xDDFC

Summary ASB Link States (Area 5) Link ID 5.5.5.5 ADV Router 192.168.6.1 Age 774 Seq# 0x8000002B Checksum 0x681B

Type-5 AS External Link States Link ID 1.1.1.0

2000, Cisco Systems, Inc.

ADV Router 192.168.6.1

Age 266

Seq# 0x8000002C

Checksum Tag 0x31D0 0

33

Other show Commands

R6R6-2500#sh ip ospf database adv -router 5.5.5.5

(shows the LSAs neighbor is generating) OSPF Router with ID (192.168.6.1) (Process ID 100) Router Link States (Area 0) Link ID 5.5.5.5 Link ID 1.1.1.0 5.5.5.0 135.4.1.0 R6-2500#
2000, Cisco Systems, Inc.

ADV Router 5.5.5.5 ADV Router 5.5.5.5 5.5.5.5 5.5.5.5

Age 1479 Age 1478 1478 1478

Seq#

Checksum Link count 1

0x80000034 0xA52A Seq#

Type-5 AS External Link States Checksum Tag 0 0 0 0x80000032 0xACCB 0x80000030 0x204E 0x80000032 0xB33B

34

Show IP OSPF stat

r4r4-1#sh ip ospf stat Area 0: SPF algorithm executed 16 times Area 33: SPF algorithm executed 8 times SPF calculation time Delta T Intra D-Intra Summ 3d18h 4 0 4 3d18h 8 0 4 3d18h 4 0 4 3d18h 4 0 4 3d18h 4 0 4 3d18h 4 0 4 3d17h 4 0 4 00:04:45 4 0 4 00:02:51 4 0 4 00:02:41 4 0 8

D-Summ 4 0 0 0 0 0 0 8 0 8

Ext 8 8 8 8 8 8 8 8 8 8

D-Ext Total Reason 0 20 R, SN, 0 20 R, N, SN, 0 16 N, 0 20 N, SN, 0 16 R, N, 0 16 R, SN, 0 28 R, N, SN, SA, X 0 24 N, SN, 0 20 R, N, 0 28 R, SN,

2000, Cisco Systems, Inc.

35 22

Common Issues

2000, Cisco Systems, Inc.

2206 1209_05_2000_c3

2000, Cisco Systems, Inc.

36

Troubleshooting
Most common issues
Adjacency is not coming up OSPF neighbor stuck in ? state Information is in database not in routing table CPU Hogs, SPF running constantly Seq # mismatch Neighbor flapping (Frame-relay) NSSA ABR not translating DC is constantly bringing up the link
2000, Cisco Systems, Inc.

37

Troubleshooting

Adjacencies not coming up

Command to look for the neighbor adj show ip ospf neighbor show ip ospf interface debug ip ospf adjacency

2000, Cisco Systems, Inc.

38

Troubleshooting
Layer 2 is down
r4-4k#show ip ospf neighbor r4-4k# no neighbor r4-4k#show ip ospf interface s2 Serial2 is down, line protocol is down Internet Address 10.10.1.4/16, Area 0 Process ID 1, Router ID 10.34.1.1, Network Type POINT_TO_POINT, Cost: 64 Transmit Delay is 1 sec, State DOWN, Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5

2000, Cisco Systems, Inc.

39

Troubleshooting
OSPF not enabled on the interface
r4-4k#show ip ospf neighbor r4-4k# no neighbor r4-4k#show ip ospf interface s0 Serial0 is up, line protocol is up OSPF not enabled on this interface

In 12.0:
r4-4k#show ip ospf interface s0 r4-4k#

Tip: check for the wrong network statement re-enter the network statement
2000, Cisco Systems, Inc.

40

Troubleshooting
Interface is defines as passive
r4-4k#show ip ospf neighbor r4-4k# no neighbor r4-4k#show ip ospf interface e0 Ethernet0 is up, line protocol is up Internet Address 172.16.133.6/26, Area 0 Process ID 1, Router ID 1.1.1.1, Network Type BROADCAST, Cost: 10 Transmit Delay is 1 sec, State DR, Priority 1 Designated Router (ID) 1.1.1.1, Interface address 172.16.133.6 No backup designated router on this network Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5 No Hellos (Passive interface)

2000, Cisco Systems, Inc.

41

Troubleshooting
Mismatched subnet mask
r4-4k#debug ip ospf adj OSPF adjacency events debugging is on r4-4k# OSPF: Mismatched hello parameters from 172.16.133.6 Dead R 40 C 40, Hello R 10 C 10 Mask R 255.255.255.0 C 255.255.255.192

2000, Cisco Systems, Inc.

42

Troubleshooting
Mismatched hello/dead interval
r4-4k#debug ip ospf adj OSPF adjacency events debugging is on r4-4k# OSPF: Mismatched hello parameters from 172.16.133.6 Dead R 40 C 40, Hello R 15 C 10 Mask R 255.255.255.192 C 255.255.255.192

2000, Cisco Systems, Inc.

43

Troubleshooting
Mismatched Authentication key
r4-4k#debug ip ospf adj OSPF adjacency events debugging is on r4-4k# OSPF: Rcv pkt from 172.16.133.6, Ethernet0 : Mismatch Authentication Key - Clear Text Tip: Watch for the space at the end of the Authentication key

2000, Cisco Systems, Inc.

44

Troubleshooting
Mismatched area ID
r9-2500#show ip ospf neighbor r9-2500# no neighbors r9-2500#debug ip ospf adj OSPF adjacency events debugging is on r9-2500# OSPF: Rcv pkt from 172.16.133.6, Ethernet0, area 0.0.0.1 mismatch area 0.0.0.2 in the header Neighbor is in area 0 but we are not:
%OSPF-4-ERRRCV: Received invalid packet: mismatch area ID, from backbone area must be virtual-link but not found from 172.16.133.29, Ethernet0

2000, Cisco Systems, Inc.

45

Troubleshooting
Mismatched Transit/Stub area
r9-2500#show ip ospf neighbor r9-2500# no neighbors r9-2500#debug ip ospf adj OSPF adjacency events debugging is on r9-2500# OSPF: Hello from 172.16.133.6 with mismatched Stub/Transit area option bit

2000, Cisco Systems, Inc.

46

Troubleshooting

OSPF neighbor stuck in ? state

Command to look for the neighbor adj show ip ospf neighbor debug ip ospf adjacency

2000, Cisco Systems, Inc.

47

Bringing Up Adjacencies
Neighbor states
Down
No information has been received from the neighbor

Attempt
Valid for neighbors on NBMA network, no information is received from the neighbor but serious effort is made to contact the neighbor

Init
Hello packet has been received form the neighbor, but the router itself is not listed in that hello packet
2000, Cisco Systems, Inc.

48

Bringing Up Adjacencies
Neighbor States (Cont.)
2Way
Bi-directional communication with the neighbor is established; this is the beginning of adjacency; DR/BDR are elected in this state

Exstart
This is used in deciding the DBD sync, which router will be master/slave and what will be the first sequence number for DBD packet
2000, Cisco Systems, Inc.

49

Bringing Up Adjacencies
Neighbor states (Cont.)
Exchange
In this state the router describes the entire link state database through the DBD packet, each DBD sequence is explicitly acked, only one DBD packet is allowed outstanding at one time, link state request packets are also sent to request the newer LSA

Loading
In this state, link-state request packets are requested for the more recent LSA that have not been received during Exchange

Full
In this state complete information has been exchanged
2000, Cisco Systems, Inc.

50

Troubleshooting
Problem:
OSPF stuck in ATTEMPT

Reasons:
We are sending hellos to contact neighbor on NBMA but received no reply Neighbor hellos are getting lost in NBMA cloud Neighbor received our hello but rejecting that for some reason (Layer 2) Neighbor is not pingable
2000, Cisco Systems, Inc.

51

Troubleshooting
Problem:
OSPF stuck in INIT (one way hello)

Reasons:
1. One side is blocking the hello packet with 2. One side is translating (NAT) ospf hello 3. One side multicast capabilities is broken 4. Must be a Layer 2 problem 5. Dialer map or frame-relay map is missing broadcast http://www.cisco.com/warp/customer/104/7.html
2000, Cisco Systems, Inc.

52

Troubleshooting

Problem:
OSPF stuck in 2-WAY

Reasons:
This is normal on broadcast network types This is to reduce the amount of flooding on the wire Problems can happen if Layer 2 is broken http://www.cisco.com/warp/customer/104/11.html

2000, Cisco Systems, Inc.

53

Troubleshooting
Problem:
OSPF stuck in EXSTART/EXCHANGE

Reasons:
1. If the neighbor is bay router adjust the interface mtu to match bays. 3. Neighbor RID is same as ours 4. Unicast is broken
a. wrong VC/DLCi mapping in frame/ATM environment in highly redundant network b. mtu problem, cant ping across with more than certain length packet c. access-list blocking unicast. After 2-way OSPF send unicast packet except p2p links d. NAT is translating unicast packet http://www.cisco.com/warp/customer/104/12.html

5. Between PRI and BRI/dialer and network type is p2p

http://www.cisco.com/warp/customer/104/20.html
2000, Cisco Systems, Inc.

54

Troubleshooting

Problem:
OSPF stuck in LOADING

Reasons:
1. LS request is being made and neighbor is sending bad packet or mem corrupt a. do show ip ospf request-list <neighbor RID> <interface> to see bad lsa b. show log will show OSPF-4-BADLSATYPE msg 2. LS request is being made and neighbor is ignoring the request 3. MTU mismatch problem. Old IOS wont detect it (RFC 1583)

2000, Cisco Systems, Inc.

55

Troubleshooting

Information is in the database but not in the routing table

Command to look for show ip ospf database <x> x can be router, network, summary, summary-asbr, external, nssa

2000, Cisco Systems, Inc.

56

Troubleshooting
Mismatched Network Types
r9-2500#show ip ospf interface s 0.2 Serial0.2 is up, line protocol is up Internet Address 10.1.2.3/24, Area 0 Process ID 1, Router ID 10.1.2.3, Network Type BROADCAST, Cost: 64 r4-4k#show ip ospf interface s 0.1 Serial0.1 is up, line protocol is up Internet Address 10.1.2.1/24, Area 0.0.0.0 Process ID 1, Router ID 10.1.2.1, Network Type POINT_TO_POINT, Cost: 64

TIP: Adv-router not reachable msg

http://www.cisco.com/warp/customer/104/26.html
2000, Cisco Systems, Inc.

57

Troubleshooting
Problem:
OSPF routes in the database but not in the routing table

Reasons:
One side is numbered and the other unnumbered (O, O IA, O E1, O E2) ip addresses are flipped, dual serial (O, O IA, O E1, O E2) Forwarding address is not known or is known via external/static (O E1, O E2) - route sum and redistribute conn? Different mask or ip address in p2p (O, O IA, O E1, O E2) Distribute-list in is configured Backbone area became discontigous (O, OIA, OE1, OE2) OSPF is enabled on secondary but not on primary
2000, Cisco Systems, Inc.

58

Troubleshooting
SPF is running constantly
Look at sh ip ospf stat, Seq #, LS Age Debug ip ospf monitor show ip ospf database database-sum

For CPU hogs:

summarize if too many inter area or external routes 12.0 LSA group pacing
2000, Cisco Systems, Inc.

59

Show IP OSPF stat

r4r4-1#sh ip ospf stat Area 0: SPF algorithm executed 16 times Area 33: SPF algorithm executed 8 times SPF calculation time Delta T Intra D-Intra Summ 3d18h 4 0 4 3d18h 8 0 4 3d18h 4 0 4 3d18h 4 0 4 3d18h 4 0 4 3d18h 4 0 4 3d17h 4 0 4 00:04:45 4 0 4 00:02:51 4 0 4 00:02:41 4 0 8

D-Summ 4 0 0 0 0 0 0 8 0 8

Ext 8 8 8 8 8 8 8 8 8 8

D-Ext Total Reason 0 20 R, SN, 0 20 R, N, SN, 0 16 N, 0 20 N, SN, 0 16 R, N, 0 16 R, SN, 0 28 R, N, SN, SA, X 0 24 N, SN, 0 20 R, N, 0 28 R, SN,

2000, Cisco Systems, Inc.

60 22

Debug ip ospf monitor

r4r4-1#deb ip ospf mon OSPF: Schedule SPF in area 0 Change in LS ID 1.1.1.1, LSA type R, OSPF: schedule SPF: spf_time 0ms wait_interval 861421816s OSPF: Begin SPF at 0x33585480ms, process time 752ms spf_time 0ms, wait_interval 861421816s OSPF: End SPF at 0x33585488ms, Total elapsed time 8ms Intra: 4ms, Inter: 0ms, External: 0ms

2000, Cisco Systems, Inc.

61 22

LSA Group Pacing

All LSA refresh every 30 min (bad!) Timers of each LSA get sync (worse!) With group pacing only LSAs that reach max-age get refreshed periodically interval is configurable

2000, Cisco Systems, Inc.

62

Troubleshooting

Seq # mismatch
Command to look for debug ip ospf adjacency

2000, Cisco Systems, Inc.

63

Troubleshooting
Seq # mismatch:
1. LSA should one of 5 LSAs 2. If LSA is type 5 and the neighbor is associated with a stub area 3. If one of the options change 4. If the state of MS bit is inconsistent with master slave connection
2000, Cisco Systems, Inc.

64

Troubleshooting

Seq # mismatch
5. If the I-bit is set 6. If the master receives a DBD packet after a dead interval 7. If the requested LSA is not found, then something has gone wrong with the database exchange
2000, Cisco Systems, Inc.

65

Troubleshooting
Neighbor flapping
Command to look for debug ip ospf adjacency ospf log-adjacency-change show ip ospf neighbors det show interface

2000, Cisco Systems, Inc.

66

Troubleshooting
If the media is Frame Relay look for broadcast queue drops:
Serial0 is up, line protocol is up Hardware is MK5025 Description: Charlotte Frame Relay Port DLCI 100 MTU 1500 bytes, BW 1024 Kbit, DLY 20000 usec, rely 255/255, load 44/255 Encapsulation FRAME-RELAY, loopback not set, keepalive set (10 sec) LMI enq sent 7940, LMI stat recvd 7937, LMI upd recvd 0, DTE LMI up LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0 LMI DLCI 1023 LMI type is CISCO frame relay DTE Broadcast queue 64/64, broadcasts sent/dropped 1769202/1849660 interface broadcasts 3579215 1769202/1849660,

http://www.cisco.com/warp/customer/104/6.shtml

2000, Cisco Systems, Inc.

67

Troubleshooting NSSA
Only NSSA ABR with the highest RID does the conversion
NSSA ASBR
Type 7
router ospf 1 network 10.10.10.10 0.0.0.0 areaNo 0

Area 1 NSSA

RID: 9.9.9.9

NSSA ABR

Area 0

Type 7/5 Translation

RID: 10.10.10.10

Type 7-5 conversion

2000, Cisco Systems, Inc.

68

Troubleshooting Demand Circuits

OSPF (DC) 172.16.1.1/30 172.16.2.0/24

DC is bringing up the link There is a change in OSPF topology debug ip ospf monitor is helpful in this case network type on DC is defined broadcast The reduces flooding but doesnt suppress hellos
2000, Cisco Systems, Inc.

69

Troubleshooting Demand Circuits

OSPF (DC) 172.16.1.1/30 RIP 172.16.2.0/24

RIPOSPF redistribution

DC is bringing up the link PPP host route is also own by RIP When PPP host route disappears, the database is change Solution1: no peer neighbor-route Solution2: distribute-list Solution3: Use different majornet for RIP
2000, Cisco Systems, Inc.

70

Troubleshooting Tips

2000, Cisco Systems, Inc.

2206 1209_05_2000_c3

2000, Cisco Systems, Inc.

71

Troubleshooting Tips
Cannot allocate router ID

OSPF unknown routing protocol OSPF and secondary address Options and flags How to read debug ip ospf adj Route Preference
2000, Cisco Systems, Inc.

72

Troubleshooting Tips
Problem:
OSPF: Could not allocate router id

Reasons:
Common new install problem If no interface up/up with valid ip address if no ip addresses assigned Configure a loopback with an ip address

2000, Cisco Systems, Inc.

73

Troubleshooting Tips
Problem:
OSPF unknown routing protocol

Reason: OSPF is not supported on low end platform For 1000 and 1600 routers download plus version 800 routes are not supported to run ospf
http://www.cisco.com/warp/customer/471/99.html
2000, Cisco Systems, Inc.

74

Troubleshooting Tips
OSPF and secondary addresses
Primary and secondary address should be in same area OSPF hellos are always send from primary interface To form neighbors on secondary interface configure sub-interface Use ISL or dot1Q encapsulation
2000, Cisco Systems, Inc.

75

Troubleshooting Tips
Problem:
OSPF not sending hellos on async interface

Reasons:
async default routing is not configured under the interface

Problem:
OSPF not redistributing default static route

Reasons:
need default-information originate to propagate default

Problem:
OSPF-4-ERRRCV msg on the console

Reasons:
Mismatch area ID, BAD Checksum etc.

http://www.cisco.com/warp/customer/104/19.html
2000, Cisco Systems, Inc.

76

OSPF Protocol Packets

Options
Normal area: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0xBC4 opt 0x2 flag 0x3 len 492
E bit is 1, Allow externals, option: 0x2(HEX)

= 00000010(Bin)

Stub area: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x1866 opt 0x0 flag 0x3 len 372
E bit is 0, no external allowed, options: 0x0

= 00000000

MC not supported - ospf ignore lsa mospf command NSSA: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x118 opt 0x8 flag 0x3 len 372
N/P bit is on, options: 0x8

= 00001000

EA not supported yet Demand circuit : OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x1A1E opt 0x20 flag 0x3 len 392
DC bit is negotiated, options: 0x20

= 00100000

* *
2000, Cisco Systems, Inc.

DC EA N/P MC E

*
77

OSPF Protocol Packets

Flags Useful in debugging, defines I, M and MS bits
OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0xBC4 opt 0x2 flag 0x3 len 492

Flag 0x7--> 111 means I(Initial) = 0, M = 1(More), MS = 1(Master) Flag 0x6 --> 110 not possible Flag 0x5 --> 101 not possible Flag 0x4 --> 100 not possible Flag 0x3 --> 011 means master has more data to send Flag 0x2 --> 010 means slave has more data to send Flag 0x1 --> 001 means master has no more data left to send Flag 0x0 --> 000 means slave has no more data left to send

0
2000, Cisco Systems, Inc.

MS
78

How to read debug ip ospf adj?

OSPF: Rcv hello from 141.108.10.2 area 0 from Serial0.1 141.108.10.2 OSPF: 2 Way Communication to 141.108.10.2 on Serial0.1, state 2WAY OSPF: Neighbor change Event on interface Serial0.1 OSPF: DR/BDR election on Serial0.1 OSPF: Elect BDR 0.0.0.0 OSPF: Elect DR 141.108.10.2 DR: 141.108.10.2 (Id) BDR: none OSPF: Send DBD to 141.108.10.2 on Serial0.1 seq 0x236D opt 0x2 flag 0x7 len 32 OSPF: Rcv DBD from 141.108.10.2 on Serial0.1 seq 0x1996 opt 0x2 flag 0x7 len 32 state EXSTART OSPF: First DBD and we are not SLAVE OSPF: Rcv DBD from 141.108.10.2 on Serial0.1 seq 0x236D opt 0x2 flag 0x2 len 352 state EXSTART OSPF: NBR Negotiation Done. We are the MASTER OSPF: Send DBD to 141.108.10.2 on Serial0.1 seq 0x236E opt 0x2 flag 0x3 len 352 OSPF: Rcv DBD from 141.108.10.2 on Serial0.1 seq 0x236E opt 0x2 flag 0x0 len 32 state EXCHANGE OSPF: Send DBD to 141.108.10.2 on Serial0.1 seq 0x236F opt 0x2 flag 0x1 len 32 OSPF: Rcv DBD from 141.108.10.2 on Serial0.1 seq 0x236F opt 0x2 flag 0x0 len 32 state EXCHANGE OSPF: Exchange Done with 141.108.10.2 on Serial0.1 OSPF: Synchronized with 141.108.10.2 on Serial0.1, state FULL
2000, Cisco Systems, Inc.

79

Route Preference
O, O IA, OE1, OE2
192.1.2.0
3

192.1.1.4 R4
1 8

R1 R2

R5

Area 0
2 8 1

R3

Area 1

R6

R7

Area 2

R7 routing table: O IA 192.1.2.0/24 via R5 metric 20 ??? Summary LSA through R6 has a metric of 15! How to reach the Advertising router R4 (ABR)? O vs O IA
2000, Cisco Systems, Inc.

80

Please Complete Your Evaluation Form

Troubleshooting OSPF
2203 2000, Cisco Systems, Inc. 2648_03_2001_c1
2001, Cisco Systems, Inc.

81

2206 1209_05_2000_c3

2000, Cisco Systems, Inc.

82