Troubleshooting Switch Port and Interface Problems
Troubleshooting Switch Port and Interface Problems
Problems
Document ID: 12027
Introduction
Prerequisites
Requirements
Components Used
Conventions
Physical Layer Troubleshooting
Using the LEDs to Troubleshoot
Check the Cable and Both Sides of the Connection
Ethernet Copper and Fiber Cables
Gigabit Ethernet Troubleshooting
Connected vs Notconnected
Most Common Port and Interface Troubleshooting Commands for CatOS and Cisco
IOS
Understanding Specific Port and Interface Counter Output for CatOS and Cisco IOS
Show Port for CatOS and Show Interfaces for Cisco IOS
Show Mac for CatOS and Show Interfaces Counters for Cisco IOS
Show Counters for CatOS and Show Counters Interface for Cisco IOS
Show Controller Ethernet−Controller for Cisco IOS
Show Top for CatOS
Common System Error Messages
Error Messages on WS−X6348 Modules
%PAGP−5−PORTTO / FROMSTP and %ETHC−5−PORTTO / FROMSTP
%SPANTREE−3−PORTDEL_FAILNOTFOUND
%SYS−4−PORT_GBICBADEEPROM: / %SYS−4−PORT_GBICNOTSUPP
%AMDP2_FE−3−UNDERFLO
%INTR_MGR−DFC1−3−INTR: Queueing Engine (Blackwater) [1]: FIC Fabric−A
Received Unexpected Control Code
Command Rejected: [Interface] not a Switching Port
Common Port and Interface Problems
Port or Interface Status is Disable or Shutdown
Port or Interface Status is errDisable
Port or Interface Status is Inactive
Uplink Port or Interface Status is Inactive
Deferred Counter on the Catalyst Switch Interface Starts Incrementing
Intermittent Failure to set timer [value] from vlan [vlan no]
Trunking Mode Mismatch
Jumbos, Giants, and Baby Giants
Cannot Ping End Device
Using Set Port Host or Switchport Host to Fix Startup Delays
Speed/Duplex, Autonegotiation, or NIC Issues
Spanning Tree Loops
UDLD: One−Way Link
Deferred Frames (Out−Lost or Out−Discard)
Software Problems
Hardware Problems
Input Errors on a Layer 3 Interface Connected to a Layer 2 Switchport
Rapidly Incrementing Rx−No−Pkt−Buff Counter and Input Errors
Trunking between a Switch and a Router
Connectivity Issues due to Oversubscription
Subinterfaces in SPA Modules
Troubleshooting rxTotalDrops
NetPro Discussion Forums − Featured Conversations
Related Information
Introduction
This document is intended to help determine why a port or interface experiences problems. This document
applies to Catalyst switches that run CatOS Software on the Supervisor or Cisco IOS® System Software on
the Supervisor.
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
This document is not restricted to specific software and hardware versions.
Conventions
Refer to Cisco Technical Tips Conventions for more information on document conventions.
Platform
URL
Catalyst 6000 Series Switches
Ethernet Module LEDs
Catalyst 5000 Series Switches
Ethernet Module LEDs
Catalyst 4000 Series Switches
Ethernet Module LEDs
Catalyst 3750 Series Switches
Front Panel LEDs
Catalyst 3550 Series Switches
Front Panel LEDs
Catalyst 2950/2955 Series Switches
Front Panel LEDs
Catalyst 2900/3500XL Series Switches
Front Panel LEDs
Catalyst 1900 and 2820 Series
Switches
Front Panel LEDs
Catalyst G−L3 Series Switches
Front Panel LEDs
Ensure that both sides have a link. A single broken wire or one shutdown port can cause the problem where
one side has a link light, but the other side does not.
A link light does not guarantee that the cable is fully functional. The cable can have encountered physical
stress that causes it to be functional at a marginal level. Normally you can identify this situation if the port has
many packet errors, or the port constantly flaps (loses and regains link).
Possible Cause
Corrective Action
No cable
connected Connect cable from switch to a known good
device.
Wrong Port Make sure that both ends of the cable are
plugged into the correct ports.
Device has no
power
Ensure that both devices have power.
Wrong cable type Verify the cable selection. Refer to the
Catalyst Switch Cable Guide.
Swap suspect cable with known good cable.
Bad cable
Look for broken or missing pins on
connectors.
Check for loose connections. Sometimes a
Loose connections
cable appears to be seated in the jack, but is
not. Unplug the cable and reinsert it.
Eliminate faulty patch panel connections.
Patch Panels
Bypass the patch panel if possible to rule it
out.
Eliminate faulty media convertors:
Media Convertors
fiber−to−copper, etc. Bypass the media
convertor if possible to rule it out.
Bad or wrong Swap suspect GBIC with known good
Gigabit Interface GBIC.
Convertor (GBIC)
Verify Hw and Sw support for this type of
GBIC. See the Gigabit Ethernet
Troubleshooting section of this document.
Bad Port or Move the cable to a known good port to
Module troubleshoot a suspect port or module.
Warning: Category 5e and Category 6 cables can store high levels of static electricity because of the
dielectric properties of the materials used in their construction. Always ground the cables (especially in new
cable runs) to a suitable and safe earth ground before you connect them to the module.
For fiber, make sure you have the correct cable for the distances involved and the type of fiber ports that are
used. The two options are singlemode fiber (SMF) or multimode fiber (MMF). Make sure the ports on the
devices that are connected together are both SMF, or both are MMF ports.
Note: For fiber connections, make sure the transmit lead of one port is connected to the receive lead of the
other port. Connections for transmit−to−transmit and receive−to−receive do not work.
Transceiver
Maximum
Speed Duplex
Cable Type Distance Between
Mode
Stations
10 Mbps Full and
Category 3 UTP 328 ft (100 m)
half
10 Mbps Full and
MMF 1.2 mi (2 km)
half
100 Mbps Category 5 UTP Full and
328 ft (100 m)
Category 5e UTP half
100 Mbps Full and
Category 6 UTP 328 ft (100 m)
half
100 Mbps
MMF Half 1312 ft (400 m)
Full 1.2 mi (2 km)
100 Mbps
SMF Half 1312 ft (400 m)
Full 6.2 mi (10 km)
For more details on the different types of cables/connectors, cabling requirements, optical requirements
(distance, type, patch cables, etc.), how to connect the different cables, and which cables are used by most
Cisco switches and modules, refer to Catalyst Switch Cable Guide.
1. Verify device A and B use the same GBIC, short wavelength (SX), long wavelength (LX), long haul
(LH), extended wavelength (ZX), or copper UTP (TX). Both devices must use the same type of GBIC
to establish link. An SX GBIC needs to connect with an SX GBIC. An SX GBIC does not link with
an LX GBIC. Refer to Mode−Conditioning Patch Cord Installation Note for more information.
2. Verify distance and cable used per GBIC as defined in this table.
Modal
GBIC Wavelength Copper/Fiber Core Size1
Bandwidth Cable
(nm) Type (Microns)
(MHz/km) Distance2
Category 5
WS−G5483 UTP
1000Base − T Category
(copper)
5e UTP 328 ft
Category (100 m)
6 UTP
722 ft
62.5
(220 m)
WS−G5484 160
62.5
1000BASE−SX3 902 ft
850 MMF
200 (275 m)
50.0
400 1640 ft
50.0
(500 m)
500
1804 ft
(550 m)
1804 ft
62.5
(550 m)
WS−G5486 500
MMF4 50.0
1000BASE−LX/LH 1804 ft
1310
400 (550 m)
SMF 50.0
500 1804 ft
8.3/9/10
(550 m)
−
6.2 miles
(10 km)
43.5
miles (70
WS−G5487
MMF 8.3/9/10 km)7
1000BASE−ZX5
1550
SMF6 8.3/9/10 62.1
miles
(100 km)
a. The numbers given for multimode fiber−optic cable refer to the core diameter. For
single−mode fiber−optic cable, 8.3 microns refers to the core diameter. The 9−micron and
10−micron values refer to the mode−field diameter (MFD), which is the diameter of the
light−carrying portion of the fiber. This area consists of the fiber core plus a small portion of
the surrounding cladding. The MFD is a function of the core diameter, the wavelength of the
laser, and the refractive index difference between the core and the cladding.
b. Distances are based on fiber loss. Multiple splices and substandard fiber−optic cable reduce
cabling distances.
c. Use with MMF only.
d. When you use an LX/LH GBIC with 62.5−micron diameter MMF, you must install a
mode−conditioning patch cord (CAB−GELX−625 or equivalent) between the GBIC and the
MMF cable on both the transmit and receive ends of the link. The mode−conditioning patch
cord is required for link distances less than 328 feet (100 m) or greater than 984 feet (300 m).
The mode−conditioning patch cord prevents overdriving the receiver for short lengths of
MMF and reduces differential mode delay for long lengths of MMF. Refer to
Mode−Conditioning Patch Cord Installation Note for more information.
e. Use with SMF only.
f. Dispersion−shifted single−mode fiber−optic cable.
g. The minimum link distance for ZX GBICs is 6.2 miles (10 km) with an 8−dB attenuator
installed at each end of the link. Without attenuators, the minimum link distance is 24.9 miles
(40 km).
3. If either device has multiple Gigabit ports, connect the ports to each other. This tests each device and
verifies that the Gigabit interface functions correctly. For example, you have a switch that has two
Gigabit ports. Wire Gigabit port one to Gigabit port two. Does the link come up? If so, the port is
good. STP blocks on the port and prevents any loops (port one receive (RX) goes to port two transmit
(TX), and port one TX goes to port two RX).
4. If single connection or Step 3 fails with SC connectors, loop the port back to itself (port one RX goes
to port one TX). Does the port come up? If not, contact the TAC, as this can be a faulty port.
5. If steps 3 and 4 are successful, but a connection between device A and B cannot be established, loop
ports with the cable that adjoins the two devices. Verify that there is not a faulty cable.
6. Verify that each device supports 802.3z specification for Gigabit auto−negotiation. Gigabit Ethernet
has an auto−negotiation procedure that is more extensive than the one used for 10/100 Ethernet
(Gigabit auto−negotiation spec: IEEE Std 802.3z−1998). When you enable link negotiation, the
system auto−negotiates flow control, duplex mode, and remote fault information. You must either
enable or disable link negotiation on both ends of the link. Both ends of the link must be set to the
same value or the link cannot connect. Problems have been seen when you connect to devices
manufactured before the IEEE 802.3z standard was ratified. If either device does not support Gigabit
auto−negotiation, disable the Gigabit auto−negotiation, and it forces the link up. It takes 300msec for
the card firmware to notify the software that a 10/100/1000BASE−TX link/port is down. The
300msec default debounce timer comes from the firmware polling timer to the linecards, which
occurs every 300 msec. If this link is run in 1G (1000BASE−TX) mode, Gigabit sync, which occurs
every 10msec, must be able to detect the link down faster. There is a difference in the link failure
detection times when you run GigabitEthenet on copper versus GigabitEthernet over Fibre. This
difference in detection time is based on the IEEE standards.
Warning: Disabling auto−negotiation hides link drops or physical layer problems. Disabling
auto−negotiation is only required if end−devices such as older Gigabit NICs are used which cannot
support IEEE 802.3z. Do not disable auto−negotiation between switches unless absolutely required to
do so, as physical layer problems can go undetected, which results in STP loops. The alternative is to
contact the vendor for software/hardware upgrade for IEEE 802.3z Gigabit auto−negotiation support.
For GigabitEthernet system requirements as well as Gigabit Interface Converters (GBICs), Coarse
Wavelength Division Multiplexing (CWDM), and Small Form−Factor Pluggable (SFP) system requirements,
refer to these:
For general configuration and troubleshooting information, refer to Configuring and Troubleshooting Ethernet
10/100/1000 MB Half/Full Duplex Auto−Negotiation.
Connected vs Notconnected
Most Cisco switches default to having a port in the notconnect state. This means it is currently not connected
to anything, but it will connect if it has a good connection to another operational device. If you connect a good
cable to two switch ports in the notconnect state, the link light must become green for both ports, and the port
status must indicate connected. This means that the port is up as far as Layer 1 (L1) is concerned.
For CatOS, you can use the show port command to verify whether the port has a connected or notconnect
status, or whether it is another state that would cause connectivity to fail, like disabled or errdisable.
For Cisco IOS, you can use the show interfaces command to verify whether the interface is "up, line protocol
is up (connected)". The first "up" refers to the physical layer status of the interface. The "line protocol up"
message shows the data link layer status of the interface and says that the interface can send and receive
keepalives.
If show port shows connected or show interfaces shows up/ line protocol up (connected) but you see errors
incrementing in the output of either command, refer to the Understanding Specfic Port and Interface Counter
Output for CatOS or Cisco IOS or Common Port and Interface Problems sections of this document for
troubleshooting advice.
Note: Choose a command in the left hand column to go to documentation for that command. The right hand
column gives a brief description of what the command does and lists any exceptions to it's use per platform.
These commands are supported by the Output Interpreter tool for CatOS and can be used to assist in
troubleshooting switch port or problems: show version, show module, show port, show counters, or show
mac.
If you have the output of the supported commands from your Cisco device, you can use to display potential
issues and fixes. In order to use Output Interpreter, you must be a registered user, be logged in, and have
JavaScript enabled.
CatOS
Commands Cisco IOS
Description
Commands
show show version For switches that run CatOS,
version this command displays
software and hardware
version info per module and
system memory sizes.
Helpful in determining
whether a misconfiguration
of the mod/port or interface,
can cause a problem.
show port show interfaces For CatOS, the show port
command displays whether
the port is connected, what
VLAN it is in, what
speed/duplex it is that run at,
channel information, errors,
etc.
Note: There can be differences in the implementation of the counters across various platforms and releases.
Although the values of the counters are largely accurate, they are not very precise by design. In order to pull
the exact statistics of the traffic, it is suggested that you use a sniffer to monitor the necessary ingress and
egress interfaces.
Excessive errors for certain counters usually indicate a problem. When you operate at half−duplex setting,
some data link errors incrementing in Frame Check Sequence (FCS), alignment, runts, and collision counters
are normal. Generally, a one percent ratio of errors to total traffic is acceptable for half−duplex connections. If
the ratio of errors to input packets is greater than two or three percent, performance degradation can be
noticed.
In half−duplex environments, it is possible for both the switch and the connected device to sense the wire and
transmit at exactly the same time and result in a collision. Collisions can cause runts, FCS, and alignment
errors due to the frame not being completely copied to the wire, which results in fragmented frames.
When you operate at full−duplex, errors in FCS, Cyclic Redundancy Checks (CRC), alignment, and runt
counters must be minimal. If the link operates at full−duplex, the collision counter is not active. If the FCS,
CRC, alignment, or runt counters increment, check for a duplex mismatch. Duplex mismatch is a situation
where the switch operates at full−duplex and the connected device operates at half−duplex, or vice versa. The
results of a duplex mismatch are extremely slow performance, intermittent connectivity, and loss of
connection. Other possible causes of data link errors at full−duplex are bad cables, faulty switch ports, or NIC
software/hardware issues. See the Common Port and Interface Problems section of this document for more
information.
Show Port for CatOS and Show Interfaces for Cisco IOS
The show port {mod/port} command is used when that run CatOS on the Supervisor. An alternative to this
command is the show port counters {mod/port} which only displays the port error counters. Refer to Table 1
for explanations of the error counter output.
The show interfaces card−type {slot/port} command is the equivalent command for Cisco IOS on the
Supervisor. An alternative to this command (for Catalyst 6000, 4000, 3550, 2970 2950/2955, and 3750 series
switches) is the show interfaces card−type {slot/port} counters errors command which only displays the
interface error counters.
Note: For 2900/3500XL Series switches use the show interfaces card−type {slot/port} command with the
show controllers Ethernet−controller command.
The show interfaces command output up to this point is explained here (in order) :
• up, line protocol is up (connected) − The first "up" refers to the physical layer status of the interface.
The "line protocol up" message shows the data link layer status of the interface and says that the
interface can send and receive keepalives.
• MTU − The Maximum Transmission Unit (MTU) is 1500 bytes for Ethernet by default (for the max
data portion of the frame).
• Full−duplex, 100Mb/s − Full−duplex and 100Mbps is the current speed and duplex setting of the
interface. This does not tell you whether autoneg was used to achieve this. Use the show interfaces
fas 6/1 status command to display this:
Note: Variables that can affect routing (for example, load and reliability) are not cleared when the
counters are cleared.
• Input queue − The number of packets in the input queue. Size/max/drops = the current number of
frames in the queue / the max number of frames the queue can hold before it must start dropping
frames / the actual number of frames dropped because the max queue size was exceeded. Flushes is
used to count Selective Packet Discard (SPD) drops on the the Catalyst 6000 Series that run Cisco
IOS. (The flushes counter can be used but never increments on the Catalyst 4000 Series that run Cisco
IOS.) SPD is a mechanism that quickly drops low priority packets when the CPU is overloaded in
order to save some processing capacity for high priority packets. The flushes counter in the show
interface command output increments as part of selective packet discard (SPD), which implements a
selective packet drop policy on the IP process queue of the router. Therefore, it applies to only
process switched traffic.
The purpose of SPD is to ensure that important control packets, such as routing updates and
keepalives, are not dropped when the IP input queue is full. When the size of the IP input queue is
between the minimum and maximum thresholds, normal IP packets are dropped based on a certain
drop probability. These random drops are called SPD flushes.
• Total output drops − The number of packets dropped because the output queue is full. A common
cause of this might be traffic from a high bandwidth link being switched to a lower bandwidth link or
traffic from multiple inbound links being switched to a single outbound link. For example, if a large
amount of bursty traffic comes in on a gigabit interface and is switched out to a 100Mbps interface,
this might cause output drops to increment on the 100Mbps interface. This is because the output
queue on that interface is overwhelmed by the excess traffic due to the speed mismatch between the
inbound and outbound bandwidths.
• Output queue − The number of packets in the output queue. Size/max means the current number of
frames in the queue/the max number of frames the queue can hold before it is full and must start
dropping frames.
• 5 minute input/output rate − The average input and output rate seen by the interface in the last five
minutes. In order to get a more accurate reading by specifying a shorter period of time (to better
detect traffic bursts for example), issue the load−interval <seconds> interface command.
The rest of the show interfaces command displays error counter output which is similar or equivalent to
CatOS error counter output. See Table 1 for explanations of the error counter output.
Note: There is a difference between the counter of show interface command output for a physical
interface and a VLAN interface. The input packet counters increment in the output of show interface for a
VLAN interface when that packet is Layer 3 (L3) processed by the CPU. Traffic that is Layer 2 (L2) switched
never makes it to the CPU and is not counted in the show interface counters for the VLAN interface. It would
be counted on the show interface output for the appropriate physical interface.
The show interfaces card−type {slot/port} counters errors command is the equivalent Cisco IOS command
to show port counters for CatOS. See Table 1 for explanations of the error counter output.
Table 1:
CatOS error counter output for show port or show port counters for the Catalyst 6000, 5000 and 4000
Series. Cisco IOS error counter output for show interfaces or show interfaces card−type x/y counters errors
for the Catalyst 6000 and 4000 Series.
Counters (in
alphabetical
order) Description and Common Causes of
Incrementing Error Counters
Align−Err Description: CatOS sh port and Cisco IOS sh
interfaces counters errors. Alignment errors
are a count of the number of frames received
that don't end with an even number of octets
and have a bad Cyclic Redundancy Check
(CRC).
Platform Exceptions:
Show Mac for CatOS and Show Interfaces Counters for Cisco IOS
The show mac {mod/port}command is useful when that run CatOS on the supervisor to monitor inbound and
outbound traffic on the port as displayed by the receive (Rcv) and transmit (Xmit) counters for unicast,
multicast, and broadcast traffic. This output is from a Catalyst 6000 that run CatOS:
Port Last−Time−Cleared
−−−−− −−−−−−−−−−−−−−−−−−−−−−−−−−
3/1 Sun Jun 1 2003, 12:22:47
This command also has these error counters: Dely−Exced, MTU−Exced, In−Discard and Out−Discard.
• Dely−Exced − The number of frames discarded by this port due to an excessive transmit delay
through the switch. This counter must never go up unless the port is under very high utilization.
• MTU Exceed − This is an indication that one of the devices on that port or segment transmits more
than the allowed frame size (1518 bytes for non−jumbo Ethernet).
• In−Discard − The result of inbound valid frames that were discarded because the frame did not need
to be switched. This can be normal if a hub is connected to a port and two devices on that hub
exchange data. The switch port still sees the data but does not have to switch it (since the CAM table
shows the MAC address of both devices associated with the same port), and so it is discarded. This
counter can also increment on a port configured as a trunk if that trunk blocks for some VLANs, or on
a port that is the only member of a VLAN.
• Out−Discard − The number of outbound packets chosen to be discarded even though no packet errors
have been detected. One possible reason to discard such a packet can be to free up buffer space.
Catalyst 4000 and 5000 series switches that run CatOS have two additional error counters in the show mac
command. These are the In−Lost and Out−Lost counters:
MAC Dely−Exced MTU−Exced In−Discard Lrn−Discrd In−Lost Out−Lost
−−−−−−−− −−−−−−−−−− −−−−−−−−−− −−−−−−−−−− −−−−−−−−−− −−−−−−−−−− −−−−−−−−−−
5/1 0 0 0 0 0 0
• In−Lost − On the Catalyst 4000, this counter is the sum of all error packets received on the port. The
In−Lost counter on the Catalyst 5000, on the other hand, tracks the sum of all receive buffer failures.
• Out−Lost − On the Catalyst 4000 and 5000, these are outbound frames that were lost before they were
forwarded (due to insufficient buffer space). This is commonly caused by oversubscribing the port.
The show interfaces card−type {slot/port} counters command is used when you run Cisco IOS on the
Supervisor.
Note: There are no equivalent counters to the CatOS show mac error counters: Dely−Exced, MTU−Exced
and In−Discard in this command There is, however, an Out−Discard counter in the Cisco IOS show
interfaces counters errors command which is explained in Table 1.
Show Counters for CatOS and Show Counters Interface for Cisco IOS
The show counters [mod/port] command offers even more detailed statistics for ports and interfaces. This
command is available for CatOS and the equivalent show counters interface card−type {slot/port} command
was introduced in Cisco IOS software version 12.1(13)E for the Catalyst 6000 series only. These commands
display the 32−bit and 64−bit error counters per port or interface. Refer to the CatOS command
documentation for show counters for more information.
Note: Counter stats for Catalyst 6000 series switches that run Cisco IOS are in Hex.
3550−1#
!−−− See table for additional counter output for 2900/3500XL Series switches.
Counter
Description Possible Causes
Transmitted Frames
The total number of
frames whose
The traffic load on the
Discarded transmission attempt is
interface is excessive
frames abandoned due to
and causes the frames
insufficient resources.
to be discarded.
This total includes
Reduce the traffic
frames of all destination
load on the interface
types.
if you see an
increasing number of
Number of frames that packets in this field.
The traffic load for
took longer than two
this switch is
seconds to travel
Too old excessive and causes
through the switch. For
frames the frames to be
this reason, they were
discarded. Reduce the
discarded by the switch.
switch load if you see
This only happens
an increasing number
under extreme, high
of packets in this
stress conditions.
field. You can need to
modify your network
topology to reduce the
traffic load for this
switch.
The total number of
frames whose first
transmission attempt
was delayed, due to
traffic on the network The traffic load
Deferred
media. This total destined for this
frames
includes only those switch is excessive
frames that are and causes the frames
subsequently to be discarded.
transmitted without Reduce the switch
error and without load if you see an
experiencing a increasing number of
collision. packets in this field.
You can need to
modify your network
The collision frames
topology to reduce the
counters are the number
traffic load for this
of times a packet was
switch.
attempted to be
transmitted but was not
successful, but was
Collision
successful on its next
frames
attempt. This means
that if the 2 collision The traffic load on the
frames counter interface is excessive
incremented, the switch and causes the frames
attempted to send the to be discarded.
packet twice and failed Reduce the traffic
but was successful on load on the interface
its third attempt. if you see an
increasing number of
packets in these
fields.
The excessive collisions
counter increases after
16 consecutive late If this counter
Excessive collisions have occurred increments, it is an
collisions in a row. After 16 indication of a wiring
attempts have been problem, an
made to send the packet excessively loaded
the packet is dropped, network, or a duplex
and the counter mismatch. An
increments. excessively loaded
network can be
caused by too many
devices on a shared
A late collision occurs
Ethernet.
when two devices
transmit at the same
time, and neither side of
the connection detects a
collision. The reason
for this occurrence is
because the time to
propagate the signal
from one end of the
network to another is
longer than the time to
put the entire packet on
Late collisions the network. The two
devices that cause the
late collision never see
that the other is sending
until after it puts the
entire packet on the
network. Late collisions
are not detected by the
transmitter until after
the first 64 byte slot
time. This is because Late collisions are a
they are only detected result of incorrect
in transmissions of cabling or a
packets longer than 64 non−compliant
bytes. number of hubs in the
network. Bad NICs
The total number of
can also cause late
Good (1 coll) frames which
Collisions
collisions. in a
frames experience exactly one
half−duplex
collision and are then
environment are
successfully
normal expected
transmitted.
behavior.
Good (>1 coll) The total number of Collisions in a
frames frames which half−duplex
experience between 2 environment are
and 15 collisions, normal expected
inclusive, and are then behavior. Frames that
successfully increment at the upper
transmitted. end of this counter
run the risk of
exceeding 15
collisions and being
counted as Excessive
collisions.
The Canonical Format
Indicator (CFI) bit in
the TCI of an 802.1q
frame is is set to 0 for
the ethernet canonical
VLAN The number of frames
frame format. If the
discardframes dropped on an interface
CFI bit is set to 1, this
because the CFI bit is
indicates the presence
set.
of a RIF (Routing
Information Field) or
Token Ring
noncanonical frame
which is discarded.
Received Frames
2900/3500XL only. The
number of times that a
port received a packet
from the network, but
the switch did not have
the resources to receive
it. This only happens
No bandwidth
under stress conditions
frames
but can happen with
bursts of traffic on
several ports. So, a
The traffic load on the
small number of No
interface is excessive
bandwidth frames is not
and causes the frames
a cause for concern. (It
to be discarded.
still must be far less
Reduce the traffic
than one percent of the
load on the interface
frames received.)
if you see an
No buffers 2900/3500XL only. The The traffic number
increasing load on of
the
frames number of times that a interface
packets inisthis
excessive
field.
port received a packet and causes the frames
from the network, but to be discarded.
the switch did not have Reduce the traffic
the resources to receive load on the interface
it. This only happens if you see an
under stress conditions increasing number of
but can happen with packets in this field.
bursts of traffic on
several ports. So, a
small number of No
buffers frames is not a
cause for concern. (It
still must be far less
than one percent of the
frames received.)
No destination unicast These are brief
No dest,
are the number of descriptions of when
unicast
unicast packets that the the No dest, (unicast,
port did not forward to multicast, and
any other ports. broadcast) counters
can increment:
No destination
No dest, multicast are the
• If a port is an
multicast number of multicast
access port,
packets that the port did
and the port is
not forward to any other
connected to
ports.
an
No No destination Inter−Switch
dest,broadcast broadcast are the Link Protocol
number of broadcast (ISL) trunk
packets that the port did port, the No
not forward to any other dest counter
ports. is very large
since all
inbound ISL
packets are
not
forwarded.
This is an
invalid
configuration.
• If a port is
blocked by
Spanning
Tree Protocol
(STP), most
packets are
not orwarded,
which results
in No dest
packets. If a
port just
acquired a
link, there is a
very brief
(less than one
second)
period where
inbound
packets are
not
forwarded.
• If the port is
in a VLAN by
itself, and no
other ports on
the switch
belong to that
VLAN, all
inbound
packets are
dropped and
the counter
increments.
• The counter
also
increments
when the
destination
address of the
packet is
learned on the
port that the
packet was
received on.
If a packet
was received
on port 0/1,
with
destination
MAC address
X, and the
switch has
already
learned that
MAC address
X resides on
port 0/1, it
increments
the counter
and discards
the packet.
This can
happen in
these
situations:
♦ If a
hub is
connected
to
port
0/1,
and a
workstation
connected
to the
hub
transmits
a
packets
to
another
workstation
connected
to the
hub,
port
0/1
does
not
forward
this
packet
anywhere
because
the
destination
MAC
resides
on the
same
port.
♦ This
can
also
occur
if a
switch
is
connected
to
port
0/1,
and
starts
to
flood
packets
to all
of its
ports
to
learn
MAC
addresses.
• If a static
address has
been set up on
another port
in the same
VLAN, and
no static
address was
set up for the
receiving
port, the
packet is
dropped. For
example, if a
static map for
MAC address
X was
configured on
port 0/2 to
forward
traffic to port
0/3, the
packet must
be received
on port 0/2
otherwise the
packet is
dropped. If a
packet is sent
from any
other port, in
the same
VLAN as port
0/2, the
packet is
dropped.
• If the port is a
secure port,
packets with
disallowed
source MAC
addresses are
not forwarded
and increment
the counter.
Alignment errors are
due to the frame not
being completely
copied to the wire,
Alignment errors are which results in
Alignment the number of frames fragmented frames.
errors received that do not end Alignment errors are
with an even number of the result of collisions
octets and have a bad at half−duplex, a
CRC. duplex mismatch, bad
hardware (NIC, cable,
or port), or connected
device generating
frames that do not end
with an octet and have
a bad FCS.
FCS error count is the
number of frames that
were received with a
bad checksum (CRC
FCS errors FCS errors are the
value) in the Ethernet
result of collisions at
frame. These frames are
half−duplex, a duplex
dropped and not
mismatch, bad
propagated onto other
hardware (NIC, cable,
ports.
or port), or a
These are the total connected device
number of packets generating frames
Undersize received that were less with bad FCS.
This is an indication
frames than 64 octets long
of a bad frame
(which exclude framing
generated by the
bits, but include FCS)
connected device.
and have a good FCS
Verify that the
value.
connected device
Number of packets operates correctly.
Oversize This can be an
received by the port
frames indication of faulty
from the network,
hardware, dot1q or
where the packets were
ISL trunking
more than 1514 bytes.
configuration issues.
The total number of
frames whose length is
Collision If this counter
less than 64 octets
fragments increments, this is an
(which exclude framing
indication that the
bits, but which include
ports are configured
FCS) and have a bad
at half−duplex.
FCS value.
Change the duplex
The number of times setting to full−duplex.
Overrun
the receiver hardware The input rate of
frames
was unable to hand traffic exceeded the
received data to a ability of the receiver
hardware buffer. to handle the data.
The port can be
The total number of configured to filter
VLAN filtered frames which are 802.1Q tagged
frames filtered because of the frames. When a frame
type of VLAN is received which
information contained contains an 802.1Q
in the frame. tag the frame is
filtered and this
statistic is
incremented.
Source routed The total number of This kind of source
frames receive frames that are routing is only
discarded due to the defined for Token
source route bit being Ring and FDDI. The
set in the source address IEEE ethernet
of the native frame. specification forbids
this bit to be set in
any Ethernet frame.
Therefore, the switch
discards such frames.
This statistic counts
The total number of frames that exceed the
Valid oversize
frames received whose configured System
frames
length exceeds the MTU but which can
System MTU yet which have been increased
have good FCS values. from 1518 bytes to
allow for Q−in−Q or
MPLS encapsulations.
Gigabit Ethernet (1000
Base−X) uses 8B/10B
Encoding to translate
8bit data from the MAC
Symbol error A Symbol error
sublayer(layer 2) to a
frames means the interface
10bit Symbol to send
detects an undefined
over the wire. When a
(invalid) Symbol
port receives a Symbol,
received. Small
it extracts the 8 bit data
amounts of symbol
from the Symbol (10
errors can be ignored.
bits).
Large amounts of
Giant frames or frames symbol errors can
received that exceed the indicate a bad device,
Invalid frames, maximum IEEE 802.3 cable, or hardware.
too large frame size (1518 bytes In many cases, this is
for non−jumbo the result of a bad
Ethernet) and have a NIC. Try to find the
bad Frame Check offending device and
Sequence (FCS). remove it from the
network.
Runt frames or frames
received that are less
Invalid frames, than 64 bytes (which
This can be caused by
too small includes the FCS bits
a duplex mismatch
and excludes the frame
and physical
header) and have either
problems, such as a
an FCS error or an
bad cable, port, or
alignment error.
NIC on the attached
device.
Show Top for CatOS
The command show top allows you to collect and analyze data for each physical port on a switch. The
command displays this data for each physical port:
Note: When you calculate port utilization, the command bundles the Tx and Rx lines into the same counter
and also looks at the full−duplex bandwidth when it calculates the percentage of utilization. For example, a
Gigabit Ethernet port is 2000 Mbps full duplex.
Buffer Overflow means that the port receivesg more traffic than it can store in its buffer. This can be caused a
bursty traffic, as well an overrun of buffers. The suggested action is to decrease the transmission of the source
device.
Also refer to 'In−Lost' and 'Out−Lost' counters from the show mac command.
You can see syslog messages with one of those errors listed:
• Troubleshooting WS−X6348 Module Port Connectivity for Catalyst 6000 Using CatOS
• Troubleshooting WS−X6348 Module Port Connectivity on a Catalyst 6500/6000 that run Cisco IOS
System Software
This message can cause concern for customers but for the most part it is informational in nature.
The Port Aggregation Protocol (PAgP) negotiates Etherchannel links between switches. Whenever a device
joins or leaves a bridge port, an informational message is displayed on the console. In most cases this message
is completely normal, but if you see these messages on ports that should not flap for any reason, you must
investigate further.
In CatOS software version 7.x and later, "PAGP−5" was changed to "ETHC−5" to make the message more
understandable.
This message is specific to the Catalyst 4000, 5000, and 6000 series switches that run CatOS. There are no
error messages for switches that run Cisco IOS that are equivalent to this message. For more information,
about error messages on switches that run CatOS, refer to these documents for your platform:
%SPANTREE−3−PORTDEL_FAILNOTFOUND
This message does not indicate a problem with the switch. It normally occurs along with
%PAGP−5−PORTFROMSTP messages.
The Port Aggregation Protocol (PAgP) negotiates Etherchannel links between switches. Whenever a device
joins or leaves a bridge port, an informational message is displayed on the console. In most cases this message
is completely normal, but if you see these messages on ports that should not flap for any reason, you must
investigate further.
This message is specific to the Catalyst 4000, 5000, and 6000 series switches that run CatOS. There are no
error messages for switches that run Cisco IOS that are equivalent to this message. For more information
about error messages on switches that run CatOS, refer to these documents for your platform:
GBIC modules WS−G5484, WS−G5486, and WS−G5487 used with a WS−X6408−GBIC can also cause
these error messages to appear, but there is not an actual problem with the card or GBICs and there is a
software upgrade fix.
Refer to this Common CatOS Error Messages on Catalyst 6000/6500 Series Switches for more information.
%AMDP2_FE−3−UNDERFLO
This error message is caused when a frame is transmitted, and the local buffer of the controller chip local
buffer receives insufficient data. The data cannot be transferred to the chip fast enough to keep pace with
output rate. Normally, such a condition is temporary, dependent upon transient peak loads within the system.
The issue occurs when an excessive amount of traffic is processed by the Fast Ethernet interface. The error
message is received when the traffic level reaches about 2.5 Mb. This traffic level constrain is due to hardware
limitation. Because of this, a chance exists for the device connected to the catalyst switch to drop packets.
The resolution is that ordinarily the system recovers automatically. No action is required. If the switch
overwhelms the Ethernet interface, check the speed and duplex settings. Also use a sniffer program to analyze
packets that come in and out of the router fast Ethernet interface. In order to avoid packet drops on the device
connected to the catalyst switch, issue the ip cef command on the fast Ethernet interface of the device
connected to the switch.
In order to convert the interface from layer 3 mode to layer 2 mode, issue the interface configuration
command switchport. After you issue this command, configure the port for any layer 2 properties.
Make sure the port or module has not been disabled or powered down for some reason. If a port or module is
manually shut down on one side of the link or the other, the link does not come up until you re−enable the
port. Check the port status on both sides.
For CatOS, check show port and, if the port is disabled, re−enable it:
Use the show module command to determine if the module is disabled. If it is, re−enable it:
For Cisco IOS, use the show run interface command and check to see if the interface is in a shutdown state:
If the port goes into shutdown mode immediately after a reboot of the switch, the probable cause is the port
security setting. If unicast flooding is enabled on that port, it can cause the port to shut down after a reboot.
Cisco recommends that you disable the unicast flooding because it also ensure that no flooding occurs on the
port once the MAC address limit is reached.
When you look at show port command for CatOS the status can read errdisable:
Or use the show interface card−type {slot/port} status command for Cisco IOS:
The show logging buffer command for CatOS and the show logging command for Cisco IOS also display
error messages (exact message format varies) that relate to the errdisable state.
Ports or interlaces being shut down as a result of errdisable are referred to as reasons in CatOS and causes in
Cisco IOS. The reasons or causes for this happening range from EtherChannel misconfiguration that causes a
PAgP flap, duplex mismatch, BPDU port−guard and portfast configured at the same time, UDLD that detects
a one−way link, etc.
You have to manually re−enable the port or interface to take it out the errdisable state unless you configure an
errdisable recovery option. In CatOS software 5.4(1) and later you have the ability to automatically re−enable
a port after a configurable amount of time spent in the errdisable state. Cisco IOS on most switches also has
this functionality. The bottom line is that even if you configure the interface to recover from errdisable the
problem reoccurs until the root cause is determined.
For more information on the causes of and recovery from the errdisable status for switches that run CatOS,
refer to Recovering From errDisable Port State on the CatOS Platforms.
Note: Use this link as a reference for errdisable status on switches that run Cisco IOS, as well since the root
causes are the same no matter which operating system you run.
This table shows a comparison of the commands used to configure verify and troubleshoot the errdisable
status on switches that run CatOS and Cisco IOS. Choose a command to go to the command documentation.
CatOS errdisable
Cisco IOS
Commands
Action errdisable
Commands
errdisable detect
set errdisable−timeout
cause
{enable | disable} {reason} set or
configure
errdisable
recovery cause
set errdisable−timeout
interval {interval set or errdisable
configure recovery {interval
show errdisable
detect
show errdisable−timeout verify &
troubleshoot
show interfaces
status err−disabled
Note: Some switches show a steady orange (amber) light on each port when this happens.
For CatOS, use the show port or show port status command along with the show vlan command to verify:
For Cisco IOS, use the show interfaces card−type {slot/port} switchport command along with show vlan to
verify.
Router#sh vlan
If the switch that deleted the VLAN is a VTP server for the VTP domain, every server and client switch in the
domain has the VLAN removed from their VLAN table as well. When you add the VLAN back into the
VLAN table from a VTP server switch, the ports of the switches in the domain that belong to that restored
VLAN become active again. A port remembers what VLAN it is assigned to, even if the VLAN itself is
deleted.
Refer to Understanding and Configuring VLAN Trunk Protocol (VTP) for more information on VTP.
Note: If the output of the show interface <interface number> switchport command displays the port as a
trunk port even after you configure the port as an access port with the switchport access vlan <vlan no:>
command, issue the switchport mode access command in order to make the port an access port.
Cisco IOS Software Release 12.2(25)SG supports the simultaneous use of 10−Gigabit Ethernet and the
Gigabit Ethernet SFP interfaces on Catalyst 4500 switches.
Note: On the Catalyst 4503, 4506, and 4507R series switches, this capability is automatically enabled.
• Hardcode both ends of the link to full duplex so that the negotiation mismatch can be avoided.
• Change the cable and patch panel cord to ensure that the cable and patch cords are not defective.
Note: If the Deferred Counter error increments on a GigabitEthernet of a Supervisor 720, turn on speed
negotiation on the interface as a workaround.
When the VLAN is already in fast aging, EARL cannot set the VLAN to fast aging, and aging timer set
process is blocked. The default CAM aging time is five minutes, which means that the switch flushes the table
of learned MAC addresses every five minutes. This ensures that the MAC address table (the CAM table)
contains the most recent entries.
Fast aging temporarily sets the CAM aging time to the number of seconds that the user specifies, and is used
in conjunction with the Topology Change Notification (TCN) process. The idea is that when a topology
change occurs, this value is necessary to flush the CAM table faster, to compensate for the topology change.
Issue the show cam aging command to check the CAM aging time on the switch. TCNs and fast aging are
fairly rare. As a result, the message has a severity level of 3. If the VLANs are frequently in fast aging, check
the reason for fast aging.
The most common reason for TCNs is client PCs connected directly to a switch. When you power up or down
the PC, the switch port changes state, and the switch starts the TCN process. This is because the switch does
not know that the connected device is a PC; the switch only knows that the port has changed the state.
In order to resolve this issue, Cisco has developed the PortFast feature for host ports. An advantage of
PortFast is that this feature suppresses TCNs for a host port.
Note: PortFast also bypasses spanning−tree calculations on the port, and is therefore only suitable for use on a
host port.
or
set port host mod/port Cisco recommends this command if the switch runs CatOS5.4 or higher versions.
For CatOS, use the show trunk {mod/port}command to verify the trunk status and Native VLAN (for dot1q)
matches on both sides.
For Cisco IOS, use the show interfaces card−type {mod/port} trunk command to verify the trunking
configuration and Native VLAN.
Refer to these documents for more information on the different trunking modes, guidelines, and restrictions:
• Jumbo frames are not defined as part of the IEEE Ethernet standard and are vendor−dependent. They
can be defined as any frame bigger than the standard ethernet frame of 1518 bytes (which includes the
L2 header and Cyclic Redundancy Check (CRC)). Jumbos have larger frame sizes, typically > 9000
bytes.
• Giant frames are defined as any frame over the maximum size of an ethernet frame (larger than 1518
bytes) that has a bad FCS.
• Baby Giant frames are just slightly larger than the maximum size of an ethernet frame. Typically this
means frames up to 1600 bytes in size.
Support for jumbo and baby giants on Catalyst switches varies by switch platform, sometimes even by
modules within the switch. The software version is also a factor.
Refer to Configuring Jumbo/Giant Frame Support on Catalyst Switches for more information on system
requirements, configuring and troubleshooting for jumbo and baby giant issues.
VLAN Dest MAC/Route Des [CoS] Destination Ports or VCs / [Protocol Type]
−−−− −−−−−−−−−−−−−−−−−− −−−−− −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
2 00−40−ca−14−0a−b1 3/1 [ALL]
!−−− A workstation on VLAN 2 with MAC address 00−40−ca−14−0a−b1 is seen in the CAM table
!−−− on the trunk port of a switch running CatOS.
Total Matching CAM Entries Displayed =1
Console> (enable)
For Cisco IOS, use the show mac address−table dynamic command, or substitute the interface keyword.
Once you know the switch actually has the MAC address of the device in it's CAM table, determine whether
this device is on the same or different VLAN from where you are trying to ping.
If the end device is on a different VLAN from where you are trying to ping, a L3 switch or router must be
configured to allow the devices to communicate. Make sure your L3 addressing on the end device and on the
router/ L3 switch is correctly configured. Check the IP address, subnet mask, default gateway, dynamic
routing protocol configuration, static routes, etc.
The set port host command was introduced in CatOS Version 5.4. This command sets the trunking and
channel modes to off and puts the port in a STP forwarding state.
Note: For CatOS versions earlier than version 5.4, the set spantree portfast {mod/port} enable command
was used. In current versions of CatOS, you still have the option to use only this command, but this requires
that you turn off trunking and etherchannel separately to help fix workstation startup delays. The additional
commands to do this are: set port channel {mod/port} off and set trunk {mod/port} off.
For Cisco IOS, you can use the switchport host command to disable channeling and to enable spanning−tree
portfast and the switchport nonegotiate command to turn off DTP negotiation packets. Use the
interface−range command to do this on multiple interfaces at once.
Cisco IOS has the option to use the global spanning−tree portfast default command to automatically apply
portfast to any interface configured as a layer 2 access switchport. Check the Command Reference for your
release of software to verify the availability of this command. You can also use the spanning−tree portfast
command per interface, but this requires that you turn off trunking and etherchannel separately to help fix
workstation startup delays.
Refer to Using Portfast and Other Commands to Fix Workstation Startup Connectivity Delays for more
information how to fix startup delays.
• Duplex Mismatch
• Bad or Damaged Cable
• NIC Card Issues
Duplex Mismatch
A common issue with speed/duplex is when the duplex settings are mismatched between two switches,
between a switch and a router or between the switch and a workstation or server. This can occur when
manually hardcoding the speed and duplex or from autonegotiation issues between the two devices.
If the mismatch occurs between two Cisco devices with the Cisco Discovery Protocol (CDP) enabled, you see
the CDP error messages on the console or in the logging buffer of both devices. CDP is useful to detect errors,
as well as port and system statistics on nearby Cisco devices. CDP is Cisco proprietary and works by sending
packets to a well−known mac address 01−00−0C−CC−CC−CC.
The example shows the log messages that result from a duplex mismatch between two Catalyst 6000 series
switches: one that runs CatOS, and the other that runs Cisco IOS. These messages generally tell you what the
mismatch is and where it occurs.
For CatOS, use the show cdp neighbor [mod/port] detail command to display CDP information for Cisco
neighbor devices.
For Cisco IOS, use the show cdp neighbors card−type {slot/port} detail command to display CDP
information for Cisco neighbor devices.
Setting auto speed/duplex on one side and 100/Full−duplex on the other side is also a misconfiguration, and
can result in a duplex mismatch. If the switch port receives a lot of late collisions, this usually indicates a
duplex mismatch problem and can result in the port being placed in an errdisable status. The half duplex side
only expects packets at certain times, not at any time, and therefore counts packets received at the wrong time
as collisions. There are other causes for late collisions besides duplex mismatch but this is one of the most
common reasons. Always set both sides of the connection to auto−negotiate speed/duplex, or set the
speed/duplex manually on both sides.
For CatOS, use the show port status [mod/port] command to display the speed and duplex status as well as
other information. Use the set port speed and set port duplex commands to hardcode both sides to 10 or 100
and half or full as necessary.
For Cisco IOS, use the show interfaces card−type {slot/port} status command to display speed and duplex
settings as well as other information. Use the speed and duplex commands from interface configuration mode
to hardcode both sides to 10 or 100 and half or full as necessary.
If you use the show interfaces command without the status option, you see a setting for speed and duplex,
but you do not know whether this speed and duplex was achieved through autonegotiation or not.
Always check the cable for marginal damage or failure. A cable can be just good enough to connect at the
physical layer, but it corrupts packets as a result of subtle damage to the wiring or connectors. Check or swap
the copper or fiber cable. Swap the GBIC (if removable) for fiber connections. Rule out any bad patch panel
connections or media convertors between source and destination. Try the cable in another port or interface if
one is available and see if the problem continues.
In order to troubleshoot autonegotiation problems it is often recommended to try hardcoding both sides. If
neither autonegotiation or hardcoding seem to work, there can be a problem with the firmware or software on
your NIC card. Upgrade the NIC card driver to the latest version available on the web site of the manufacture
to resolve this.
Refer to Configuring and Troubleshooting Ethernet 10/100/1000 MB Half/Full Duplex Auto−Negotiation for
details on how to resolve speed/duplex and autonegotiation issues.
Refer to Troubleshooting Cisco Catalyst Switches to NIC Compatibility Issues for details on how to resolve
third−party NIC issues.
The STP loop guard feature provides additional protection against Layer 2 forwarding loops (STP loops). An
STP loop is created when an STP blocking port in a redundant topology erroneously transitions to the
forwarding state. This usually happens because one of the ports of a physically redundant topology (not
necessarily the STP blocking port) no longer receives STP BPDUs. In its operation, STP relies on continuous
reception or transmission of BPDUs based on the port role. The designated port transmits BPDUs, and the
non−designated port receives BPDUs.
When one of the ports in a physically redundant topology no longer receives BPDUs, the STP conceives that
the topology is loop free. Eventually, the blocking port from the alternate or backup port becomes designated
and moves to a forwarding state. This situation creates a loop.
The loop guard feature makes additional checks. If BPDUs are not received on a non−designated port, and
loop guard is enabled, that port is moved into the STP loop−inconsistent blocking state, instead of the
listening / learning / forwarding state. Without the loop guard feature, the port assumes the designated port
role. The port moves to the STP forwarding state and creates a loop. Refer to Spanning−Tree Protocol
Enhancements using Loop Guard and BPDU Skew Detection Features for more information on the loop guard
feature.
This document covers reasons that STP can fail, what information to look for to identify the source of the
problem, and what kind of design minimizes STP risks.
Loops can also be caused by a uni−directional link. For more information, refer to the UDLD: One−Way link
problems section of this document.
A broken fiber cable or other cabling/port issues can cause this one−way only communication. These partially
functional links can cause problems such as STP loops when the switches involved do not know that the link
is partially broken. UDLD can put a port in errdisable state when it detects a unidirectional link. The
command udld aggressive−mode can be configured on switches that run CatOS and Cisco IOS (check release
notes for command availability) for point−to−point connections between switches where malfunctioning links
cannot be tolerated. The use of this feature can help you identify difficult to find unidirectional link problems
Refer to Understanding and Configuring the Unidirectional Link Detection Protocol (UDLD) Feature for
configuration information on UDLD.
For CatOS, use the show mac command for the module and port or the entire module to look at out−discards:
For Cisco IOS, use the show interfaces counters error command.
Your network can send too many packets through this port for the port to handle at its current speed/duplex
setting. This can happen where you have multiple high−speed ports flowing to a single (usually slower) port.
You can move the device that hangs off this port to faster media. For example, if the port is 10 Mbps, move
this device to a 100 Mbps or Gigabit port. You can change the topology to route frames differently.
If the segment is shared, other devices on this segment can transmit so much that the switch has no
opportunity to transmit. Avoid daisy−chained hubs whenever possible. Congestion can lead to packet loss.
Packet loss causes retransmissions at the transport layer which in turn causes users to experience latency at the
application level. You can upgrade10Mbps links to 100Mbps or Gigabit Ethernet links when possible. You
can remove some devices from crowded segments to other less populated segments. Make congestion
avoidance a priority on your network.
Applications
At times the traffic transmission characteristics of the applications used can lead to output buffer problems.
NFS file transfers that come from a Gigabit attached server that uses user datagram protocol (UDP) with a
32K window size is one example of an application setting that can bring out this type of problem. If you have
checked or tried the other suggestions in this document (checked speed/duplex, no physical errors on the link,
all the traffic is normal valid traffic, and so on), then reducing the unit size that is sent by the application can
help alleviate this problem.
Software Problems
If you see behavior that can only be considered "strange," you can isolate the behavior to a specific box, and
you have looked at everything suggested so far, this can indicate software or hardware problems. It is usually
easier to upgrade the software than it is to upgrade hardware. Change the software first.
For CatOS, use the show version command to verify the current software version and free flash memory for
the upgrade.
For Cisco IOS, use the show version command to verify the current software version along with the dir flash:
or dir bootflash: (dependent upon the platform) command to verify the available flash memory for the
upgrade:
Router#sh ver
Cisco Internetwork Operating System Software
IOS (tm) Catalyst 4000 L3 Switch Software (cat4000−IS−M), Version 12.1(13)EW, EA
RLY DEPLOYMENT RELEASE SOFTWARE (fc1)
TAC Support: http://www.cisco.com/tac
Copyright (c) 1986−2002 by cisco Systems, Inc.
Compiled Fri 20−Dec−02 13:52 by eaarmas
Image text−base: 0x00000000, data−base: 0x00E638AC
ROM: 12.1(12r)EW
Dagobah Revision 71, Swamp Revision 24
trunk−4500 uptime is 2 weeks, 2 days, 6 hours, 27 minutes
System returned to ROM by redundancy reset
System image file is "bootflash:cat4000−is−mz.121−13.EW.bin"
!−−− Typical Cisco IOS show version output.
Router#dir bootflash:
Directory of bootflash:/
1 −rw− 8620144 Mar 22 2002 08:26:21 cat4000−is−mz.121−13.EW.bin
61341696 bytes total (52721424 bytes free)
!−−− Verify available flash memory on switch running Cisco IOS.
Router
For information on upgrading software for Catalyst switches, choose your platform under LAN & ATM
Switches and look at the Software Configuration > Software Upgrade and Working With Configuration Files
section.
There can be a situation where the software is not compatible with the hardware. This happens when new
hardware comes out and requires special support from the software. For more information on software
compatibility, use the Software Advisor tool.
Software Bugs
The operating system can have a bug. If you load a newer software version, it can often fix this. You can
search known software bugs with the Software Bug Toolkit.
Corrupt Images
An image can have become corrupted or is missing. For information in regard to the recovery from corrupted
images, choose your platform under LAN & ATM Switches and look at the Troubleshooting > Recovery from
Corrupted or Missing Software section.
Hardware Problems
Check the results of show module for Catalyst 6000 and 4000 series switches that run CatOS or Cisco IOS.
Check the results of the POST results from the switch to see if there were any failures indicated for any part of
the switch. Failures of any test of a module or port show an 'F' in the test results.
For CatOS, use the show test command to see all test results. In order to see test results per module, use the
show test {mod} command:
For Cisco IOS, on modular switches like the Cat6000 and 4000, use the command show diagnostics. In order
to see POST results per module, use the show diagnostics module {mod} command.
!−−− If these steps fail, open a case with Cisco Technical Support.
Note: For Catalyst 3750, 3550, 2970 , 2950/2955, and 2900/3500XL Series switches use the show post
command, which indicates a simple pass or fail for the hw status. Use the LEDs on these switches to help you
understand the POST results. Refer to Understanding Post Results.
For further information on troubleshooting hardware problems on Catalyst switches that run CatOS and Cisco
IOS, go to the LAN and ATM Switches support pages, choose your platform and look at the Troubleshooting
> Hardware section.
For possible issues related to Field Notices, refer to Field Notices for LAN and ATM Switches.
Switch2(config)#int fa1/0/12
Switch2(config−if)#switchport mode access
or
Switch2(config)#int fa1/0/12
Switch2(config−if)#switchport trunk encapsulation dot1q
These types of errors increase rapidly, especially when the traffic that passes through that link is high or when
it has devices such as servers connected to that interface. This high load of traffic oversubscribes the ports,
which exhausts the input buffers and causes the Rx−No−Pkt−Buff counter and input errors to increase rapidly.
If a packet cannot be completely received because the switch is out of packet buffers, this counter is
incremented once for every dropped packet. This counter indicates the internal state of the Switching ASICs
on the Supervisor and does not necessarily indicate an error condition.
Pause Frames
When the receive part (Rx) of the port has its Rx FIFO queue filled and reaches the high water mark, the
transmit part (Tx) of the port starts to generate pause frames with an interval value mentioned in it. The
remote device is expected to stop / reduce the transmission of packets for the interval time mentioned in the
pause frame.
If the Rx is able to clear the Rx queue or reach low water mark within this interval, Tx sends out a special
pause frame that mentions the interval as zero (0x0). This enables the remote device to start to transmit
packets.
If the Rx still works on the queue, once the interval time expires, the Tx sends a new pause frame again with a
new interval value.
If Rx−No−Pkt−Buff is zero or does not increment and the TxPauseFrames counter increments, it indicates
that our switch generates pause frames and the remote end obeys, hence Rx FIFO queue depletes.
If Rx−No−Pkt−Buff increments and TxPauseFrames also increments, it means that the remote end disregards
the pause frames (does not support flow control) and continues to send traffic despite the pause frames. In
order to overcome this situation, manually configure the speed and duplex, as well as disable the flow control,
if required.
These types of errors on the interface are related to a traffic problem with the ports oversubscribed. The
WS−X4448−GB−RJ45, WS−X4548−GB−RJ45, and WS−X4548−GB−RJ45V switching modules have 48
oversubscribed ports in six groups of eight ports each:
• Ports 1, 2, 3, 4, 5, 6, 7, 8
• Ports 9, 10, 11, 12, 13, 14, 15, 16
• Ports 17, 18, 19, 20, 21, 22, 23, 24
• Ports 25, 26, 27, 28, 29, 30, 31, 32
• Ports 33, 34, 35, 36, 37, 38, 39, 40
• Ports 41, 42, 43, 44, 45, 46, 47, 48
The eight ports within each group use common circuitry that effectively multiplexes the group into a single,
nonblocking, full−duplex Gigabit Ethernet connection to the internal switch fabric. For each group of eight
ports, the frames that are received are buffered and sent to the common Gigabit Ethernet link to the internal
switch fabric. If the amount of data received for a port begins to exceed buffer capacity, flow control sends
pause frames to the remote port to temporarily stop traffic and prevent frame loss.
If the frames received on any group exceeds the bandwidth of 1 Gbps, the device starts to drop the frames.
These drops are not obvious as they are dropped at the internal ASIC rather than the actual interfaces. This
can lead to slow throughput of packets across the device.
The Rx−No−Pkt−Buff does not depend on the total traffic rate. It depends on the amount of the packets that
are stored in the Rx FIFO buffer of the module ASIC. The size of this buffer is only 16 KB. It is counted with
short bursty traffic flows when some packets fill this buffer. Thus, Rx−No−Pkt−Buff on each port can be
counted when the total traffic rate of this ASIC port group exceeds 1 Gbps, since WS−X4548−GB−RJ45 is
8:1 oversubscribed module.
When you have devices that need to carry a large amount of traffic through that interface, consider the use of
one port of each group so that the common circuitry that shares a single group is not affected by this amount
of traffic. When the Gigabit Ethernet switching module is not fully utilized, you can connect balancing port
connections across port groupings to maximize available bandwidth. For example, with the
WS−X4448−GB−RJ45 10/100/1000 switching module, you can connect ports from different groups, such as
ports 4, 12, 20, or 30 (in any order), before you connect ports from the same group, such as ports 1, 2, 3, 4, 5,
6, 7, and 8.
If this does not solve the issue, you need to consider a module without any oversubscription of ports.
1. Make sure Cisco Discovery Protocol (CDP) runs between the switch and router and both can see each
other.
2. Disable the Keepalives on the interface of the router.
3. Reconfigure the trunk encapsulation on both devices.
When the keepalives are disabled, the CDP enables link to operate normally.
In order to resolve this issue, create trunk ports instead of subinterfaces. That way, the VLAN can be seen in
all interfaces.
Troubleshooting rxTotalDrops
If all other counters are zero, and the only error counter that reports errors is rxTotalDrops, the most likely
cause is that the Spanning Tree blocks one or more VLANs on the uplink port, so the Color Blocking Logic
(CBL) drops.
32 bit counters
0 rxCRCAlignErrors = 0
1 rxUndersizedPkts = 0
2 rxOversizedPkts = 0
3 rxFragmentPkts = 0
4 rxJabbers = 0
5 txCollisions = 0
6 ifInErrors = 0
7 ifOutErrors = 0
8 ifInDiscards = 0
9 ifInUnknownProtos = 0
10 ifOutDiscards = 98
11 txDelayExceededDiscards = 0
12 txCRC = 0
13 linkChange = 1
14 wrongEncapFrames = 0
0 dot3StatsAlignmentErrors = 0
1 dot3StatsFCSErrors = 0
2 dot3StatsSingleColFrames = 0
3 dot3StatsMultiColFrames = 0
4 dot3StatsSQETestErrors = 0
5 dot3StatsDeferredTransmisions = 0
6 dot3StatsLateCollisions = 0
7 dot3StatsExcessiveCollisions = 0
8 dot3StatsInternalMacTransmitErrors = 0
9 dot3StatsCarrierSenseErrors = 0
10 dot3StatsFrameTooLongs = 0
11 dot3StatsInternalMacReceiveErrors = 0
12 dot3StatsSymbolErrors = 0
0 txPause = 0
1 rxPause = 0
0 rxTotalDrops = 253428855
1 rxFIFOFull = 0
2 rxBadCode = 0
Last−Time−Cleared
−−−−−−−−−−−−−−−−−−−−−−−−−−
Sat Oct 27 2007, 08:24:35
6509> (enable)
When the port blocks VLANs on one side, but the remote side forwards on those VLANs, the interface
increments the rxTotalDrops counters.
Compare the VLANs allowed in the trunk on both sides of the link. Also verify the spanning tree state for
these allowed VLANs on both sides. BPDUs are still sent on actively configured VLAN, so switch A sends
BPDUs on all configured and forwarding ports, but switch B drops them since it does not have those VLANs
configured. In other words, switch B gets packets for VLANs for which it is not configured, so it simply drops
them. These are not really errors but simple misconfiguration.
Related Information
• Troubleshooting Cisco Catalyst Switches to NIC Compatibility Issues
• Using PortFast and Other Commands to Fix Workstation Startup Connectivity Delays
• Configuring and Troubleshooting Ethernet 10/100/1000Mb Half/Full Duplex Auto−Negotiation
• Recovering From errDisable Port State on the CatOS Platforms
• Upgrading Software Images and Working with Configuration Files on Catalyst Switches
• Recovering Catalyst Switches Running CatOS from Booting Failures
• Recovery From Corrupt or Missing Software Image on Cisco Catalyst 2900XL and 3500XL
Series Switches
• Technical Support & Documentation − Cisco Systems