C4900M High CPU utilization and dropped packets

[breakwatersupport]

Recently we started seeing higher ping replies from a switch, which I realize in itself is not an indication of a problem, but this is the only switch that has a high count of "Tx Mode Drop" and "Sa Miss". I am unsure how to debug further and try to figure out where these packets are coming from and why they are being kicked up to the software and being dropped. I am hoping someone can make a suggestion for debug/troubleshooting techniques in order to figure out the cause of the CPU utilization.

I could not find any reference to what the reason "Tx Mode Drop" means, debugging is an issue because this only occurs during production hours, which consists mostly of multicast traffic. This switch is used as a core routing switch.

Cisco IOS Software, Catalyst 4500 L3 Switch Software, Version 12.2(50)SG2, RELEASE SOFTWARE (fc2)

WS-C4900M

#show platform cpu packet statistics

RkiosSysPacketMan:

Packet allocation failures: 0

Packet Buffer(Software Common) allocation failures: 0

Packet Buffer(Software ESMP) allocation failures: 0

Packet Buffer(Software EOBC) allocation failures: 0

Packet Buffer(Software SupToSup) allocation failures: 0

IOS Packet Buffer Wrapper allocation failures: 0

Packets Dropped In Processing Overall

Total                5 sec avg 1 min avg 5 min avg 1 hour avg

-------------------- --------- --------- --------- ----------

          4041851749      2083      2211      1888       1890

Packets Dropped In Processing by CPU event

Event             Total                5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Sa Miss                     3016958566      2033      2160      1842       1844

Input Acl Fwd                    37574         0         0         0          0

Input ACl Copy              1024781409        49        45        42         36

Sw Packet for Bridge                74200         0         0         0          0

Packets Dropped In Processing by Priority

Priority          Total                5 sec avg 1 min avg 5 min avg 1 hour avg

----------------- -------------------- --------- --------- --------- ----------

Normal                      1024789544        49        45        42         36

Medium                      3017031879      2033      2160      1842       1844

High                             30326         0         0         0          0

Packets Dropped In Processing by Reason

Reason             Total                5 sec avg 1 min avg 5 min avg 1 hour avg

------------------ -------------------- --------- --------- --------- ----------

SrcAddrTableFilt                     21         0         0         0          0

STPDrop                              69         0         0         0          0

NoDstPorts                       111771         0         0         0          0

Tx Mode Drop                 4041739888      2083      2211      1888       1890

Total packet queues 64

[deyadav]

SA Miss = This CPU event, is caused when a new MAC is seen on the port, and it is thus, a copy of such packet is forwarded to CPU for MAC learning.

TxModeDrop = This copy of packet is then dropped by CPU once it has been processed, and the counter increments. So, when a packet encounters SA miss event, a copy comes to cpu and it is dropped as a TxModeDrop.

Generally the cause of too much SA miss, is MAC flapping, caused either due to network loop or server with improper NIC binding, resulting in MAC's seen on different port of the switch, and causing SA Miss event.

Can, you check logs and see if any MAC flapping logs are observed during that period. I am not 100% sure, but it is by default enabled on 4500, if not then configure this command to find details of such frames:

"mac-address-table notification mac-move"

You may try to Span packets hitting CPU, using this procedure:

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml#tool1

monitor session 1 source cpu queue unknown-sa

monitor session 1 destination interface

Run the capture when issue persists.

Or try tool no. 2:

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml#tool2

This debug is safe as no messages are printed, and show command is used to check them.

HTH to locate the culprit.

Warm Regards,

Deepak Yadav