High CPU on Cisco 4948E switch

[wipronitin]

Hello,

We need help to troubleshoot high CPU on Cisco 4948E switch. IOS image is cat4500e-lanbase-mz.122-54.SG1.bin

Tried finding the root cause using Cisco documentation for High CPU on 4948 switches but unable to find anythin specific.

Below is the output of show cpu | ex 0.00 command.

CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

  55   4533312041368514536        331  4.95%  4.84%  4.83%   0 Cat4k Mgmt HiPri

  56   414621502 831220970        498 93.44% 93.64% 93.71%   0 Cat4k Mgmt LoPri

  92       98582 185950416          0  0.07%  0.03%  0.02%   0 UDLD            

116    63326430  57579164       1099  0.63%  0.60%  0.59%   0 Spanning Tree

The show platform health command shows K5 L2 Hardware Address process high actual cpu as well, while CpuMan Review seems to be normal.

                                  %CPU   %CPU    RunTimeMax   Priority       Average %CPU   Total

                                   Target Actual      Target  Actual   Fg   Bg      5Sec Min Hour   CPU      

K5 L2 Hardware Addre   2.00    92.88         20      10       100  500      126 118   93     3615:30

K5CpuMan Review       30.00    6.68           30      7         100  500        7   6      5      5362:24

We have checked the switch logs, no relevant logs available.

As far as toploogy is concerned, this switch is connected to another 4948E switch via etherchannel. Both switches are also connected to seperate firewalls & routers for redundancy.

Traffic on switch interface is normal & MAC addresses learnt are less than 10 per switch. No MAC flapping logs seen on switch.

Let me know if any other details are required.

[InayathUlla Sharieff · Accepted solution]

Nitin,

Have you gone through the link mentioned below:

http://www.cisco.com/en/US/customer/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml

Ref:

https://supportforums.cisco.com/message/628786#628786

'Cat4k Mgmt LoPri' schedules all cat4k platform-dependent processes, which

exceed the CPU target utilization.

You can see target and actual CPU utilization in the output of 'show platform health'

'Cat4k Mgmt HiPri' manages processes which are staying under CPU target

utilization. Misbehaving processes are handled by 'Cat4k Mgmt LoPri'.

NOTE: The common misconception is that high CPU utilization indicates the

depletion of resources on a device and the threat of a crash. A capacity issue is one of

the symptoms of high CPU utilization on Cisco IOS routers. However, a capacity issue is

almost never a symptom of high CPU utilization with hardware-based forwarding switches like

the Catalyst 4500. The Catalyst 4500 is designed to forward packets in the hardware

application-specific integrated circuit (ASIC) and reach traffic-forwarding

speeds of up to 102 million packets per second (Mpps).

>>These two processes aggregate multiple platform-specific processes which

perform the essential management functions on the switch. These

processes process control plane as well as data packets that need to be

software-switched or processed.

Please follow the steps mentioned in the above link to findout the root cause. In case if you face any difficulty do let me know .

HTH

Regards

Inayath.

*Plz rate usefull posts.

[wipronitin]

Hello Inayath,

Thanks for your inputs.

I have already tried troubleshooting the issue using the document you provided but found nothing specific to the outputs I am getting.

Only K5 L2 Hardware Address is showing high cpu rest all are OK. Not too sure how to further troubleshoot this to find the exact root cause.

One more thing I want to highlight is that the CPU usage is showing high on both 4948E switches interconnected via etherchannel. The high CPU is observed since last 3 days but no changes have been made on the switches recently.

Need your help to find out the exact root cause so that this can be corrected.

Regards.

[InayathUlla Sharieff · Accepted solution]

Nitin,

I see that the process causing the high CPU specifically is K5 L2 Hardware Address. This process is associated with MAC
address learning.


Please send me the following outputs:

.         Network topology

.         Show platform cpu packet statistics all
 

Also we need to find out why mac address table is updated so frequently
(excessive mac moving/re-learning). One way is enable mac notification:
 
MAC notification can be enabled the following
 
global config lines, using the maximum buffer size:
 
--------------------------------------------------
 
.         mac address-table notification change
.         mac address-table notification mac-move
.         mac address-table notification change history-size 500
.         mac address-table notification change interval 10
 
Please configure this, wait 30 seconds or so (the switch may generate
logging messages if there are MAC flaps also), then grab 'sh mac
address-table notification change'
 
Thank you in advance.

Regards

Inayath

[InayathUlla Sharieff]

Did you debug the packets?

debug platform packet all receive buffer (Captures all packets sent to CPU in a circular buffer)

show platform CPU packet buffered

If yes please provide me above outputs?

Regards

Inayath

[wipronitin]

Hello Inayath,

I am really sorry as I cannot debug packets since the switch is in production carrying customer traffic.

However, seems like I have got a possible cause which might have caused this spike in CPU.

Although there are no mac flaps seen in the switch logs. However, as per your last comment that this might be related to excessive mac moving/re-learning. I checked the MAC address table of the switch several times in a minute & found that one MAC address was indeed moving on different switch interfaces for a particular VLAN.

This MAC address belong to VRRP type & thus I checked the routers connected on both the switches. I found that the sub-interface configuration for that particular VLAN on the router has been misconfigured. There are logs on routers for dupilcate IP address on that particular subinterface, subinterfaces on both routers have been configured with same physical IP address.

This sub-interface were created on the routers about the same time when the CPU spiked up on the switches.

We have asked the team who manage the router to correct the VRRP & will then check the switch CPU. By all means it shoud be back to normal again.

I will keep you updated about the same. Many thanks for your immense help on this.

Regards.

[InayathUlla Sharieff]

Nitin,

Thanks for the update. Yes that was the culprit.

Anyways hope that resolve kindly keep us updated. Also feel free to ask in case any further help is required.

HTH

Regards

Inayath

****************Plz rate all usefull posts.******************

[wipronitin]

Hello All,

For enveryone's information, I confirm that the high CPU usage issue faced by me has been resolved.

Root cause was wrong VRRP configuration on connected routers due to which Duplicate IP error messages were generating on Router. Also, on switch we had excessive MAC moving/re-learning causing the issue.

Same was corrected on rotuers which resolved the issue.

Once again many thanks to Inayath for pointing me towards the right direction.

Regards.