High CPU utilization on WS-C4507R

[shanfrancis]

Hi,

Seeing CPU uilization at 90% very often , initial analysis done as per the Support forums output but no where I got the solution for our problem. Please see the output provided. We see the out put of CPU where Cat4k Mgmt LoPri is more and under the Platform health K2L2 Address Table R   (%CPU Target) 2.00  (%CPU Actual)39.10

CPU utilization for five seconds: 79%/1%; one minute: 79%; five minutes: 73%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  55     2628422    492155       5340 49.43% 44.50% 45.20%   0 Cat4k Mgmt LoPri
  54      645851   1144987        564 13.03% 13.65% 13.44%   0 Cat4k Mgmt HiPri
  64       22786      6755       3373  8.71% 12.06%  5.63%   1 SSH Process
 123      131150     97235       1348  2.55%  2.84%  2.86%   0 Spanning Tree
  39      106872     12972       8238  1.59%  1.58%  1.57%   0 IDB Work
 116       28347    110646        256  0.71%  0.69%  0.50%   0 IP Input
  98        9879     70313        140  0.39%  0.34%  0.32%   0 UDLD
  14       23227     41132        564  0.31%  0.32%  0.32%   0 ARP Input
 248        7807     55771        139  0.23%  0.20%  0.21%   0 EIGRP-IPv4 Hello

#sh platform health

K2L2 Address Table R   2.00  39.10     12      5  100  500   55  52   27  39:06

Please help here.... Attached Tech support for ref.

[Arumugam Muthaiah · Accepted solution]

Hi Shanfrancis,

• The "K2L2 Address Table R" process is related to L2 entries and adjacency updates. The learning of MAC addresses on this box is done in software, so if we keep on learning too many mac addresses you will see a spike of CPU on this process. This may happen in the following conditions:

• Constant receipt of spanning tree topology change BPDUs: on PVST that reduces cam aging and you will see entries being deleted too fast, on rapid-PVST/MST this flushes the entries. Overall a number of entries may need to be deleted/added again. Make sure you have "switchport host" configured on ports which directly connect to end devices. This will avoid TCN generation when a workstation reboots.

• Mac-address moves: where a user will move from one port to another or a mac address may be looping in the network.

• I verified the capture and note seeing more MAC utilization as well

HW: WS-C4507R
SW: cat4500-ipbasek9-mz.122-54.SG1.bin

------------------ show mac-address-table count ------------------

MAC Entries for all vlans:
Dynamic Unicast Address Count:                  458
Static Unicast Address (User-defined) Count:    0
Static Unicast Address (System-defined) Count:  21
Total Unicast MAC Addresses In Use:             479
Total Unicast MAC Addresses Available:          32768
Multicast MAC Address Count:                    98
Total Multicast MAC Addresses Available:        32768

Next Action:

I would suggest you to enable the MAC address move notification on the switch which helps to identify us do we see any MAC flap

Switch(config)#mac address-table notification mac-move
Switch#show mac address-table notification mac-move

MAC Move Notification:  enabled

Switch#show log
Switch#show spanning detail | in ieee|occur|from
Switch#show platform health | inc K2L2 Address
Switch#show platform cpu packet statistics all
Switch#show process cpu history

If none of the above helping to narrow down the issue, we need to run the below debug to investigate further
Switch#debug platform packet all receive buffer
Switch#show platform cpu packet buffered

Regards,

Aru

[shanfrancis]

Hi Aru,

Thank You for the Suggestion , will follow the fix suggested and will come back to you

[shanfrancis]

Hi Aru,

This really helped mitigating the issue.

Switch(config)#mac address-table notification mac-move
Switch#show mac address-table notification mac-move

We found Tape server creating Flaps... , removed the same from NW and now the CPU utilization is 30 %

Thanks a Lot for the Solution !!!

[shanfrancis]

Hi Aru,

We see the CPU utilization again going high, but the Mac move command didnt help this time. Please help here and the logs are shared.

CPU utilization for five seconds: 61%/5%; one minute: 70%; five minutes: 68%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  55   7997840231229082236        650 33.83% 43.40% 40.79%   0 Cat4k Mgmt LoPri
  54  22917106732021706016       1133  7.19%  7.47%  7.73%   0 Cat4k Mgmt HiPri
 111  1403195490 360579353       3891  5.27%  5.12%  5.46%   0 IP Input
  14   6866746791181031593        581  3.11%  3.16%  3.12%   0 ARP Input
 154   4977106151168578510        425  1.51%  1.13%  1.15%   0 DHCPD Receive
 108   4314278321513124988        285  1.19%  0.94%  0.97%   0 DHCP Snooping

Output of platform health and captured few criterias where we see the values fo beyond the target

K2CpuMan Review       30.00  37.03     30    107  100  500   48  42   33  88777:43

Gigaport3 Review       0.40   0.08      4     68  100  500    0   0    0  239:19

Gigaport18 Review      0.40   0.40      4     51  100  500    0   0    0  893:03

[Arumugam Muthaiah]

Hi Shanfrancis,

Apologies for the late response. I can see the highest cpu utilization is IP Input Process, it means too many packets punt to cpu.

You can capture the below commands if you see high CPU again and share the detail

#show log

#show proc cpu sort | ex 0.00%

#show platform health

#show plat cpu packet statistics

#show interface | include line|\drops (3 times in gap of few min)

#debug platform packet all receive buffer – This command doesn’t impact the performance and will helpful to know the packet which pun to CPU
#show platform cpu packet buffered

Regards,

Aru