CPU utilization on WS-C4507R

[Psycho_naT]

I have WS-C4507R with WS-X4515 Supervisor, and WS-C4507R with WS-X45-SUP6-E Supervisor. Same network topology, same traffic rates, same IOS version: Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500-ENTSERVICESK9-M), Version 12.2(53)SG3, RELEASE SOFTWARE (fc1).

show proccess cpu history

WS-X4515 Supervisor

100                                          *
 90 * * * * * * * * * * * * * * * * * * *** **
 80 * * * * * * * * * * * * * * * * * * *** **
 70 * * * * * * * * * * * * * * * * * * *** **
 60 * * * * * * * * * * * * * * *** * * *****#
 50 * * * * * ***** * ***** * ***** * ******##
 40 #*#*#*#*#*#*#*#*#*#*#*#*#*#*###*#*#*######
 30 ##########################################
 20 ##########################################
 10 ##########################################

               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

WS-X45-SUP6-E Supervisor

100
 90
 80
 70
 60                                *
 50 * * *******  ****  ***       *** *     * * * *** * *   * *
 40 **********************************************************
 30 **********************************************************
 20 ##########################################################
 10 ##########################################################

               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

show platform cpu packet statistics all | i Esmp

WS-X4515 Supervisor

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------

EsmpNoTxInfo         0                0            0          0             0
Esmp                  462652         204       182       156         65
Esmp                        0                0            0          0             0

WS-X45-SUP6-E Supervisor

EsmpNoTxInfo         0                0            0          0             0
Esmp                        0                0            0          0             0
Esmp                        0                0            0          0             0

In peak CPU rates i see:

WS-X4515 Supervisor

show platform health | i K2L2
K2L2 Address Table R   2.00  68.38     12      5  100  500    7   6    2  3:17

WS-X45-SUP6-E Supervisor

show platform health | i K5 L2
K5 L2 Hardware Addre   2.00  36.34     20     14  100  500    1   2    2  6036:44

What is the reason of high rate process K2/K5 ?

Why WS-X4515 Supervisior have high rate in ESMP Queue ? (Maybe this is a root cause of CPU utilization?)

This situation presents on other WS-X4515 Supervisor in network with same topology.

[Reza Sharifi]

Can you post the output of "sh processes cpu | exclude 0.00" from both switches?

[Psycho_naT]

Usual state (normal state)

WS-X4515 Supervisior

show platform health | i K2L2
K2L2 Address Table R   2.00   0.00     12      5  100  500    2   7    4  7:04
K2L2 New Static Addr   2.00   0.00     10      5  100  500    0   0    0  0:10
K2L2 New Multicast A   2.00   0.00     10      5  100  500    0   0    0  0:00
K2L2 Dynamic Address   2.00   0.00     10      0  100  500    0   0    0  0:00
K2L2 Vlan Table Revi   2.00   0.00     12      8  100  500    0   0    0  0:01

show processes cpu | ex 0.00
CPU utilization for five seconds: 28%/0%; one minute: 34%; five minutes: 34%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  14        6536     17651        370  0.07%  0.07%  0.07%   0 ARP Input
  36       74140     12734       5822  1.11%  1.11%  1.11%   0 IDB Work
  51      727340   1190440        610 17.91% 17.88% 18.08%   0 Cat4k Mgmt HiPri
  52      703808    233054       3019  7.03% 11.58% 11.87%   0 Cat4k Mgmt LoPri
  59        1312      1779        737  0.07%  0.05%  0.07%   0 Compute load avg
 110       26964     25588       1053  0.39%  0.53%  0.52%   0 Spanning Tree
 119        1300     37045         35  0.07%  0.06%  0.07%   0 TCP Timer
 144        1824     13898        131  0.07%  0.02%  0.01%   0 DHCPD Receive
 146        1108     10889        101  0.07%  0.04%  0.05%   0 CEF: IPv4 proces
 170        1648     50947         32  0.15%  0.05%  0.05%   0 PM Callback
 221        5640    400191         14  0.15%  0.18%  0.16%   0 IP SLAs Responde

WS-X45-SUP6-E Supervisor

show platform health | i K5 L2
K5 L2 Aging Table Re   2.00   0.10     20      4  100  500    0   0    0  286:32
K5 L2 Unicast Addres   2.00   0.00     20      4  100  500    0   0    0  62:49
K5 L2 Multicast Addr   2.00   0.00     20      0  100  500    0   0    0  0:08
K5 L2 Hardware Addre   2.00   0.00     20     14  100  500    0   2    2  6038:25
K5 L2 Hardware Mac L   1.00   0.00      2      0  100  500    0   0    0  0:00

show processes cpu | ex 0.00
CPU utilization for five seconds: 17%/0%; one minute: 20%; five minutes: 20%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  38     8402612  13983424        600  0.07%  0.10%  0.09%   0 IDB Work
  53   8357815361526515953        547  9.43%  9.41%  9.50%   0 Cat4k Mgmt HiPri
  54  1249257212 732864962       1704  6.23%  9.90% 10.31%   0 Cat4k Mgmt LoPri
 115    25601072  14978298       1709  0.15%  0.15%  0.15%   0 Spanning Tree
 180         200       148       1351  0.23%  0.24%  0.05%   1 SSH Process

Unusual state (abnormal state)

WS-X4515 Supervisior

show platform health | i K2L2
K2L2 Address Table R   2.00  50.32     12      5  100  500    8   6    4  6:55
K2L2 New Static Addr   2.00   1.18     10      5  100  500    0   0    0  0:10
K2L2 New Multicast A   2.00   0.00     10      5  100  500    0   0    0  0:00
K2L2 Dynamic Address   2.00   0.00     10      0  100  500    0   0    0  0:00
K2L2 Vlan Table Revi   2.00   0.00     12      8  100  500    0   0    0  0:01

show processes cpu | ex 0.00
CPU utilization for five seconds: 96%/1%; one minute: 38%; five minutes: 35%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  14        6684     18101        369  0.23%  0.08%  0.08%   0 ARP Input
  36       76112     13080       5818  1.11%  1.11%  1.11%   0 IDB Work
  51      746320   1226120        608 18.60% 18.38% 18.21%   0 Cat4k Mgmt HiPri
  52      723932    239808       3018 70.76% 15.21% 12.41%   0 Cat4k Mgmt LoPri
  59        1356      1822        744  0.07%  0.05%  0.07%   0 Compute load avg
  82       19340      9902       1953  3.19%  0.45%  0.28%   0 Port-Security
 104        4816     21446        224  0.07%  0.03%  0.02%   0 IP Input
 110       27808     26046       1067  0.39%  0.52%  0.52%   0 Spanning Tree
 119        1328     37668         35  0.07%  0.06%  0.07%   0 TCP Timer
 217        5612     35286        159  0.23%  0.22%  0.19%   0 LLDP Protocol
 221        5732    411308         13  0.15%  0.19%  0.16%   0 IP SLAs Responde

WS-X45-SUP6-E Supervisor

show processes cpu | ex 0.00
CPU utilization for five seconds: 40%/0%; one minute: 21%; five minutes: 20%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   7    12131488   1889767       6419  0.87%  0.12%  0.07%   0 Check heaps
  38     8402696  13983564        600  0.15%  0.10%  0.10%   0 IDB Work
  53   8357900761526532409        547 10.07%  9.77%  9.62%   0 Cat4k Mgmt HiPri
  54  1249270912 732873117       1704 28.31% 10.47% 10.34%   0 Cat4k Mgmt LoPri
  87     5730484  10681034        536  0.23%  0.04%  0.05%   0 Port-Security
 115    25601384  14978446       1709  0.15%  0.15%  0.15%   0 Spanning Tree
 224     2227224  69492087         32  0.07%  0.06%  0.07%   0 LLDP Protocol

show platform health | i K5 L2
K5 L2 Aging Table Re   2.00   0.12     20      4  100  500    0   0    0  286:32
K5 L2 Unicast Addres   2.00   0.74     20      4  100  500    0   0    0  62:49
K5 L2 Multicast Addr   2.00   0.00     20      0  100  500    0   0    0  0:08
K5 L2 Hardware Addre   2.00  40.12     20     14  100  500   10   3    2  6038:23
K5 L2 Hardware Mac L   1.00   0.00      2      0  100  500    0   0    0  0:00

WS-X4515 Supervisior - usual state scare me, because it works in 30-35% CPU utilization.

WS-X45-SUP6-E Supervisor - usual state 15-20% CPU utilization, this is better and good.

[Reza Sharifi]

It probably is a good idea to open a ticket with TAC and have them look at their data base for any bug related to this sup. The 2 Mgmt processes are consuming almost 31% of the total CPU usage.

show processes cpu | ex 0.00
CPU utilization for five seconds: 96%/1%; one minute: 38%; five minutes: 35%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  14        6684     18101        369  0.23%  0.08%  0.08%   0 ARP Input
  36       76112     13080       5818  1.11%  1.11%  1.11%   0 IDB Work
  51      746320   1226120        608 18.60% 18.38% 18.21%   0 Cat4k Mgmt HiPri
  52      723932    239808       3018 70.76% 15.21% 12.41%   0 Cat4k Mgmt LoPri

Also, have a look at this doc:

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-4000-series-switches/65591-cat4500-high-cpu.html

HTH

[Psycho_naT]

I've already read this document, and do not find any usable information about my problem.

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-4000-series-switches/65591-cat4500-high-cpu.html

Any other ideas ?

I'm very interesting about this situation:

show platform cpu packet statistics all | i Esmp

WS-X4515 Supervisor

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------

EsmpNoTxInfo         0                0            0          0             0
Esmp                  462652         204       182       156         65
Esmp                        0                0            0          0             0

WS-X45-SUP6-E Supervisor

EsmpNoTxInfo         0                0            0          0             0
Esmp                        0                0            0          0             0
Esmp                        0                0            0          0             0

Maybe this is a root cause ? Why so many packets in ESMP queue ?