I've already read this

Psycho_naT · ‎03-29-2017

I have WS-C4507R with WS-X4515 Supervisor, and WS-C4507R with WS-X45-SUP6-E Supervisor. Same network topology, same traffic rates, same IOS version: Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500-ENTSERVICESK9-M), Version 12.2(53)SG3, RELEASE SOFTWARE (fc1).

show proccess cpu history

WS-X4515 Supervisor

100 *
90 * * * * * * * * * * * * * * * * * * *** **
80 * * * * * * * * * * * * * * * * * * *** **
70 * * * * * * * * * * * * * * * * * * *** **
60 * * * * * * * * * * * * * * *** * * *****#
50 * * * * * ***** * ***** * ***** * ******##
40 #*#*#*#*#*#*#*#*#*#*#*#*#*#*###*#*#*######
30 ##########################################
20 ##########################################
10 ##########################################

CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

WS-X45-SUP6-E Supervisor

100
90
80
70
60 *
50 * * ******* **** *** *** * * * * *** * * * *
40 **********************************************************
30 **********************************************************
20 ##########################################################
10 ##########################################################

CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

show platform cpu packet statistics all | i Esmp

WS-X4515 Supervisor

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------

EsmpNoTxInfo         0            0        0         0         0
Esmp                  462652         204       182       156         65
Esmp                        0            0            0          0             0

WS-X45-SUP6-E Supervisor

EsmpNoTxInfo         0            0            0        0         0
Esmp                        0          0            0        0         0
Esmp                        0            0        0         0         0

In peak CPU rates i see:

WS-X4515 Supervisor

show platform health | i K2L2
K2L2 Address Table R 2.00 68.38 12 5 100 500 7 6 2 3:17

WS-X45-SUP6-E Supervisor

show platform health | i K5 L2
K5 L2 Hardware Addre 2.00 36.34 20 14 100 500 1 2 2 6036:44

What is the reason of high rate process K2/K5 ?

Why WS-X4515 Supervisior have high rate in ESMP Queue ? (Maybe this is a root cause of CPU utilization?)

This situation presents on other WS-X4515 Supervisor in network with same topology.

Reza Sharifi · ‎03-29-2017

Can you post the output of "sh processes cpu | exclude 0.00" from both switches?

Psycho_naT · ‎03-29-2017

Usual state (normal state)

WS-X4515 Supervisior

show platform health | i K2L2
K2L2 Address Table R   2.00   0.00     12      5 100 500    2   7    4 7:04
K2L2 New Static Addr   2.00   0.00     10      5 100 500    0   0    0 0:10
K2L2 New Multicast A   2.00   0.00     10      5 100 500    0   0    0 0:00
K2L2 Dynamic Address   2.00   0.00     10      0 100 500    0   0    0 0:00
K2L2 Vlan Table Revi   2.00   0.00     12      8 100 500    0   0    0 0:01

show processes cpu | ex 0.00
CPU utilization for five seconds: 28%/0%; one minute: 34%; five minutes: 34%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
14        6536     17651        370 0.07% 0.07% 0.07%   0 ARP Input
36       74140     12734       5822 1.11% 1.11% 1.11%   0 IDB Work
51      727340   1190440        610 17.91% 17.88% 18.08%   0 Cat4k Mgmt HiPri
52      703808    233054       3019 7.03% 11.58% 11.87%   0 Cat4k Mgmt LoPri
59        1312      1779        737 0.07% 0.05% 0.07%   0 Compute load avg
110       26964     25588       1053 0.39% 0.53% 0.52%   0 Spanning Tree
119        1300     37045         35 0.07% 0.06% 0.07%   0 TCP Timer
144        1824     13898        131 0.07% 0.02% 0.01%   0 DHCPD Receive
146        1108     10889        101 0.07% 0.04% 0.05%   0 CEF: IPv4 proces
170        1648     50947         32 0.15% 0.05% 0.05%   0 PM Callback
221        5640    400191         14 0.15% 0.18% 0.16%   0 IP SLAs Responde

WS-X45-SUP6-E Supervisor

show platform health | i K5 L2
K5 L2 Aging Table Re   2.00   0.10     20      4 100 500    0   0    0 286:32
K5 L2 Unicast Addres   2.00   0.00     20      4 100 500    0   0    0 62:49
K5 L2 Multicast Addr   2.00   0.00     20      0 100 500    0   0    0 0:08
K5 L2 Hardware Addre   2.00   0.00     20     14 100 500    0   2    2 6038:25
K5 L2 Hardware Mac L   1.00   0.00      2      0 100 500    0   0    0 0:00

show processes cpu | ex 0.00
CPU utilization for five seconds: 17%/0%; one minute: 20%; five minutes: 20%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
38     8402612 13983424        600 0.07% 0.10% 0.09%   0 IDB Work
53   8357815361526515953        547 9.43% 9.41% 9.50%   0 Cat4k Mgmt HiPri
54 1249257212 732864962       1704 6.23% 9.90% 10.31%   0 Cat4k Mgmt LoPri
115    25601072 14978298       1709 0.15% 0.15% 0.15%   0 Spanning Tree
180         200       148       1351 0.23% 0.24% 0.05%   1 SSH Process

Unusual state (abnormal state)

WS-X4515 Supervisior

show platform health | i K2L2
K2L2 Address Table R   2.00 50.32     12      5 100 500    8   6    4 6:55
K2L2 New Static Addr   2.00   1.18     10      5 100 500    0   0    0 0:10
K2L2 New Multicast A   2.00   0.00     10      5 100 500    0   0    0 0:00
K2L2 Dynamic Address   2.00   0.00     10      0 100 500    0   0    0 0:00
K2L2 Vlan Table Revi   2.00   0.00     12      8 100 500    0   0    0 0:01

show processes cpu | ex 0.00
CPU utilization for five seconds: 96%/1%; one minute: 38%; five minutes: 35%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
14        6684     18101        369 0.23% 0.08% 0.08%   0 ARP Input
36       76112     13080       5818 1.11% 1.11% 1.11%   0 IDB Work
51      746320   1226120        608 18.60% 18.38% 18.21%   0 Cat4k Mgmt HiPri
52      723932    239808       3018 70.76% 15.21% 12.41%   0 Cat4k Mgmt LoPri
59        1356      1822        744 0.07% 0.05% 0.07%   0 Compute load avg
82       19340      9902       1953 3.19% 0.45% 0.28%   0 Port-Security
104        4816     21446        224 0.07% 0.03% 0.02%   0 IP Input
110       27808     26046       1067 0.39% 0.52% 0.52%   0 Spanning Tree
119        1328     37668         35 0.07% 0.06% 0.07%   0 TCP Timer
217        5612     35286        159 0.23% 0.22% 0.19%   0 LLDP Protocol
221        5732    411308         13 0.15% 0.19% 0.16%   0 IP SLAs Responde

WS-X45-SUP6-E Supervisor

show processes cpu | ex 0.00
CPU utilization for five seconds: 40%/0%; one minute: 21%; five minutes: 20%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   7    12131488   1889767       6419 0.87% 0.12% 0.07%   0 Check heaps
38     8402696 13983564        600 0.15% 0.10% 0.10%   0 IDB Work
53   8357900761526532409        547 10.07% 9.77% 9.62%   0 Cat4k Mgmt HiPri
54 1249270912 732873117       1704 28.31% 10.47% 10.34%   0 Cat4k Mgmt LoPri
87     5730484 10681034        536 0.23% 0.04% 0.05%   0 Port-Security
115    25601384 14978446       1709 0.15% 0.15% 0.15%   0 Spanning Tree
224     2227224 69492087         32 0.07% 0.06% 0.07%   0 LLDP Protocol

show platform health | i K5 L2
K5 L2 Aging Table Re   2.00   0.12     20      4 100 500    0   0    0 286:32
K5 L2 Unicast Addres   2.00   0.74     20      4 100 500    0   0    0 62:49
K5 L2 Multicast Addr   2.00   0.00     20      0 100 500    0   0    0 0:08
K5 L2 Hardware Addre   2.00 40.12     20     14 100 500   10   3    2 6038:23
K5 L2 Hardware Mac L   1.00   0.00      2      0 100 500    0   0    0 0:00

WS-X4515 Supervisior - usual state scare me, because it works in 30-35% CPU utilization.

WS-X45-SUP6-E Supervisor - usual state 15-20% CPU utilization, this is better and good.

Reza Sharifi · ‎03-29-2017

It probably is a good idea to open a ticket with TAC and have them look at their data base for any bug related to this sup. The 2 Mgmt processes are consuming almost 31% of the total CPU usage.

show processes cpu | ex 0.00
CPU utilization for five seconds: 96%/1%; one minute: 38%; five minutes: 35%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
14        6684     18101        369 0.23% 0.08% 0.08%   0 ARP Input
36       76112     13080       5818 1.11% 1.11% 1.11%   0 IDB Work
51      746320   1226120        608 18.60% 18.38% 18.21%   0 Cat4k Mgmt HiPri
52      723932    239808       3018 70.76% 15.21% 12.41%   0 Cat4k Mgmt LoPri

Also, have a look at this doc:

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-4000-series-switches/65591-cat4500-high-cpu.html

HTH

Psycho_naT · ‎03-31-2017

I've already read this document, and do not find any usable information about my problem.

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-4000-series-switches/65591-cat4500-high-cpu.html

Any other ideas ?

I'm very interesting about this situation:

show platform cpu packet statistics all | i Esmp

WS-X4515 Supervisor

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------

EsmpNoTxInfo         0            0        0         0         0
Esmp                  462652         204       182       156         65
Esmp                        0            0            0          0             0

WS-X45-SUP6-E Supervisor

EsmpNoTxInfo         0            0            0        0         0
Esmp                        0          0            0        0         0
Esmp                        0            0        0         0         0

Maybe this is a root cause ? Why so many packets in ESMP queue ?

CPU utilization on WS-C4507R