Re: CPU Utilization become high (CISCO 3750 layer-III)

ZAHIDHASEEB · ‎08-02-2021

We have one CISCO switch (model 3750 layer-III) installed on core layer on top of access layer switches. For last 10 days we notice(thru NPM tool) that its CPU utilization reaches to 90% after 24 hours gradually. So we need to restart switch every 24 hours. is it possible that we can see what process taking 90% of CPU resource

marce1000 · ‎08-03-2021

- FYI : https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

Leo Laohoo · ‎08-03-2021

Start with the basic. Post the complete output to the following commands:

sh version
sh proc cpu sort | ex 0.00

NOTE: Try to take the output of the 2nd command as-close-as-possible to when the CPU is high or spikes.

ZAHIDHASEEB · ‎08-03-2021

sh proc cpu sort | ex 0.00
CPU utilization for five seconds: 42%/14%; one minute: 45%; five minutes: 53%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
156 761337 20917 36398 12.95% 10.40% 10.74% 0 HL3U bkgrd proce
10 251248 476391 527 2.07% 2.54% 2.54% 0 ARP Input
198 110191 313547 351 1.43% 0.62% 0.65% 0 IP Input
330 150073 22072 6799 1.11% 1.04% 1.08% 0 CEF: IPv4 proces
215 201057 140348 1432 1.11% 1.21% 1.19% 0 Spanning Tree
94 328875 403056 815 0.79% 0.95% 1.70% 0 HLFM address lea
114 73912 14463 5110 0.63% 0.52% 0.55% 0 hpm counter proc
202 109587 304586 359 0.63% 0.99% 0.93% 0 ADJ resolve proc
78 112601 5822 19340 0.47% 0.58% 0.61% 0 Adjust Regions
326 45397 47234 961 0.31% 0.23% 0.25% 0 SNMP ENGINE
292 2671 27402 97 0.31% 0.29% 0.26% 0 EEM ED Syslog
72 6656 141797 46 0.31% 0.07% 0.08% 0 RedEarth Tx Mana
186 8834 4956 1782 0.15% 0.05% 0.04% 0 CDP Protocol
166 27223 2971 9162 0.15% 0.13% 0.15% 0 HQM Stack Proces
36 1894 7743 244 0.15% 0.18% 0.17% 0 Logger

sh version
Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 12.2(55)SE1, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Thu 02-Dec-10 06:08 by prod_rel_team
Image text-base: 0x00003000, data-base: 0x02800000

ROM: Bootstrap program is C3750E boot loader
BOOTLDR: C3750E Boot Loader (C3750E-HBOOT-M) Version 12.2(44r)SE3, RELEASE SOFTWARE (fc3)

COMPANYSWITCH uptime is 4 hours, 6 minutes
System returned to ROM by power-on
System restarted at 08:56:27 utc Tue Aug 3 2021
System image file is "flash:c3750e-universalk9-mz.122-55.SE1.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: ipbase
License Type: Permanent
Next reload license Level: ipbase

cisco WS-C3750E-24TD (PowerPC405) processor (revision M0) with 262144K bytes of memory.
Processor board ID FDO1525Y4L3
Last reset from power-on
15 Virtual Ethernet interfaces
1 FastEthernet interface
28 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 30:E4:DB:CB:AF:80
Motherboard assembly number : 73-11177-14
Motherboard serial number : FDO15261FHQ
Model revision number : M0
Motherboard revision number : C0
Model number : WS-C3750E-24TD-S
Daughterboard assembly number : 73-11426-02
Daughterboard serial number : FDO15251VQC
System serial number : FDO1525Y4L3
Top Assembly Part Number : 800-28924-04
Top Assembly Revision Number : E0
Version ID : V05
CLEI Code Number : COMCF00ARB
Daughterboard revision number : A0
Hardware Board Revision Number : 0x00

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 3 30 WS-C3750E-24TD 12.2(55)SE1 C3750E-UNIVERSALK9-M

Configuration register is 0xF

CPU UTILIZATION ACCORDING TO THE NPM GRAPH

52%

ZAHIDHASEEB · ‎08-03-2021

sh proc cpu sort | ex 0.00
CPU utilization for five seconds: 55%/30%; one minute: 59%; five minutes: 56%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
156 1158918 24632 47049 9.49% 10.20% 9.94% 0 HL3U bkgrd proce
10 345179 560926 615 2.25% 3.12% 2.66% 0 ARP Input
215 250684 167197 1499 1.35% 1.15% 1.13% 0 Spanning Tree
202 145274 359790 403 1.20% 0.99% 1.01% 0 ADJ resolve proc
330 199357 26235 7598 1.05% 1.08% 1.09% 0 CEF: IPv4 proces
198 130843 364101 359 0.90% 0.55% 0.57% 0 IP Input
94 537119 474119 1132 0.90% 3.34% 3.21% 0 HLFM address lea
78 140350 7016 20004 0.60% 0.58% 0.62% 0 Adjust Regions
326 56235 56147 1001 0.60% 0.27% 0.27% 0 SNMP ENGINE
114 98891 17405 5681 0.45% 0.59% 0.54% 0 hpm counter proc
325 26922 55755 482 0.45% 0.12% 0.11% 0 PDU DISPATCHER
36 7999 39095 204 0.45% 0.14% 0.15% 0 Logger
324 33443 110832 301 0.30% 0.16% 0.15% 0 IP SNMP
155 54208 482280 112 0.30% 0.48% 0.40% 0 Hulc LED Process
161 504 181 2784 0.30% 0.20% 0.09% 1 SSH Process
166 34051 3580 9511 0.15% 0.20% 0.16% 0 HQM Stack Proces

sh version
Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 12.2(55)SE1, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Thu 02-Dec-10 06:08 by prod_rel_team
Image text-base: 0x00003000, data-base: 0x02800000

ROM: Bootstrap program is C3750E boot loader
BOOTLDR: C3750E Boot Loader (C3750E-HBOOT-M) Version 12.2(44r)SE3, RELEASE SOFTWARE (fc3)

CORESWITCH uptime is 5 hours, 0 minutes
System returned to ROM by power-on
System restarted at 08:56:27 utc Tue Aug 3 2021
System image file is "flash:c3750e-universalk9-mz.122-55.SE1.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: ipbase
License Type: Permanent
Next reload license Level: ipbase

cisco WS-C3750E-24TD (PowerPC405) processor (revision M0) with 262144K bytes of memory.
Processor board ID FDO1525Y4L3
Last reset from power-on
15 Virtual Ethernet interfaces
1 FastEthernet interface
28 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 30:E4:DB:CB:AF:80
Motherboard assembly number : 73-11177-14
Motherboard serial number : FDO15261FHQ
Model revision number : M0
Motherboard revision number : C0
Model number : WS-C3750E-24TD-S
Daughterboard assembly number : 73-11426-02
Daughterboard serial number : FDO15251VQC
System serial number : FDO1525Y4L3
Top Assembly Part Number : 800-28924-04
Top Assembly Revision Number : E0
Version ID : V05
CLEI Code Number : COMCF00ARB
Daughterboard revision number : A0
Hardware Board Revision Number : 0x00

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 3 30 WS-C3750E-24TD 12.2(55)SE1 C3750E-UNIVERSALK9-M

Configuration register is 0xF

ZAHIDHASEEB · ‎08-03-2021

@Leo Laohoo Results are attached for your reference

Leo Laohoo · ‎08-03-2021

Version 12.2(55)SE1

Upgrade the firmware of the switch first.

And check the logs and see if there are any flapping interfaces.

ZAHIDHASEEB · ‎08-03-2021

@Leo Laohoo

Please find the sh version result

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 3 30 WS-C3750E-24TD 12.2(55)SE1 C3750E-UNIVERSALK9-M

ZAHIDHASEEB · ‎08-03-2021

I also note one more thing sometime (when issue occurs) at my NPM tool that the Core switch ms increased.

ZAHIDHASEEB · ‎08-03-2021

FYI

#sh proc cpu sort | ex 0.00
CPU utilization for five seconds: 62%/28%; one minute: 59%; five minutes: 66%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
4 525428 22239 23626 8.88% 1.53% 1.19% 0 Check heaps
156 3423326 56575 60509 7.88% 9.26% 9.37% 0 HL3U bkgrd proce
94 1428931 853373 1674 3.01% 2.87% 5.14% 0 HLFM address lea
215 511347 305451 1674 1.86% 1.17% 1.14% 0 Spanning Tree
10 791684 991409 798 1.43% 2.02% 2.15% 0 ARP Input
330 459994 48125 9558 1.00% 1.19% 1.13% 0 CEF: IPv4 proces
202 327828 639335 512 0.71% 0.94% 0.96% 0 ADJ resolve proc
78 285412 13288 21478 0.57% 0.66% 0.63% 0 Adjust Regions
114 231720 32892 7044 0.57% 0.61% 0.57% 0 hpm counter proc
161 730 299 2441 0.42% 0.15% 0.09% 1 SSH Process
155 157290 887559 177 0.42% 0.53% 0.51% 0 Hulc LED Process
292 65694 745534 88 0.28% 0.22% 0.22% 0 EEM ED Syslog
166 70238 6798 10332 0.28% 0.17% 0.16% 0 HQM Stack Proces
324 66050 207095 318 0.28% 0.20% 0.16% 0 IP SNMP
326 113258 104660 1082 0.28% 0.28% 0.26% 0 SNMP ENGINE
325 54223 103862 522 0.14% 0.13% 0.12% 0 PDU DISPATCHER
257 23625 82147 287 0.14% 0.04% 0.05% 0 DHCPD Receive
228 38161 32892 1160 0.14% 0.10% 0.08% 0 PI MATM Aging Pr
95 17256 32893 524 0.14% 0.04% 0.02% 0 HLFM aging proce
72 24992 282455 88 0.14% 0.12% 0.06% 0 RedEarth Tx Mana
167 14911 13577 1098 0.14% 0.05% 0.01% 0 HRPC qos request

Leo Laohoo · ‎08-03-2021

I am not troubleshooting anything until the firmware gets updated. Go to the latest 12.2(55)SE.

ZAHIDHASEEB · ‎08-03-2021

@Leo Laohoo I can see the version of my CISCO Core switch is 12.2(55)SE1means the version is same on my Core switch as you said it should be. Please see the sh version output which I shared above

Mohsin Alam · ‎08-04-2021

You can refer this for troubleshooting high CPU:

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

## Make sure to mark post as helpful, If it resolved your issue. ##

ZAHIDHASEEB · ‎08-06-2021

I think I am close to the issue.

sh platform tcam utilization

CAM Utilization for ASIC# 0 Max Used
Masks/Values Masks/values

Unicast mac addresses: 6364/6364 785/785
IPv4 IGMP groups + multicast routes: 1120/1120 1/1
IPv4 unicast directly-connected routes: 6144/6144 6144/6144
IPv4 unicast indirectly-connected routes: 2048/2048 271/271
IPv4 policy based routing aces: 452/452 12/12
IPv4 qos aces: 512/512 21/21
IPv4 security aces: 964/964 33/33

Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization

show sdm prefer
The current template is "desktop default" template.
The selected template optimizes the resources in
the switch to support this level of features for
8 routed interfaces and 1024 VLANs.

number of unicast mac addresses: 6K
number of IPv4 IGMP groups + multicast routes: 1K
number of IPv4 unicast routes: 8K
number of directly-connected IPv4 hosts: 6K
number of indirect IPv4 routes: 2K
number of IPv4 policy based routing aces: 0
number of IPv4/MAC qos aces: 0.5K
number of IPv4/MAC security aces: 1K

is there any workaround instead of increasing the hardware limit thru replacing a Layer-III switch with more hardware specification