Re: High CPU utilization on 3750

Sgj · ‎02-24-2022

Hi,

I see very high utilization at random times.

XX-DS17-1#show logging
Syslog logging: enabled (0 messages dropped, 0 messages rate-limited, 12248 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

Console logging: level debugging, 103362 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 67296 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
File logging: disabled
Persistent logging: disabled

No active filter modules.

Trap logging: level warnings, 66692 message lines logged
Logging to 10.1.12.12 (udp port 514, audit disabled,
link up),
66689 message lines logged,
0 message lines rate-limited,
0 message lines dropped-by-MD,
xml disabled, sequence number disabled
filtering disabled
Logging to 10.85.130.70 (udp port 514, audit disabled,
link up),
9888 message lines logged,
0 message lines rate-limited,
0 message lines dropped-by-MD,
xml disabled, sequence number disabled
filtering disabled
Logging Source-Interface: VRF Name:
GigabitEthernet1/0/28

Log Buffer (4096 bytes):
on(Total/Intr) 65%/0%.
Feb 15 02:06:25.785: %SYS-5-CONFIG_I: Configured from console by xxx on vty0 (10.1.160.142)
Feb 15 05:28:57.375: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 100%/0%, Top 3 processes(Pid/Util): 392/90%, 89/2%, 252/0%
Feb 15 15:47:07.892: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 13%/0%.
Feb 17 11:49:10.150: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 80%/0%, Top 3 processes(Pid/Util): 10/70%, 89/2%, 5/1%
Feb 17 18:49:35.518: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 48%/0%.
Feb 20 02:52:13.424: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 86%/0%, Top 3 processes(Pid/Util): 10/79%, 5/1%, 252/0%
Feb 21 07:25:39.159: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxx] [Source: 10.1.161.28] [localport: 22] at 14:25:39 gmt+7 Mon Feb 21 2022
Feb 21 07:45:39.594: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxx] [Source: 10.1.161.28] [localport: 22] at 14:45:39 gmt+7 Mon Feb 21 2022
Feb 21 08:09:39.401: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxx] [Source: 10.1.161.28] [localport: 22] at 15:09:39 gmt+7 Mon Feb 21 2022
Feb 21 08:24:49.938: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 43%/0%.
Feb 21 08:41:27.467: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxx] [Source: 10.1.161.28] [localport: 22] at 15:41:27 gmt+7 Mon Feb 21 2022
Feb 21 08:56:50.914: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxx] [Source: 10.1.161.28] [localport: 22] at 15:56:50 gmt+7 Mon Feb 21 2022
Feb 21 09:18:15.010: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 100%/0%, Top 3 processes(Pid/Util): 138/90%, 89/2%, 252/0%
Feb 21 09:19:19.981: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 46%/0%.
Feb 21 10:03:54.927: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxx] [Source: 10.1.161.28] [localport: 22] at 17:03:54 gmt+7 Mon Feb 21 2022
Feb 21 10:06:48.608: %SYS-5-CONFIG_I: Configured from console by xxx on vty0 (10.1.161.28)
Feb 21 10:18:18.882: %SYS-5-CONFIG_I: Configured from console by xxx on vty0 (10.1.161.28)
Feb 21 10:18:35.064: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/0%, Top 3 processes(Pid/Util): 138/91%, 5/1%, 252/0%
Feb 21 10:19:40.035: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 55%/0%.
Feb 21 10:22:20.075: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/0%, Top 3 processes(Pid/Util): 138/89%, 89/2%, 252/1%
Feb 21 10:23:10.038: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 7%/0%.
Feb 21 10:25:17.673: %SYS-5-CONFIG_I: Configured from console by xxx on vty0 (10.1.161.28)
Feb 22 02:54:30.891: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 88%/0%, Top 3 processes(Pid/Util): 10/76%, 5/6%, 252/0%
Feb 22 16:55:16.615: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 10%/0%.
Feb 23 07:30:18.742: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxx] [Source: 10.1.161.87] [localport: 22] at 14:30:18 gmt+7 Wed Feb 23 2022
Feb 23 07:50:59.704: %SYS-5-CONFIG_I: Configured from console by xxx on vty0 (10.1.161.87)

XX-DS17-1#
XX-DS17-1#show stacks 10
Process 10: crypto sw pk proc
Stack segment 0x50ABBBC - 0x50AD32C
FP: 0x50AD2E0, RA: 0x2EED340
FP: 0x50AD318, RA: 0x3163F8C
FP: 0x50AD320, RA: 0x19ED108
FP: 0x0, RA: 0x19E76B8
XX-DS17-1#
XX-DS17-1#show stacks 138
No such process #138
XX-DS17-1#
XX-DS17-1#show stacks 392
Process 392: Archive Config
Stack segment 0x9C7BBC - 0x9CAA9C
FP: 0x9CAA70, RA: 0x2EEDDD8
FP: 0x9CAA88, RA: 0x20B3674
FP: 0x9CAA90, RA: 0x19ED108
FP: 0x0, RA: 0x19E76B8
XX-DS17-1#

Can anyone suggest what's going on with this?

Thank you in advance.

Leo Laohoo · ‎02-24-2022

Post the complete output to the following commands:

sh verson
sh proc cpu sort | ex 0.00

Sgj · ‎02-24-2022

XX-DS17-1#show version
Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 15.0(2)SE6, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Wed 09-Apr-14 03:37 by prod_rel_team

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(25r)SEE4, RELEASE SOFTWARE (fc1)

XX-DS17-1 uptime is 2 years, 4 weeks, 6 days, 23 hours, 38 minutes
System returned to ROM by power-on
System restarted at 10:07:30 gmt+7 Wed Jan 22 2020
System image file is "flash:/c3750-ipservicesk9-mz.150-2.SE6.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C3750G-24TS-1U (PowerPC405) processor (revision H0) with 131072K bytes of memory.
Processor board ID FOC1425Z4CY
Last reset from power-on
42 Virtual Ethernet interfaces
28 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 18:EF:63:F6:27:00
Motherboard assembly number : 73-10219-09
Power supply part number : 341-0098-02
Motherboard serial number : FOC142555ST
Power supply serial number : DCA1419900A
Model revision number : H0
Motherboard revision number : A0
Model number : WS-C3750G-24TS-E1U
System serial number : FOC1425Z4CY
Top Assembly Part Number : 800-27484-03
Top Assembly Revision Number : A0
Version ID : V05
CLEI Code Number : COMBL10BRA
Hardware Board Revision Number : 0x09

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 28 WS-C3750G-24TS-1U 15.0(2)SE6 C3750-IPSERVICESK9-M

Configuration register is 0xF

XX-DS17-1#
XX-DS17-1#show processes cpu sorted | exclude 0.00
CPU utilization for five seconds: 10%/0%; one minute: 11%; five minutes: 11%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
89 328260528 4195937 78234 2.87% 2.33% 2.29% 0 Adjust Regions
252 540791618 584800209 924 1.27% 1.15% 1.09% 0 Spanning Tree
239 96956268 236542719 409 0.31% 0.13% 0.13% 0 IP Input
138 1809 666 2716 0.15% 0.11% 0.16% 1 SSH Process
183 37798423 26357996 1434 0.15% 0.09% 0.04% 0 HRPC qos request
13 174020334 287721934 604 0.15% 0.08% 0.09% 0 ARP Input
XX-DS17-1#

Leo Laohoo · ‎02-24-2022

@Sgj wrote:

Feb 21 10:22:20.075: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/0%, Top 3 processes(Pid/Util): 138/89%, 89/2%, 252/1%

PID 138 is SSH.

@Sgj wrote:

* 1 28 WS-C3750G-24TS-1U 15.0(2)SE6 C3750-IPSERVICESK9-M

I have never used this IOS version before. I recommend to upgrade to 15.0(2)SE11.

Sgj · ‎02-24-2022

PID 138 is SSH.

Is there a link that points to your statement?

I have never used this IOS version before. I recommend to upgrade to 15.0(2)SE11.

I've tried to compare with other devices with the same model and version, but no problem.

Leo Laohoo · ‎02-24-2022

@Sgj wrote:
Top 3 processes(Pid/Util): 138/89%, 89/2%, 252/1%

Is there a link that points to your statement?

When the EEM got triggered, the CPU reading of 89% came from the SSH process.

Sgj · ‎02-25-2022

So what about:

- PID 392?

- PID 10?

Log Buffer (4096 bytes):
on(Total/Intr) 65%/0%.
Feb 15 02:06:25.785: %SYS-5-CONFIG_I: Configured from console by xxx on vty0 (10.1.160.142)
Feb 15 05:28:57.375: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 100%/0%, Top 3 processes(Pid/Util): 392/90%, 89/2%, 252/0%
Feb 15 15:47:07.892: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 13%/0%.
Feb 17 11:49:10.150: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 80%/0%, Top 3 processes(Pid/Util): 10/70%, 89/2%, 5/1%
Feb 17 18:49:35.518: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 48%/0%.
Feb 20 02:52:13.424: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 86%/0%, Top 3 processes(Pid/Util): 10/79%, 5/1%, 252/0%
Feb 21 07:25:39.159: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxx] [Source: 10.1.161.28] [localport: 22] at 14:25:39 gmt+7 Mon Feb 21 2022
Feb 21 07:45:39.594: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxx] [Source: 10.1.161.28] [localport: 22] at 14:45:39 gmt+7 Mon Feb 21 2022
Feb 21 08:09:39.401: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxx] [Source: 10.1.161.28] [localport: 22] at 15:09:39 gmt+7 Mon Feb 21 2022
Feb 21 08:24:49.938: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 43%/0%.
Feb 21 08:41:27.467: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxx] [Source: 10.1.161.28] [localport: 22] at 15:41:27 gmt+7 Mon Feb 21 2022
Feb 21 08:56:50.914: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxx] [Source: 10.1.161.28] [localport: 22] at 15:56:50 gmt+7 Mon Feb 21 2022
Feb 21 09:18:15.010: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 100%/0%, Top 3 processes(Pid/Util): 138/90%, 89/2%, 252/0%
Feb 21 09:19:19.981: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 46%/0%.
Feb 21 10:03:54.927: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxx] [Source: 10.1.161.28] [localport: 22] at 17:03:54 gmt+7 Mon Feb 21 2022
Feb 21 10:06:48.608: %SYS-5-CONFIG_I: Configured from console by xxx on vty0 (10.1.161.28)
Feb 21 10:18:18.882: %SYS-5-CONFIG_I: Configured from console by xxx on vty0 (10.1.161.28)
Feb 21 10:18:35.064: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/0%, Top 3 processes(Pid/Util): 138/91%, 5/1%, 252/0%
Feb 21 10:19:40.035: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 55%/0%.
Feb 21 10:22:20.075: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/0%, Top 3 processes(Pid/Util): 138/89%, 89/2%, 252/1%
Feb 21 10:23:10.038: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 7%/0%.
Feb 21 10:25:17.673: %SYS-5-CONFIG_I: Configured from console by xxx on vty0 (10.1.161.28)
Feb 22 02:54:30.891: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 88%/0%, Top 3 processes(Pid/Util): 10/76%, 5/6%, 252/0%
Feb 22 16:55:16.615: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 10%/0%.
Feb 23 07:30:18.742: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: xxx] [Source: 10.1.161.87] [localport: 22] at 14:30:18 gmt+7 Wed Feb 23 2022
Feb 23 07:50:59.704: %SYS-5-CONFIG_I: Configured from console by xxx on vty0 (10.1.161.87)

Leo Laohoo · ‎02-26-2022

@Sgj wrote:

- PID 392?

- PID 10?

Sorry, I do not memorize PID mappings. Have a look at the output to the command "sh proc cpu sort | ex 0.00".

marce1000 · ‎02-24-2022

- Upgrade to current-advisory software version for your platform, check if that can help.

M.

-- ' 'Good body every evening' ' this sentence was once spotted on a logo at the entrance of a Weight Watchers Club !

MHM Cisco World · ‎02-25-2022

show cpu history <- this give the history for 60 min. or mote

MHM Cisco World · ‎02-25-2022

The Adjust Regions process is used for Adjusting TCAM Regions for holding host and route adjacencies. This process will run specifically when adjustments are made to the tables that populate these adjacencies - the routing table and the ARP table. If this process is observed to be running higher than expected, investigate if there are frequent routing updates or changes to the routing table. Also investigate if there are frequent ARP gleans or many incomplete ARP entries. If there are frequent incomplete ARP entries, it is likely there are hosts attempting to route through the switch for a destination address that is no longer present - this will result in ARP failing to complete and displaying as Incomplete in show ip arp.

this is normal if you see many ARP incomplete.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCtf03505?rfs=iqvred