High CPU Utilization due to SSH process in catalyst 2960X POE Switch

[MParashar]

Hi, 

We are using around 50 + cisco catalyst 2960X POE switches in our office Network for Wi-Fi & L2-functionality. All of the switches are configured for SNMP monitoring & sys logs(Observium & Zabbix).We had a Large vlan database of approx. 50+ vlan. SSH access is configured for remote login. All the switches are having uniform configuration apart from vlan database. Earlier we were getting continuous spikes in CPU utilization in all switches due to HULC LED, IP SNMP ,SNMP ENGINE & STP Processes. Last month I had updated the VTP(using VTP version2) configuration of all the switches from client mode to transparent mode to overcome the high cpu utilization due to STP, IP SNMP & SNMP ENGINE processes.  I had manually configured only required vlans on switches as per requirement. It worked well for me and frequent cpu spikes had been stopped and cpu utilization had been dropped by 8-12% in 24P switches  and 15-20% in 48 P switches. But since configuration changes I had observed some unusual behaviour from switches end. I had observed  cpu spikes daily  in around 20 switches at a fixed Interval of time in morning time. I am not aware whether earlier also same thing was happening or not(before configuration changes). It's happening since last 1 month.  Upon checking the details i have found the main culprit i.e: ssh process (consuming 30-75% of cpu resource). Can anyone please tell me the root cause of this and how can i resolve it.

Thanks in Advance

Observed CPU spikes in different switches due to SSH process

[marce1000]

 - FYI : https://community.cisco.com/t5/switching/high-cpu-usage-on-2960-x/m-p/3323462#M403621

 M.

[MParashar]

Thanks for providing the information but I am looking for the specific cause of spikes in CPU due to ssh process at a fixed interval of time. Is it related to generation of public key or something else?

[Leo Laohoo]

Post the complete output to the following commands: 

1. sh version
2. sh logs
3. sh proc cpu sort | ex 0.00

[MParashar]

Hi Leo,

Below are the details.

1. Sh version

Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.2(7)E4, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2021 by Cisco Systems, Inc.
Compiled Mon 08-Mar-21 11:26 by prod_rel_team

ROM: Bootstrap program is C2960X boot loader
BOOTLDR: C2960X Boot Loader (C2960X-HBOOT-M) Version 15.2(6r)E, RELEASE SOFTWARE (fc1)

uptime is 7 weeks, 4 days, 18 hours, 53 minutes
System returned to ROM by power-on
System restarted at 05:29:18 IST Sat Jan 1 2000
System image file is "flash:c2960x-universalk9-mz.152-7.E4.bin"
Last reload reason: power-on

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960X-24PD-L (APM86XXX) processor (revision T0) with 524288K bytes of memory.
Last reset from power-on
1 Virtual Ethernet interface
1 FastEthernet interface
26 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Motherboard assembly number : 73-16689-06
Power supply part number : 341-0528-02
Model revision number : T0
Motherboard revision number : B0
Model number : WS-C2960X-24PD-L
Daughterboard assembly number : 73-14200-03
Top Assembly Part Number : 68-100467-03
Top Assembly Revision Number : E0
Version ID : V06
Daughterboard revision number : B0
Hardware Board Revision Number : 0x19

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 30 WS-C2960X-24PD-L 15.2(7)E4 C2960X-UNIVERSALK9-M

2. sh logs

  Log Buffer (4096 bytes):
LLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 40%/0%.
000324: Feb 11 06:19:20: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 93%/1%, Top 3 processes(Pid/Util): 155/61%, 179/12%, 446/1%
000325: Feb 11 06:19:35: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 60%/0%.
000326: Feb 12 06:19:20: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 81%/1%, Top 3 processes(Pid/Util): 155/47%, 179/12%, 446/2%
000327: Feb 12 06:19:35: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 74%/0%.
000328: Feb 12 17:36:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/24, changed state to down
000329: Feb 12 17:36:49: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/24, changed state to down
000330: Feb 12 17:37:02: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/24, changed state to up
000331: Feb 12 17:37:04: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/24, changed state to up
000332: Feb 12 17:37:07: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/24, changed state to down
000333: Feb 12 17:37:08: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/24, changed state to down
000334: Feb 12 17:37:15: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/24, changed state to up
000335: Feb 12 17:37:17: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/24, changed state to up
000336: Feb 13 06:19:25: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 98%/0%, Top 3 processes(Pid/Util): 155/68%, 179/11%, 446/1%
000337: Feb 13 06:19:40: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 49%/0%.
000338: Feb 14 06:19:25: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/0%, Top 3 processes(Pid/Util): 155/70%, 179/10%, 446/1%
000339: Feb 14 06:19:40: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 47%/0%.
000340: Feb 16 06:19:25: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/0%, Top 3 processes(Pid/Util): 155/71%, 179/11%, 141/0%
000341: Feb 16 06:19:40: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 45%/0%.
000342: Feb 17 06:19:25: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 98%/0%, Top 3 processes(Pid/Util): 155/68%, 179/11%, 446/1%
000343: Feb 17 06:19:40: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 48%/0%.
000344: Feb 18 06:19:20: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 96%/0%, Top 3 processes(Pid/Util): 155/63%, 179/13%, 446/1%
000345: Feb 18 06:19:35: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 60%/0%.
000346: Feb 19 06:19:25: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/0%, Top 3 processes(Pid/Util): 155/73%, 179/10%, 141/0%
000347: Feb 19 06:19:40: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 32%/0%.
000348: Feb 20 06:19:25: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/0%, Top 3 processes(Pid/Util): 155/71%, 179/11%, 141/0%
000349: Feb 20 06:19:35: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 77%/0%.
000350: Feb 21 06:19:25: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/0%, Top 3 processes(Pid/Util): 155/72%, 179/9%, 141/0%
000351: Feb 21 06:19:40: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 35%/0%.
000352: Feb 22 06:19:20: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 84%/0%, Top 3 processes(Pid/Util): 155/50%, 179/12%, 446/2%
000353: Feb 22 06:19:35: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 74%/0%.
000354: Feb 23 06:19:25: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 100%/0%, Top 3 processes(Pid/Util): 155/71%, 179/11%, 141/0%
000355: Feb 23 06:19:40: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 32%/0%.

I55  is SSH process,179 is HULC LED Process, 141 HPM Counter Process,446 SNMP Engine

3. sh proc cpu sort | ex 0.00

CPU utilization for five seconds: 33%/0%; one minute: 32%; five minutes: 31%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
179 789468296 116150840 6796 14.64% 13.64% 13.59% 0 Hulc LED Process
141 33297649 4647421 7164 0.65% 0.65% 0.62% 0 hpm counter proc
94 9859372 209189289 47 0.41% 0.23% 0.18% 0 RedEarth Tx Mana
137 14405085 74179091 194 0.41% 0.44% 0.39% 0 hpm main process
131 14226366 125657397 113 0.35% 0.31% 0.31% 0 HLFM address lea
95 17179614 210943007 81 0.29% 0.43% 0.42% 0 RedEarth Rx Mana
446 6888186 6576021 1047 0.23% 0.08% 0.06% 0 SNMP ENGINE
241 145254129 154176717 942 0.23% 0.21% 0.23% 0 Spanning Tree
229 5390766 125657519 42 0.17% 0.13% 0.12% 0 IP ARP Retry Age
444 3692902 11487798 321 0.11% 0.01% 0.01% 0 IP SNMP
443 3965052 11552633 343 0.11% 0.06% 0.06% 0 VLAN Manager

[MParashar]

CPU spikes due to SSH process happens daily in morning time at 3:33 AM IST and frequency is once in a day at the mentioned time.

[Leo Laohoo]

Wait a second ... Do you have DNAC?

[MParashar]

No , We don't have..

[Leo Laohoo]

WTF ... The "sh logs" have a time-and-date stamp of February.  

Show is the output to the command "sh proc 155".

[MParashar]

There was some NTP syncing issues which had been rectified yesterday.

Output of sh Proc 155

Memory usage [in bytes]
Holding: 93480, Maximum: 93480, Allocated: 1421072, Freed: 1339804
Getbufs: 0, Retbufs: 0, Stack: 9276/12000
CPU usage
PC: 0, Invoked: 91, Giveups: 32, uSec: 4263
5Sec: 0.29%, 1Min: 0.45%, 5Min: 0.10%, Average: 2.49%
Age: 15575 msec, Runtime: 388 msec
State: Running, Priority: Normal

[MHM Cisco World]

The High CPU come from I think STP, 
do
show spanning tree VLAN x detail (VLAN x of  g1/0/24)
check the last TCN  time is it same as High CPU Log

[MParashar]

Hi,

After changing vtp configuration to transparent mode and filtering out only required vlans on a particular switch,STP proc uses has been reduced .Currently SSH processes is consuming 30-75% cpu utilization  on more than 20 switches at a fixed interval of time i.e: 3:33 AM IST daily.  I am not been able to understand exact cause of it.

[MHM Cisco World]

generation of Key not happened with this fast interval.
can we see 
show cpu ?

[MParashar]

High CPU Utilization is not continuous.We are facing cpu spikes and that too at specific time interval daily i.e: 3:33 AM IST. Otherwise CPU utilization lies in between 30-40%.Current status of CPU utilization.

show processes cpu sorted | exc 0.00%
CPU utilization for five seconds: 30%/0%; one minute: 31%; five minutes: 31%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
179 801570859 118162180 6783 12.77% 13.70% 13.57% 0 Hulc LED Process
141 33878527 4727908 7165 0.59% 0.61% 0.64% 0 hpm counter proc
137 14761578 75549194 195 0.41% 0.42% 0.40% 0 hpm main process
95 17604135 214625510 82 0.41% 0.33% 0.35% 0 RedEarth Rx Mana
131 14562955 127865267 113 0.35% 0.27% 0.29% 0 HLFM address lea
94 10058663 212844365 47 0.29% 0.19% 0.18% 0 RedEarth Tx Mana
241 145432384 154457859 941 0.29% 0.17% 0.17% 0 Spanning Tree
155 451 281 1604 0.17% 0.05% 0.02% 1 SSH Process
230 1528147 6634096 230 0.17% 0.05% 0.01% 0 IP Input
195 7217266 943575 7648 0.11% 0.12% 0.11% 0 HQM Stack Proces
229 5520454 127865386 43 0.11% 0.10% 0.10% 0 IP ARP Retry Age
91 5557000 163024977 34 0.11% 0.11% 0.11% 0 Draught link sta
62 1095902 4727965 231 0.11% 0.07% 0.02% 0 Per-Second Jobs
196 2855855 1887128 1513 0.05% 0.05% 0.05% 0 HRPC qos request
170 1610736 23376451 68 0.05% 0.05% 0.05% 0 Hulc Storm Contr
443 4015241 11566147 347 0.05% 0.05% 0.05% 0 VLAN Manager