High CPU due to SSH

[michalweiss]

I noticed that sh run, wr and some other sh commands are taking forever to execute. After researching I have noticed that my CPU utilization is really high.  

Does anyone know how to fix high CPU utilization caused by SSH on 6509-e?

SW-CORE-6509#sh processes cpu sorted | e 0.00
CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  39  1092536464   8286094     131854 95.30% 96.16% 96.40%   1 SSH Process     
 256   952116768 435399847       2186  1.56%  1.39%  1.38%   0 CDP Protocol    
   8   311148064  19968942      15581  0.78%  0.30%  0.29%   0 Check heaps     
 507         172        87       1977  0.46%  0.18%  0.04%   2 SSH Process     
  11   144194452 436628234        330  0.23%  0.08%  0.07%   0 ARP Input       
 446   9322536282818564302          0  0.15%  0.08%  0.02%   0 Port manager per
  23   458571448 820619180        558  0.15%  0.09%  0.11%   0 IPC Seat Manager
 500   104503696 405946127        257  0.07%  0.06%  0.04%   0 SNMP ENGINE     
 365     2119476  84037544         25  0.07%  0.05%  0.05%   0 LLDP Protocol

SW-CORE-6509#sh users
Line User Host(s) Idle Location
1 vty 0 ttsit idle 1w6d 10.32.183.136
* 2 vty 1 ttsit idle 00:00:00 10.32.50.2

[Venkatesh Perumal]

SSH process go high if capturing a show tech or some debug is left enabled.

have a look on the below docs:

https://supportforums.cisco.com/document/59926/troubleshooting-high-cpu-6500-sup720

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

https://supportforums.cisco.com/discussion/11613676/6509-e-high-cpu-no-processes

regards,

-Do rate helpful posts.

[michalweiss]

Thanks for the reply.

I think that the SSH session is causing the issue, but I can figure out why. I turned off all debugging and the CPU is still at 100%

SW-CORE-6509#no debug all
All possible debugging has been turned off

SW-CORE-6509#sh processes cpu | exclude 0.00
CPU utilization for five seconds: 100%/0%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
11 144258500 436874441 330 0.07% 0.14% 0.11% 0 ARP Input
23 458892384 821256207 558 0.15% 0.14% 0.12% 0 IPC Seat Manager
39 1148772004 8702460 132013 96.83% 95.78% 96.23% 1 SSH Process
256 952926028 435598619 2187 1.26% 1.34% 1.35% 0 CDP Protocol
345 33676040 134976242 249 0.23% 0.06% 0.06% 0 CEF: IPv4 proces
347 35792032 46342825 772 0.07% 0.04% 0.05% 0 HIDDEN VLAN Proc
365 2156312 84257254 25 0.07% 0.03% 0.05% 0 LLDP Protocol
507 508 217 2341 0.07% 0.47% 0.14% 2 SSH Process

[michalweiss]

Another strange thing that I have noticed is that the vty session is connected for over a week and I have VTY session timeout enabled. 

SW-CORE-6509#sh users
Line User Host(s) Idle Location
1 vty 0 ttsit idle 1w6d 
2 vty 1 ttsit idle 00:01:47 
* 3 vty 2 ttsit idle 00:00:00 

line vty 0 4

 session-timeout 10

 logging synchronous

 login local

 transport input ssh

 transport output telnet ssh

[michalweiss]

I tried to kick the user with commands below with no luck.

SW-CORE-6509#sh use
SW-CORE-6509#sh users
Line User Host(s) Idle Location
1 vty 0 ttsit idle 1w6d 10.32.183.136
* 2 vty 1 ttsit idle 00:00:00 10.32.180.64

SW-CORE-6509#clear line 0
[confirm]y [OK]
SW-CORE-6509#clear line vt
SW-CORE-6509#clear line vty 0
[confirm]y [OK]
SW-CORE-6509#sh tcp br
TCB Local Address Foreign Address (state)
48E4F564 10.139.15.1.22 10.32.180.64.15034 ESTAB
SW-CORE-6509#sh use
SW-CORE-6509#sh users
Line User Host(s) Idle Location
1 vty 0 ttsit idle 1w6d 10.32.183.136
* 2 vty 1 ttsit idle 00:00:00 10.32.180.64

[tommar]

these occasions first thing after no luck with the clearing the vty line is reboot. that probably already happened...? and suppose it fixed the problem.?