cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
4426
Views
0
Helpful
16
Replies

High utilization on 3750x switch stack

Mohit Chauhan
Level 1
Level 1

hi

i am seeing very high utilization on a random basis on the stack. and the logs indicate me that there is a pattern where there is a stack power cable shows inserted (which was never unplugged) followed by sudden spike in the utilization.

001018: *Mar  6 16:21:22.138: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 4 stack power cable 1 inserted

001019: *Mar  6 18:18:37.982: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/1%, Top 3 processes(Pid/Util):  390/72%, 239/6%, 125/4%

001020: *Mar  6 18:20:17.983: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 16%/1%.

001021: *Mar  6 19:29:34.303: %SYS-5-CONFIG_I: Configured from console by helpdesk on vty0 (159.207.161.102)

001022: *Mar  6 19:38:11.133: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 4 stack power cable 1 inserted

001023: *Mar  6 20:20:23.067: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/2%, Top 3 processes(Pid/Util):  390/73%, 239/5%, 125/3%

001024: *Mar  6 20:23:12.987: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 24%/2%.

I would be unplugging the power stack cables to test this but wanted to share this you if someone has seen the same pattern before?

Regards,

Mohit

1 Accepted Solution

Accepted Solutions

This case is very similar to many I have faced before... I have decoded those PC values you sent me

sho stacks 390

Process 390:  SNMP ENGINE

  Stack segment 0x7F8F14C - 0x7F9202C

  FP: 0x7F91F90, RA: 0x1B97920

  FP: 0x7F92018, RA: 0x18BD1D4

  FP: 0x7F92020, RA: 0x284DA88

  FP: 0x0, RA: 0x28478FC

I found out that this problem happens when the SNMP is polling ARP and routing tables, image if the ARP is too large... it will definitely cause CPU utilization problems. the larger the arp table is the highest impact you will see in the CPU.

I would recommend you to read and follow this steps, specially the recommendation #1

http://www.cisco.com/en/US/tech/tk648/tk362/technologies_tech_note09186a00800948e6.shtml

Keep me posted...

Wilson B.

View solution in original post

16 Replies 16

Leo Laohoo
Hall of Fame
Hall of Fame

This may be contributed if the stackpower cable is loose and goes on/off constantly.  This will cause your CPU to spike.

I have reseated the stack power cable again and also shrunk the SNMP related config as it was contributing to 72% at the time of spike.

I am watching for the next spike now, and would advise the same.

Thanks leolaohoo..

Well that hasnt helped i am afraid, i still see the spike kicking in:

001034: Feb 27 12:30:45.358: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/1%, Top 3 processes(Pid/Util):  390/76%, 239/5%, 125/3%

001035: Feb 27 12:33:35.235: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 18%/2%.

this time i did not see any stack power cable related message...but this time again the 76% was against SNMP engine process. Does it mean my snmp settings are causing some issues?

Below is my snmp config:

SwitchA# sh run | sec snmp

snmp-server community RO

snmp-server community RW

snmp-server enable traps tty

snmp-server enable traps ospf state-change

snmp-server enable traps ospf errors

snmp-server enable traps event-manager

snmp-server enable traps cpu threshold

snmp-server enable traps bridge newroot topologychange

snmp-server enable traps stpx inconsistency root-inconsistency loop-inconsistency

snmp-server enable traps syslog

snmp-server enable traps flash insertion removal

snmp-server enable traps envmon fan shutdown supply temperature status

snmp-server enable traps errdisable

snmp-server enable traps mac-notification change move threshold

snmp-server host 172.20.200.9 version 2c

snmp-server host 159.207.208.21 version 2c   cpu

Not sure what is causing the CPU hog.

What is your IOS?

Can you post the first page of the output "sh proc cpu sorted"?

001034:  Feb 27 12:30:45.358: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU  Utilization(Total/Intr): 99%/1%, Top 3 processes(Pid/Util):  390/76%,  239/5%, 125/3%

001035: Feb 27 12:33:35.235: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 18%/2%.

- See more at: https://supportforums.cisco.com/message/3868725#3868725

Hello Mohit.

The logs are describing the PID causing high cpu utilization, if you pay attention closely to the PID, and then check the show process cpu you will find out what are the top offenders, so right now we know that PID 390 is the very top offender increasing utilization up to 76%, now what is the name of the process is what we need to know to figure out the cause. please post the output of

show process cpu | inc PID|390

001034: Feb 27 12:30:45.358: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/1%, Top 3 processes(Pid/Util):  390/76%, 239/5%, 125/3%

001035: Feb 27 12:33:35.235: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 18%/2%.

Regards.

Wilson.

Hi Wilson

This is SNMP engine.

And i have confirmed that once we turn off the snmp from the switch, the cpu util remains normal.

So now I need to work out why is that happening and how can resolve the tihng cos the client would need the snmp for the mgmt purpose.

Hi,

mmm, What IOS version is it running? show version.

Regards.

Wilson

15.0 (1)

Mohit

Could you post the output of the command show stacks 390, also this may be weird but could you also add attached in a notepad the show arp... =) while the problem is happening.

Regards.

Wilson

Hi Wilson,

Here is the output:

sho stacks 390

Process 390:  SNMP ENGINE

  Stack segment 0x7F8F14C - 0x7F9202C

  FP: 0x7F91F90, RA: 0x1B97920

  FP: 0x7F92018, RA: 0x18BD1D4

  FP: 0x7F92020, RA: 0x284DA88

  FP: 0x0, RA: 0x28478FC

and the show arp (I have removed the ip address for security reason as we are using public addresses) This time the cpu is 20% only. I'll have to wait for spike to occur and i'll send you the arp that moment again.

btw, whats the best way to check the server side, and what things to check on that side?

ProtocolAddressAge(min)HardwareAddrType
Internet
1790008.e3ff.fd90ARPATenGigabitEthernet1/1/2
Internet
-6c20.560c.8d43ARPATenGigabitEthernet1/1/2
Internet
1280008.e3ff.fd90ARPATenGigabitEthernet5/1/2
Internet
-6c20.560c.8d45ARPATenGigabitEthernet5/1/2
Internet
1370008.e3ff.fd90ARPATenGigabitEthernet1/1/1
Internet
-6c20.560c.8d42ARPATenGigabitEthernet1/1/1
Internet
1190008.e3ff.fd90ARPATenGigabitEthernet2/1/1
Internet
-6c20.560c.8d44ARPATenGigabitEthernet2/1/1
Internet
-6c20.560c.8d64ARPAVlan2008
Internet
-6c20.560c.8d65ARPAVlan2009
Internet
-6c20.560c.8d66ARPAVlan2010
Internet
-6c20.560c.8d67ARPAVlan2012
Internet
-6c20.560c.8d68ARPAVlan2013
Internet
-6c20.560c.8d69ARPAVlan2014
Internet
-6c20.560c.8d6aARPAVlan2016
Internet
-6c20.560c.8d6bARPAVlan2017
Internet
-6c20.560c.8d6cARPAVlan2018
Internet
-6c20.560c.8d6dARPAVlan2019
Internet
-6c20.560c.8d6eARPAVlan2021
Internet
-6c20.560c.8d6fARPAVlan2022
Internet
-6c20.560c.8d70ARPAVlan2024
Internet
-6c20.560c.8d71ARPAVlan2025
Internet
-6c20.560c.8d72ARPAVlan2026
Internet
-6c20.560c.8d73ARPAVlan2027
Internet
-6c20.560c.8d74ARPAVlan2028
Internet
-6c20.560c.8d75ARPAVlan2029
Internet
-6c20.560c.8d76ARPAVlan2030
Internet
-6c20.560c.8d77ARPAVlan2032
Internet
-6c20.560c.8d78ARPAVlan2033
Internet
-6c20.560c.8d79ARPAVlan2034
Internet
-6c20.560c.8d7aARPAVlan2035
Internet
-6c20.560c.8d7bARPAVlan2036
Internet
05c26.0a5a.07ebARPAVlan2036
Internet
05c26.0a5a.0a8dARPAVlan2036
Internet
1b8ac.6f2a.7fb5ARPAVlan2036
Internet
05c26.0a5a.0839ARPAVlan2036
Internet
10024.1dae.eef4ARPAVlan2036
Internet
4b8ac.6f24.61cbARPAVlan2036
Internet
-6c20.560c.8d7cARPAVlan2037
Internet
-6c20.560c.8d7dARPAVlan2038
Internet
-6c20.560c.8d58ARPAVlan1064
Internet
-6c20.560c.8d59ARPAVlan1080
Protocol
Age(min)HardwareAddrType
Internet
-6c20.560c.8d5aARPAVlan1096
Internet
-0000.0c07.ac68ARPAVlan1104
Internet
06c20.560c.9e5eARPAVlan1104
Internet
-6c20.560c.8d5bARPAVlan1104
Internet
50011.0aec.1bd4ARPAVlan1104
Internet
0IncompleteARPA

Internet
0IncompleteARPA

Internet
50014.384a.7c55ARPAVlan1104
Internet
12500c0.ee43.6b03ARPAVlan1104
Internet
60001.e6a7.da50ARPAVlan1104
Internet
60001.e6e1.8951ARPAVlan1104
Internet
40021.5a86.aef7ARPAVlan1104
Internet
100c0.ee1f.4b67ARPAVlan1104
Internet
0IncompleteARPA

Internet
500c0.ee42.697fARPAVlan1104
Internet
000c0.ee8f.a1c5ARPAVlan1104
Internet
5000e.7fe2.fa66ARPAVlan1104
Internet
0IncompleteARPA

Internet
0IncompleteARPA

Internet
6001b.78f5.8a60ARPAVlan1104
Internet
000c0.ee56.1db9ARPAVlan1104
Internet
160026.7357.831bARPAVlan1104
Internet
180026.7357.82e8ARPAVlan1104
Internet
10026.7357.82f9ARPAVlan1104
Internet
50026.7357.831aARPAVlan1104
Internet
170026.7357.82fbARPAVlan1104
Internet
50026.7351.dce7ARPAVlan1104
Internet
10026.7358.8b41ARPAVlan1104
Internet
00026.7358.8ae4ARPAVlan1104
Internet
160026.7358.88ebARPAVlan1104
Internet
60026.7358.88eaARPAVlan1104
Internet
80026.7349.3111ARPAVlan1104
Internet
00026.7358.88ccARPAVlan1104
Internet
00026.7358.88bdARPAVlan1104
Internet
10026.7358.8b74ARPAVlan1104
Internet
170026.7358.88efARPAVlan1104
Internet
0IncompleteARPA

Internet
1220004.75f4.fe2c