Solved: High utilization on 3750x switch stack - Page 2

Mohit Chauhan · ‎02-26-2013

hi

i am seeing very high utilization on a random basis on the stack. and the logs indicate me that there is a pattern where there is a stack power cable shows inserted (which was never unplugged) followed by sudden spike in the utilization.

001018: *Mar 6 16:21:22.138: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 4 stack power cable 1 inserted

001019: *Mar 6 18:18:37.982: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/1%, Top 3 processes(Pid/Util): 390/72%, 239/6%, 125/4%

001020: *Mar 6 18:20:17.983: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 16%/1%.

001021: *Mar 6 19:29:34.303: %SYS-5-CONFIG_I: Configured from console by helpdesk on vty0 (159.207.161.102)

001022: *Mar 6 19:38:11.133: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 4 stack power cable 1 inserted

001023: *Mar 6 20:20:23.067: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 99%/2%, Top 3 processes(Pid/Util): 390/73%, 239/5%, 125/3%

001024: *Mar 6 20:23:12.987: %SYS-1-CPUFALLINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr) 24%/2%.

I would be unplugging the power stack cables to test this but wanted to share this you if someone has seen the same pattern before?

Regards,

Mohit

Mohit Chauhan · ‎02-27-2013

Hi Wilson,

I did follow the document you mentioned about

http://www.cisco.com/en/US/tech/tk648/tk362/technologies_tech_note0#sthash.Wor721hd.dpuf and following that I have added the following on the switch:

snmp-server view cutdown iso included

snmp-server view cutdown at excluded

snmp-server view cutdown snmpUsmMIB excluded

snmp-server view cutdown snmpVacmMIB excluded

snmp-server view cutdown snmpCommunityMIB excluded

snmp-server view cutdown ip.21 excluded

snmp-server view cutdown ip.22 excluded

snmp-server view cutdown ciscoFlashMIB excluded

snmp-server community Public view cutdown RO

snmp-server community Private view cutdown RW

By doing the above, I still see spike in the cpu util when i run sh proc cpu command. however, it seems that the spike doesnt stay high for that long period cos it never shows up in the sh logg output. This wasnt the case before.

But I was thinking why couldnt I stop spike completely in first place if thats possible.

Anyway, here is the show version output.(Removed the serial details for security reason)

SwitchA#sh ver

Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.0(1)SE3, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Wed 30-May-12 13:41 by prod_rel_team

ROM: Bootstrap program is C3750E boot loader

BOOTLDR: C3750E Boot Loader (C3750X-HBOOT-M) Version 12.2(58r)SE, RELEASE SOFTWARE (fc1)

SP-DS-01 uptime is 6 days, 16 hours, 53 minutes

System returned to ROM by power-on

System restarted at 22:07:27 AWST Thu Feb 21 2013

System image file is "flash:/c3750e-universalk9-mz.150-1.SE3/c3750e-universalk9-mz.150-1.SE3.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

License Level: ipbase

License Type: Permanent

Next reload license Level: ipbase

cisco WS-C3750X-12S (PowerPC405) processor (revision A0) with 524288K bytes of memory.

Processor board ID FDO1635V046

Last reset from power-on

60 Virtual Ethernet interfaces

1 FastEthernet interface

80 Gigabit Ethernet interfaces

10 Ten Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address : 6C:20:56:0C:9E:00

Motherboard assembly number : 73-13329-03

Motherboard serial number : Fxxxxxxxxx

Model revision number : A0

Motherboard revision number : B0

Model number : WS-C3750X-12S-S

Daughterboard assembly number : 800-32727-03

Daughterboard serial number : Fxxxxxxxxx

System serial number : FDxxxxxxxx

Top Assembly Part Number : 800-35090-02

Top Assembly Revision Number : B0

Version ID : V02

CLEI Code Number : CMMFG00ARB

Hardware Board Revision Number : 0x04

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

* 1 18 WS-C3750X-12S 15.0(1)SE3 C3750E-UNIVERSALK9-M

2 18 WS-C3750X-12S 15.0(1)SE3 C3750E-UNIVERSALK9-M

3 18 WS-C3750X-12S 15.0(1)SE3 C3750E-UNIVERSALK9-M

4 18 WS-C3750X-12S 15.0(1)SE3 C3750E-UNIVERSALK9-M

5 18 WS-C3750X-12S 15.0(1)SE3 C3750E-UNIVERSALK9-M

Switch 02

---------

Switch Uptime : 6 days, 16 hours, 54 minutes

Base ethernet MAC Address : 6C:20:56:0C:2F:80

Motherboard assembly number : 73-13329-03

Motherboard serial number : FDxxxxxxxxxxxxx

Model revision number : A0

Motherboard revision number : B0

Model number : WS-C3750X-12S-S

Daughterboard assembly number : 800-32727-03

Daughterboard serial number : FDxxxxxxxxxxxx

System serial number : FDOxxxxxxxx

Top assembly part number : 800-35090-02

Top assembly revision number : B0

Version ID : V02

CLEI Code Number : CMMFG00ARB

License Level : ipbase

License Type : Permanent

Next reboot licensing Level : ipbase

Switch 03

---------

Switch Uptime : 6 days, 16 hours, 54 minutes

Base ethernet MAC Address : 6C:20:56:0C:47:80

Motherboard assembly number : 73-13329-03

Motherboard serial number : FDxxxxxxxxx

Model revision number : A0

Motherboard revision number : B0

Model number : WS-C3750X-12S-S

Daughterboard assembly number : 800-32727-03

Daughterboard serial number : FDxxxxxxxxxx

System serial number : FDxxxxxxxxxx

Top assembly part number : 800-35090-02

Top assembly revision number : B0

Version ID : V02

CLEI Code Number : CMMFG00ARB

License Level : ipbase

License Type : Permanent

Next reboot licensing Level : ipbase

Switch 04

---------

Switch Uptime : 6 days, 16 hours, 54 minutes

Base ethernet MAC Address : FC:99:47:C5:68:00

Motherboard assembly number : 73-13329-03

Motherboard serial number : Fxxxxxxxx

Model revision number : A0

Motherboard revision number : B0

Model number : WS-C3750X-12S-S

Daughterboard assembly number : 800-32727-03

Daughterboard serial number : Fxxxxxxxx

System serial number : Fxxxxxxxxx

Top assembly part number : 800-35090-02

Top assembly revision number : B0

Version ID : V02

CLEI Code Number : CMMFG00ARB

License Level : ipbase

License Type : Permanent

Next reboot licensing Level : ipbase

Switch 05

---------

Switch Uptime : 6 days, 16 hours, 54 minutes

Base ethernet MAC Address : 6C:20:56:0C:07:80

Motherboard assembly number : 73-13329-03

Motherboard serial number : Fxxxxxxxx

Model revision number : A0

Motherboard revision number : B0

Model number : WS-C3750X-12S-S

Daughterboard assembly number : 800-32727-03

Daughterboard serial number : Fxxxxxxxxx

System serial number : FDxxxxxxxx

Top assembly part number : 800-35090-02

Top assembly revision number : B0

Version ID : V02

CLEI Code Number : CMMFG00ARB

License Level : ipbase

License Type : Permanent

Next reboot licensing Level : ipbase

Configuration register is 0xF

Wilson Bonilla · ‎02-28-2013

Mohit,

Let's do the following: monitor the stack for 24hours or so, just to check if the high cpu utilization comes back, I would say by now the switch should be stable after the configuration changes.

Remember based on Cisco Docs the CPU utilizacion baseline is 50%.

Keep an eye on the performance and show process cpu, show log and let me know how it goes.

Wilson.