Solved: LMS 3.2 causing spikes on routers

kmong · ‎07-18-2010

Hi everyone,

A customer reported that their router experienced spikes (high cpu utilization) and claims that it is caused by snmp polling of the Ciscoworks server. What must be done to confirm that it is caused by snmp polling of LMS? If it is really caused by LMS, what values should be set on the polling parameters and threshold?

Thanks

Joe Clarke · ‎07-19-2010

Looks like the problem is related to VNM polling the MPLS-VPN-MIB. If you are not using VNM, you can filter out this branch by applying an SNMP view to the community string used by LMS. For example:

snmp-server view nomplsvpn iso included

snmp-server view nomplsvpn 1.3.6.1.3.118 excluded

snmp-server community mycomm view nomplsvpn RO

To fix this (bug CSCsl78965), you would need to upgrade to 12.2(18)SXF15 or higher.

View solution in original post

Joe Clarke · ‎07-18-2010

When the CPU is spiking, you need to capture the output of "show stack PID" where PID is the process ID of the process which is taking up the CPU (probably either SNMP ENGINE or IP SNMP). Post this stack trace along with the show version output. This could be a known bug, or simply an issue with over-polling or polling a large amount of data.

kmong · ‎07-18-2010

If its polling a large amount of data, what should be done to avoid spikes on the cpu? I'll be posting the output once I've received it.

Joe Clarke · ‎07-18-2010

The polling intervals can be changed, or the data can be spread out over multiple polls.

kmong · ‎07-18-2010

Polling parameters and thresholds in DFM are still in its default values. What would be the ideal settings? Where else can we change the polling intervals?

Joe Clarke · ‎07-18-2010

It will depend on what part of LMS is causing the CPU spikes.

kmong · ‎07-18-2010

How will I know which LMS process is causing the spikes? As I have mentioned, the polling parameters and thresholds are just in default value.

Joe Clarke · ‎07-18-2010

That's what the show stack output will indicate. The objects being polled will indicate which part of LMS is doing the polling.

kmong · ‎07-18-2010

Ok. This is noted. I will send the output once I receive it from the client. We have no access on their routers

kmong · ‎07-19-2010

Hi Joseph,

Please see attached output of show version and show stack. Client has reported that the spikes occur every 4 hours (0000H 0400H 0800H so on..)

VAL-IPB-RTR-003#sh ver

Cisco Internetwork Operating System Software

IOS (tm) s72033_rp Software (s72033_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(18)SXF4, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Thu 23-Mar-06 19:38 by tinhuang

Image text-base: 0x40101040, data-base: 0x42DA8000

ROM: System Bootstrap, Version 12.2(17r)S2, RELEASE SOFTWARE (fc1)

BOOTLDR: s72033_rp Software (s72033_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(18)SXF4, RELEASE SOFTWARE (fc1)

VAL-IPB-RTR-003 uptime is 4 years, 3 weeks, 1 day, 11 hours, 17 minutes

Time since VAL-IPB-RTR-003 switched to active is 4 years, 3 weeks, 1 day, 11 hours, 34 minutes

System returned to ROM by unknown reload cause - suspect boot_data[BOOT_COUNT] 0x0, BOOT_COUNT 0, BOOTDATA 19 (SP by power on)

System restarted at 04:22:14 GMT Wed Jun 28 2006

System image file is "disk1:s72033-adventerprisek9_wan-mz.122-18.SXF4.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco CISCO7609 (R7000) processor (revision 1.1) with 983008K/65536K bytes of memory.

Processor board ID FOX0835049C

SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache

Last reset from power-on

X.25 software, Version 3.0.0.

Bridging software.

TN3270 Emulation software.

39 Virtual Ethernet/IEEE 802.3 interfaces

236 Gigabit Ethernet/IEEE 802.3 interfaces

1917K bytes of non-volatile configuration memory.

8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).

Configuration register is 0x2102

VAL-IPB-RTR-003#

VAL-IPB-RTR-003# sh proc cpu | i CPU

CPU utilization for five seconds: 99%/1%; one minute: 99%; five minutes: 72%

220 0 2 0 0.00% 0.00% 0.00% 0 SLB Main CPU pro

277 40076 25601858 1 0.00% 0.00% 0.00% 0 Max CPU utilizat

VAL-IPB-RTR-003#sh proc cpu | i SNMP

29 16 107 149 0.00% 0.00% 0.00% 0 Cat6k SNMP

30 1244 1117 1113 0.00% 0.00% 0.00% 0 Cat6k SNMP Trap

158 0 126 0 0.00% 0.00% 0.00% 0 SNMP Timers

302 0 2 0 0.00% 0.00% 0.00% 0 EEM ED SNMP

311 39506708 321176354 123 0.07% 0.08% 0.02% 0 IP SNMP

316 367797992 401044214 917 94.90% 30.84% 7.60% 0 SNMP ENGINE

317 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro

318 1266468 250441 5056 0.00% 0.00% 0.00% 0 SNMP Traps

VAL-IPB-RTR-003#sh stack 316

Process 316: SNMP ENGINE

Stack segment 0x52B6DC78 - 0x52B70B58

FP: 0x52B70528, RA: 0x40FE1228

FP: 0x52B70550, RA: 0x40934C50

FP: 0x52B70578, RA: 0x40934D80

FP: 0x52B705B0, RA: 0x40937DC4

FP: 0x52B70608, RA: 0x41FC8038

FP: 0x52B70680, RA: 0x41FC8C0C

FP: 0x52B706F0, RA: 0x41FC7084

FP: 0x52B70710, RA: 0x41FC6214

FP: 0x52B70988, RA: 0x40E4FDF8

FP: 0x52B709D0, RA: 0x40E300CC

FP: 0x52B70A38, RA: 0x40E23184

FP: 0x52B70AB8, RA: 0x40E477C0

FP: 0x52B70B20, RA: 0x40FD750C

FP: 0x52B70B38, RA: 0x40FD74F8

VAL-IPB-RTR-003#

Joe Clarke · ‎07-19-2010

Looks like the problem is related to VNM polling the MPLS-VPN-MIB. If you are not using VNM, you can filter out this branch by applying an SNMP view to the community string used by LMS. For example:

snmp-server view nomplsvpn iso included

snmp-server view nomplsvpn 1.3.6.1.3.118 excluded

snmp-server community mycomm view nomplsvpn RO

To fix this (bug CSCsl78965), you would need to upgrade to 12.2(18)SXF15 or higher.

kmong · ‎07-19-2010

Thank you Sir.

sassi-zied · ‎09-05-2011

Hi Joseph,

Could you please tell us how did you manage to idenfiy that the problem is related to VNM Mib ? Is it through the show snmp stack command ? i

Thank you

Zied

sassi-zied · ‎09-05-2011

Hi Joseph,

Just to explain we have exactly the same problem. Below is the show stack. Could this be related to the same VNM pollin

*Sep 5 12:02:43.230 GMT+1: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 56%/14%, Top 3 processes(Pid/Util): 557/39%, 488/1%, 555/0%

*Sep 5 12:02:43.286 GMT+1: %HA_EM-4-LOG: CPUTH:

Process 557: SNMP ENGINE

Stack segment 0x1CFC204C - 0x1CFC4F2C

FP: 0x1CFC49F0, RA: 0x9993528

FP: 0x1CFC4A10, RA: 0x8C04F84

FP: 0x1CFC4A20, RA: 0x8BEC170

FP: 0x1CFC4A50, RA: 0xAB76E48

FP: 0x1CFC4A58, RA: 0xAB773F8

FP: 0x1CFC4A68, RA: 0xAB7776C

FP: 0x1CFC4AA0, RA: 0xAE27A84

FP: 0x1CFC4AB8, RA: 0xAFC75F4

FP: 0x1CFC4AC0, RA: 0xA59A7AC

FP: 0x1CFC4CF8, RA: 0xA5A38A8

FP: 0x1CFC4D30, RA: 0xA59F194

FP: 0x1CFC4DA0, RA: 0x973D4C0

FP: 0x1CFC4E08, RA: 0x970BA00

FP: 0x1CFC4E90, RA: 0x96F711C

FP: 0x1CFC4F18, RA: 0x9729654

FP: 0x1CFC4F20, RA: 0xAFAFF94

FP: 0x0, RA: 0xAFAA270

Thank you

Joe Clarke · ‎09-05-2011

Please start a new thread for your issue. When you do, include the output of "show version".