CSCtg62793 - CPU hog on Cat3xxx for SNMP lldpRemEntry - 1

[Nicolas Rennetaud]

Hi,

We are encountering the same issue as this bug but on Cisco 3850 version 03.07.00.

We tried to implement the workaround precised in the bug but this doesn't work at all. Also there's no recommended version where bug is fixed.

Do you already had encountered this issue ? thanks in advance.

[Mark Malone]

Hey

you sure that's the bug your hitting as i know that fix works on a 3750, that's an IOS bug too I dont see any IOS-XE versions there

I do see known bugs though out there for 3850 cpu hog on IOS-XE

CSCuq22460

COMMON-1-WDOG_CPUHOG: 1 fed: CPU usage time exceeded

CSCut27272

CPUHOG and crash due to Auth Manager process

CSCut87425

CPU hog in \"EEM TCL Proc\" after TCL script termination with long runtime

Can you post the full alert you see in logs please

You're also running an ED edition of IOS-XE which you should always avoid as there known for more bugs, you should be running 3.6.3E MD main deployment version safe harbour more thoroughly tested

[Nicolas Rennetaud]

Hi Mark,

First thanks for your fast answer.

I agree with you that the bug CSCtg62793 isn't referenced on IOS XE but on Cat 3XXX so maybe ..?

Also You will find attached the log file.

In fact, the site where the switch in installed was complaining of network issue all the day with a lot of small disruption. logs in the file are exactly the same as the bug. 

Is it possible that this bug hasn't been discovered on these versions ?

[Mark Malone]

Hi Nicolas Its very possible its a cross platform bug that's effecting IOS and IOS-XE that I have not seen before usually the bug states it like the one below , the bug below effects multiple platforms , maybe try this fix its slightly diff than yours and I think its for ios-xe looking through the other bugs

CSCtg57599

Symptom: Lots of SNMP CPUHOG messages are seen and there is a crash due to a watchdog timeout:

%SYS-3-CPUHOG: Task is running for (126004)msecs, more than (2000)msecs (252/37),process =SNMP ENGINE

%SYS-2-WATCHDOG: Process aborted on watchdog timeout, process = SNMP ENGINE

Conditions: Device is configured with SNMP and is polled for Dot3Stats.

Workaround 1: Use the following command: no snmp-server sparse-tables.

Workaround 2: Block the objects in dot3 mib that contains this table from being polled:

snmp-server view cutdown iso included

snmp-server view cutdown 1.3.6.1.2.1.10.7 excluded

Then to apply the view, use:

no snmp-server community your_string_here RO

no snmp-server community your_string_here RW

and then put it back so it looks like:

snmp-server community your_string_here view cutdown RO

snmp-server community your_string_here view cutdown RW

Further Problem Description: Cisco IOS Software contains a vulnerability that could allow an authenticated, remote attacker to trigger a high CPU on the device via walking specific SNMP objects.

The vulnerability is due to an uninitialized variable in the code. An attacker could exploit this vulnerability by performing SNMP walks against objects on the affected device. An exploit could allow the attacker to cause high CPU on the affected devices.

http://www.cisco.com/c/en/us/td/docs/ios/15_2s/release/notes/15_2s_rel_notes/15_2s_caveats_15_2_4s.html

[Nicolas Rennetaud]

Hi Mark

I just tried to implement the two workaround on switch but it's not good.

As soon as I configure the switch, SNMP logs are seen on console.

Thanks

Regards

[Mark Malone]

Only other option is to move to the S.H 3.6.3 as its newer and more stable or else go through TAC but they will prob tell you too upgrade anyway off the ED

[Nicolas Rennetaud]

I've just opened a SR at TAC.Waiting for their answer.

Thanks for your help Mark.

Best regards.

[Mark Malone]

ah good stuff if you don't mind share the fix they provide as others may run into your issue too

[Claudio Gonzalez Sequeira]

Hello, 

checking the logs we can see the following error messages :

Jan 4 12:30:50: %SNMP-3-CPUHOG: Processing GetBulk of bsnMobileStationAuthenticationAlgorithm
Jan 4 12:30:52: %SYS-3-CPUHOG: Task is running for (2150)msecs, more than (2000)msecs (1073741827/1073741827),process = SNMP ENGINE.
Jan 4 12:35:48: %SNMP-3-CPUHOG: Processing GetBulk of bsnMobileStationAuthenticationAlgorithm
Jan 4 12:35:50: %SYS-3-CPUHOG: Task is running for (2550)msecs, more than (2000)msecs (0/0),process = SNMP ENGINE.

here as you can see the problem is with "bsnMobileStationAuthenticationAlgorithm" not "lldpRemEntry" the workaround is the same but for different OID : 

Exclude the OID bsnMobileStationAuthenticationAlgorithm from snmp polling. Please configure the following:

snmp-server view cutdown iso included
snmp-server view cutdown 1.3.6.1.4.1.14179 excluded

Then assign this SNMP view to all your existing SNMP communities:
no snmp-server community (your string here) RO
snmp-server community (your string here) view cutdown RO

That should work, hope it helps!

Claudio Gonzalez S