About Preston Chilcote

Preston Chilcote · ‎02-28-2018

There is a command to increase the timeout. snmp monitor response [msec] Depending on how long this OID is being delayed in your setup, you may see issues with your NMS station. For example, if the ciscoFlashFileEntry OID takes more than say 30 seconds, then subsequent OIDs could be dropped or, in some cases, the NMS server, getting no response, will just timeout completely and declare the device "dead". So it may not be as cosmetic as you think. I imagine the NMS timeout is on the order of 30 seconds though, not just 2.

Preston Chilcote · ‎05-18-2017

This blog aims to explain what do if you see SNMP-3-INPUT_QFULL_ERR or SNMP-3-RESPONSE_DELAYED errors. There is a TL;DR toward the bottom. Working in TAC for over 10 years now, I have grown tired of seeing so many cases with this error message: %SNMP-3-INPUT_QFULL_ERR: Packet dropped due to input Very often, I saw TAC engineers giving the wrong info about what these errors are and how to deal with them. For example, the common recommendations: 1) Remove and then reconfigure SNMP configs - Problem: While this is likely to get rid of the problem in the short term, it does nothing to solve the problem for good. That's not really what Cisco is aiming for. 2) Use the "snmp queue-length" command - Problem: This is actually guaranteed NOT to work, since the queue being increased is the one used for snmp traps leaving the box, not the processing of SNMP requests received by the box. 3) Don't poll the device so often or with as many SNMP servers - Problem: It's natural to think that a queue filling up (as the error describes) could be due to congestion of too much polling. Maybe this was the root cause when CPUs were slower, but in my lab testing, I showed that modern Cisco platforms (those 5 years old or newer) do NOT suffer from having up to 3 (what I tested) servers continuously polling. 4) Find the guilty OID (which only TAC can do) then block it in an snmp-view - Problem: First, this requires customers to open a case with TAC, so there's no hope of solving the issue with a simple search in this or other support communities. As a user myself, I love it when a problem I'm seeing is already documented online, with the final fix given to me. Second, blocking the OID will have an impact on network management stations that may have good reason to fetch the OID. I took a closer look at as many cases as I could find to figure out how to make this class of problem easier to solve for all of us users. My research showed that there are a fairly small set of software bugs that address these issues, usually when an OID takes multiple seconds to process, rather than just milliseconds in the routine case. Generally each bug is uniquely identified by platform and OID. For example, if you see INPUT_QFULL_ERR messages on a 3850 running 3.2 code you are probably hitting, CSCuo12316. That's still not very user friendly. I wanted something easier and more reliable. I reasoned that if the error itself could tell us what the slow OID was, then a simple string matching search could identify the bug that fixes the issue (I don't think I found any cases where the root cause was hardware failure). The new feature to do this, added by CSCuz93302, is called SNMP monitoring, and will print an error similar to the following in cases where an SNMP request takes an IOS/IOS-XE device more than 2 seconds (by default) to process: %SNMP-3-RESPONSE_DELAYED: processing GetNext of ciscoFlashFileEntry.5.1.1.80 We tested this monitoring feature in a VERY high scale lab without seeing any of the above, which gives us confidence that the errors should not be seen in a standard setup. If you see an error like this, and can't find an existing bug from your own searching, then please open a TAC case so that Cisco can investigate the root cause and fix. Some additional useful information would be: - is the problem seen if you do a manual "snmpwalk" on the same OID from a server with snmpwalk installed. - is there high cpu at the same time the logs are seen. If the CPU is high for the SNMP ENGINE process, then that will be great for TAC to know. But, if the CPU is high, without the SNMP ENGINE process being high, then the slow snmp response is probably just a symptom of the system being oversubscribed (in the case of interrupt cpu) or non-optimized (in the case of another process showing high cpu). If CPU isn't high at all, that's still probably a software bug, perhaps on a linecard. If you see a lot of different OIDs in your messages, then the root cause is likely not with any specific OID, but rather with the the device being busy doing other, non-snmp work. This is my first post in a series I'm planning about IOS/IOS-XE serviceability improvements that I or others in TAC have been working on to make common problems easier to solve. Please feel free to leave feedback about this post, and ideas for improvements to Cisco Software that would save you lots of time. If anyone can help push those fixes into reality, it's your friendly, helpful TAC engineers. TL;DR, if you see an error starting with "SNMP-3-RESPONSE_DELAYED" then search the internet for Cisco bugs where the name of the OID in the bug matches the one in your error message. If none found, or the bug you found is unfixed, then open a TAC case and have them investigate. These problems are usually software bugs that can be fixed permanently.

Preston Chilcote · ‎02-27-2017

This is exactly why I pushed to have the error message be more specific. There are many bugs that lead to this problem. Off the top of my head, I don't know of another one for 3850, but if you open a TAC case they can lead you through the troubleshooting steps to isolate which bug. If they ask you do perform a workaround (like restarting SNMP), beware that the problem will most likely come back later, so it's important that they use this opportunity, while the problem is happening, to collect the necessary information to pinpoint the bug.

Preston Chilcote · ‎02-20-2017

I don't know of any reason to do incremental updates for 3850, so I think you should be able to just upgrade straight to the new version.

Preston Chilcote · ‎02-14-2017

Mark is right, but we don't call it "safe harbour" anymore. Rather, just "recommended release", which are those with a gold star on the download page. The bug for 3850 is CSCuo12316, which hasn't been seen in 3.3 or later. 3.6 is the recommended release, currently, for this platform.

Preston Chilcote · ‎06-07-2016

After investigating lots of TAC cases for this error, it seems the root cause is almost never related to too many snmp polls. The queue size is fixed at 1000 (it cannot be changed, even with the snmp queue-length command) which should be plenty for most cases. The reason for the queue backing up is most commonly a specific OID, taking too long, which causes all subsequent snmp requests to backup on the queue. There are ways for TAC to find the guilty OID, and there are workarounds, as discussed on the thread, but users should beware of blocking the guilty OID's, because it will prevent NMS stations from getting full data from the Cisco devices. Restarting the snmp process is only a temporary measure, and the problem is expected to return eventually. We are working on an enhancement, CSCuz93302, initially for catalyst 6500 that will make this error less vague and identify the SNMP that is taking too long to run. That should allow users to solve these cases reliably through google searches.

Preston Chilcote · ‎06-07-2016

I know this is an old thread, but it's the first that comes up in my google search, so I'm updating it to get the most attention. After investigating lots of TAC cases for this error, it seems the root cause is almost never related to too many snmp polls. The queue size is fixed at 1000 (it cannot be changed, even with the snmp queue-length command) which should be plenty for most cases. The reason for the queue backing up is most commonly a specific OID, taking too long, which causes all subsequent snmp requests to backup on the queue. There are ways for TAC to find the guilty OID, and there are workarounds, as discussed on the thread, but users should beware of blocking the guilty OID's, because it will prevent NMS stations from getting full data from the Cisco devices. Restarting the snmp process is only a temporary measure, and the problem is expected to return eventually. We are working on an enhancement, CSCuz93302, initially for catalyst 6500 that will make this error less vague and identify the SNMP that is taking too long to run. That should allow users to solve these cases reliably through google searches.

Preston Chilcote · ‎10-27-2015

Specifically this is CSCue55762 Kernel bug soft lockup - CPU stuck after about 200 days of uptime Note the absurd amount of time the cpu was stuck for: 17163091973s. That's how you'll know you're matching this bug.

Preston Chilcote · ‎12-24-2014

Turbo ACL's were designed for software forwarding platforms. ASR's handle all the ACL's (and all other data plane features) in hardware, so they don't need the optimizations that Turbo ACL's used. No need to worry about Turbo ACL anymore.

Preston Chilcote · ‎12-24-2014

Have you ever successfully done an http upgrade on this device or a similar switch? I ask because, the first thing to think about is whether the http download is working ok. Maybe you could try tftp or ftp instead to see if that's the issue.

Preston Chilcote · ‎12-24-2014

A traceback alone doesn't mean you have a bug, but a MALLOCFAIL error seen after the router has been up for a long time indicates that the device has a memory leak which causes it to run out of memory. MALLOCFAIL errors are generated when an application on the router asks for memory and is denied because not enough is available. Leo is right though, that you should try upgrading to an M release, like 15.2(4)M or 15.3M if the device has enough RAM and flash. You can use "show mem stats" to watch the amount of free memory in the Processor pool to make sure it's not leaking anymore.

Preston Chilcote · ‎06-20-2014

Check your SNMP configs. TAC has confirmed that this behavior is seen on devices configured with easy to guess SNMP strings (like "private" and "public") and RW access. Please see http://www.cisco.com/c/en/us/support/docs/ip/access-lists/13608-21.html#anc50 for tips on how to harden SNMP. The ipForwarding MIB allows users to disable routing and forwarding with an snmpset.

Preston Chilcote · ‎06-20-2014

Those must be coming from the boot image, which doesn't have all the feature support as the regular IOS image. The messages are just warnings when configuring some of the features that the boot images doesn't understand.

Preston Chilcote · ‎06-20-2014

Timmy has the right answer. TAC has confirmed that this behavior is seen with routers configured with easy to guess SNMP strings (like "private" and "public) and RW access. Please see http://www.cisco.com/c/en/us/support/docs/ip/access-lists/13608-21.html#anc50 for tips on how to harden SNMP. The ipForwarding MIB allows users to disable routing and forwarding with an snmpset.

Preston Chilcote · ‎10-31-2013

The platforms share the same stacking code, so the bug applies to cat2k and 3k.

Date Registered	‎05-17-2005 07:58 AM
Date Last Visited	‎06-27-2018 12:18 AM
Total Messages Posted	59
Total Helpful Votes Received	63

Re: CSCve12817 - percentSNMP-SW1-3-RES...

Years of wasted effort troubleshooting ...

This is exactly why I pushed

I don't know of any reason to

Mark is right, but we don't

After investigating lots of

I know this is an old thread,

Specifically this is

Turbo ACL's were designed for

Have you ever successfully

A traceback alone doesn't

Check your SNMP configs. TAC

Those must be coming from the

Timmy has the right answer.

Question About "DDTS:CSCub24886"