SNMP drop by AIPC

[ReyMartin1]

We have a few NCS540 running different OS versions experiencing SNMP query timeout.

If we do snmpwalk, it always timeout at the same spot after  the first few lines:

SNMPv2-MIB::sysDescr.0 = STRING: Cisco IOS XR Software (NCS540L), Version 7.4.2 Copyright (c) 2013-2022 by Cisco Systems, Inc.
SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.9.1.3143
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (692600407) 80 days, 3:53:24.07
SNMPv2-MIB::sysContact.0 = STRING:
SNMPv2-MIB::sysName.0 = STRING: xxxxxx
SNMPv2-MIB::sysLocation.0 = STRING: yyyy
SNMPv2-MIB::sysServices.0 = INTEGER: 78
Timeout: No Response from zzzzzz

It seems being dropped by AIPC module (AIPC counter went up by 6 for each request), what is it?

RP/0/RP0/CPU0:xxxxx#sh snmp request drop summary | inc "NMS|y.y.y.y"
Thu Jun 22 16:48:44.040 UTC
NMS Address INQ Encode Duplicate Stack AIPC Overload Timeout Internal Threshold
y.y.y.y 2 649 0 230654 30 0 0 22 0

[ReyMartin1]

Any chance it's this one? CSCve04643  

[tkarnani]

Please check "show snmp trace packet" << check if you are seeing errors
please also check 'show snmp trace slow oid" << check to see if there are slow oid's taking a long time to complete

[ReyMartin1]

Thanks for the reply tkarnani,

nothing on 'sh snmp trace slow oid' (lastest entries few days old).

This is from sh snmp trace packet for the failed requests:

Jun 22 16:48:32.758 snmp/packet 0/RP0/CPU0 10803# t8704 ERROR: Internal Dispatch failed for y.y.y.y,15582 (reqID 1238439839) 'Subsystem(8123)' detected the 'resource not available' condition 'Code(7)'
Jun 22 16:48:33.759 snmp/packet 0/RP0/CPU0 7550# t2001 ERROR: Internal Dispatch failed for y.y.y.y,15582 (reqID 1238439839) 'Subsystem(8123)' detected the 'resource not available' condition 'Code(7)'
Jun 22 16:48:34.761 snmp/packet 0/RP0/CPU0 10806# t1504 ERROR: Internal Dispatch failed for y.y.y.y,15582 (reqID 1238439839) 'Subsystem(8123)' detected the 'resource not available' condition 'Code(7)'
Jun 22 16:48:35.762 snmp/packet 0/RP0/CPU0 t8704 ERROR: Internal Dispatch failed for y.y.y.y,15582 (reqID 1238439839) 'Subsystem(8123)' detected the 'resource not available' condition 'Code(7)'
Jun 22 16:48:36.762 snmp/packet 0/RP0/CPU0 14158# t1504 ERROR: Internal Dispatch failed for y.y.y.y,15582 (reqID 1238439839) 'Subsystem(8123)' detected the 'resource not available' condition 'Code(7)'
Jun 22 16:48:37.765 snmp/packet 0/RP0/CPU0 7557# t2001 ERROR: Internal Dispatch failed for y.y.y.y,15582 (reqID 1238439839) 'Subsystem(8123)' detected the 'resource not available' condition 'Code(7)'

[smilstea · Accepted solution]

The bug you mentioned is fixed in your code.

I saw you opened a tac sr but there isnt much data in there, a show snmp trace timeout would help to see which OID is timing out.

Can you also make sure you have this line configured? 'snmp-server ifmib stats cache'

Sam

[ReyMartin1]

Thanks for the follow up, I've uploaded the command output but from a different device (unfortunately my colleague just restarted the snmp process on the exact device). added the 'snmp-server ifmib stats cache' as well.

Btw any OIDs except the first few lines are all timed out (showing dropped by AIPC) though.

[smilstea]

over 90% of the timeouts are due to ifmib, adding that cli i mentioned will help/solve the problem.

https://snmp.cloudapps.cisco.com/Support/SNMP/do/BrowseOID.do?objectInput=1.3.6.1.2.1.2.2.1.2&translate=Translate&submitValue=SUBMIT&submitClicked=true

Sam

[ReyMartin1]

Ah got it so the objective is cache the stats to help the system resources, ok I'll monitor it over the next few days. Thanks.

[smilstea]

Real time polling of many interfaces all at once is very resource intensive, instead we cache the stats with this cli and read from cache which is much faster, I believe the update period is 30 or 60s. Its one of those knobs we recommend for basically every deployment that uses snmp.

Sam