snmpwalk crashes ASA-SSC-AIP-5!?!?

[Mark^]

Setting up some snmp monitoring and as I am poking around to figure out what exactly I'd like to get, I run the following commands from my Debian based monitoring host:

• snmpwalk -v 2c -c Community_String AIP_IPADDRESS
• snmpwalk -v 1 -c Community_String AIP_IPADDRESS

While running either of these commands it begins to spit out OID information and then each time it ends with these two lines:

IF-MIB::ifSpecific.5 = OID: SNMPv2-SMI::zeroDotZero
Timeout: No Response from AIP_IPADDRESS

And at that moment, the magic happens; the module no longer responds, pings time-out, and the ASA5505 needs to be physically power cycled for everything to come back up normally.

ASA syslog reports:

May  9 10:34:15 ASA_IP May 09 2011 10:34:15 HOSTNAME : %ASA-1-505005: Module in slot 1 is initializing control communication.  Please wait...
May  9 10:34:25 ASA_IP May 09 2011 10:34:25 HOSTNAME : %ASA-1-323001: Module in slot 1 experienced a control channel communication failure.
May  9 10:34:28 ASA_IP May 09 2011 10:34:28 HOSTNAME : %ASA-1-505005: Module in slot 1 is initializing control communication.  Please wait...
May  9 10:34:38 ASA_IP May 09 2011 10:34:38 HOSTNAME : %ASA-1-323001: Module in slot 1 experienced a control channel communication failure.

(as you can see, these pairs are 10 seconds apart until the ASA is rebooted)

Is this expected behavior or otherwise already documented?

Let me add that this is not happening for the ASA or any other snmp devices.

As one could imagine -- this is driving me nuts.

[Christopher Dreier]

Hello Mark,

I just tested this locally and I believe that I was able to replicate what you are experiencing. This may be due to a known defect. Can you please run a "show tech" on your SSC-5 (after power-cycling the ASA) and post the information from the core.txt output in the show tech? This will confirm whether I am experiencing what you are experiencing and help to correlate this issue to known defects.

Alternatively, you can open a TAC case and we can carry out the investigation there.

Thank you,

Blayne Dreier

Cisco TAC Escalation Team

**Please check out our Podcasts**

TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast

TAC IPS Media Series: https://supportforums.cisco.com/docs/DOC-12758

[Mark^]

Is this what you are looking for?  I hadn't power cycled the ASA since yesterday, so if that is going to make a difference I can re-post this or open a TAC case.

Thanks.

-----

exec: ls -l /usr/cids/idsRoot/core/mainApp/core.txt
-rw-rw-rw-    1 cids     cids         2408 May  9 11:16 /usr/cids/idsRoot/core/mainApp/core.txt
exec: cat /usr/cids/idsRoot/core/mainApp/core.txt
Application thread 462 received trap: 11
--------------------------------------------------------------
eax   0xffffffff  -1
ebx   0x408be86c  1082910828
edx   0x00000000  0
ecx   0x00000001  1
edi   0x41600010  1096810512
esi   0x00000080  128
eip   0x4080f646  1082193478
ebp   0x43db1cb4  1138433204
esp   0x43db1c8c  1138433164
cs    0x00000023  35
es    0x0000002b  43
ds    0xc010002b  -1072693205
gs    0x000000d7  215
fs    0x00000000  0
ss    0x0000002b  43
efl       0x00010202  66050
uesp      0x43db1c8c  1138433164
trapno    0x0000000e  14
err       0x00000004  4
--------------------------------------------------------------
0x0x4080f646   +/lib/libc.so.6(calloc+0xd6) [0x4080f646];
0x0x407c38f0   +/lib/libc.so.6 [0x407c38f0];
0x0x4044e57d   +/lib/libnetsnmpmibs.so.9(Interface_Scan_Init+0x4ad) [0x4044e57d];
0x0x4044f161   +/lib/libnetsnmpmibs.so.9(Interface_Index_By_Name+0x1d) [0x4044f161];
0x0x4044d90f   +/lib/libnetsnmpmibs.so.9 [0x4044d90f];
0x0x4044d1f5   +/lib/libnetsnmpmibs.so.9(var_atEntry+0x83) [0x4044d1f5];
0x0x4041de42   +/lib/libnetsnmphelpers.so.9(netsnmp_old_api_helper+0x1e2) [0x4041de42];
0x0x40439bfa   +/lib/libnetsnmpagent.so.9(netsnmp_call_handler+0xb5) [0x40439bfa];
0x0x40439dad   +/lib/libnetsnmpagent.so.9(netsnmp_call_handlers+0x160) [0x40439dad];
0x0x40431922   +/lib/libnetsnmpagent.so.9(handle_var_requests+0xc7) [0x40431922];
0x0x40431fb5   +/lib/libnetsnmpagent.so.9(handle_getnext_loop+0x64) [0x40431fb5];
0x0x404324dd   +/lib/libnetsnmpagent.so.9(handle_pdu+0x268) [0x404324dd];
0x0x40432231   +/lib/libnetsnmpagent.so.9(netsnmp_handle_request+0x8b) [0x40432231];
0x0x40430a23   +/lib/libnetsnmpagent.so.9(handle_snmp_packet+0x1d2) [0x40430a23];
0x0x4049b41d   +/lib/libnetsnmp.so.9 [0x4049b41d];
0x0x4049bdc1   +/lib/libnetsnmp.so.9(_sess_read+0x882) [0x4049bdc1];
0x0x4049be10   +/lib/libnetsnmp.so.9(snmp_sess_read+0x24) [0x4049be10];
0x0x4049b52f   +/lib/libnetsnmp.so.9(snmp_read+0x35) [0x4049b52f];
0x0x4042ea84   +/lib/libnetsnmpagent.so.9(agent_check_and_process+0x12a) [0x4042ea84];
0x0x831ea7c   +/usr/cids/idsRoot/bin/mainApp(_ZN3Cid12Notification9SnmpAgent9agentTaskEPNS_2Mt12ThreadedTaskEPv+0x40) [0x831ea7c];
0x0x4008e7db   +/usr/cids/idsRoot/lib/libcidcore.002.041.so(_ZN3Cid2Mt12ThreadedTask11threadStartEPv+0x901) [0x4008e7db];
0x0x40024004   +/lib/libpthread.so.0 [0x40024004];
0x0x4087287a   +/lib/libc.so.6(clone+0x3a) [0x4087287a];
exec: ps -ew f
  PID TTY      STAT   TIME COMMAND
    1 ?        S      0:28 init
    2 ?        S      0:00 [keventd]
    3 ?        SN     0:00 [ksoftirqd_CPU0]
    4 ?        S      0:00 [kswapd]
    5 ?        S      0:00 [bdflush]
    6 ?        S      0:00 [kupdated]
   50 ?        S      0:00 [kjournald]
   75 ?        S      0:00 [kjournald]
  107 ?        Ss     0:00 /sbin/syslogd -m 0
  110 ?        Ss     0:00 /sbin/klogd
  122 ?        Ss     0:00 /usr/sbin/inetd
  126 ?        Ss     0:00 /sbin/sshd
14738 ?        Ss     0:02  \_ sshd: cisco@pts/0
14757 pts/0    Ss+    0:01      \_ -cidcli
14759 pts/0    S+     0:00          \_ -cidcli
14760 pts/0    SN+    0:05              \_ -cidcli
14768 pts/0    SN+    0:00              \_ -cidcli
15706 pts/0    SN+    0:00              \_ -cidcli
  317 ?        S<     0:00 /usr/cids/idsRoot/bin/SSM_control_proc
  341 ?        Ss     0:02 /usr/cids/idsRoot/bin/mainApp -d -c 0
  344 ?        S      0:02  \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  345 ?        SN     0:18      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  461 ?        SN     0:01      |   \_ /usr/cids/idsRoot/bin/sensorApp -z 345
  485 ?        SN     0:00      |       \_ /usr/cids/idsRoot/bin/sensorApp -z 345
  486 ?        SN     4:05      |           \_ /usr/cids/idsRoot/bin/sensorApp -z 345
  504 ?        SN     0:00      |           \_ /usr/cids/idsRoot/bin/sensorApp -z 345
  926 ?        S<    45:57      |           \_ /usr/cids/idsRoot/bin/sensorApp -z 345
  346 ?        S      1:09      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  412 ?        SN    15:07      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
15707 ?        SN     0:00      |   \_ /bin/bash /usr/cids/idsRoot/bin/cidDump -text -wxml -nostatus -stdout
15768 ?        RN     0:00      |       \_ ps -ew f
  413 ?        S      0:00      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  419 ?        S      0:10      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  433 ?        SN     0:02      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  434 ?        SN     0:00      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  435 ?        SN     0:00      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  436 ?        SN     0:00      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  437 ?        SN     0:00      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  438 ?        SN     0:06      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  439 ?        SN     0:06      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  440 ?        SN     0:06      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  441 ?        SN     0:06      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  442 ?        SN     0:06      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  443 ?        SN     0:07      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  444 ?        SN     0:06      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  445 ?        SN     0:06      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  446 ?        SN     0:08      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  447 ?        SN     0:07      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  448 ?        SN     0:01      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  451 ?        SN     3:35      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  452 ?        SN     0:00      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  462 ?        SN     0:01      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  463 ?        RN     0:11      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
15753 ?        SN     0:00      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
15754 ?        SN     0:00      \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
  383 tty1     Ss+    0:00 /sbin/getty 38400 tty1
  384 tty2     Ss+    0:00 /sbin/getty 38400 tty2
  385 ttyS0    Ss+    0:00 /sbin/getty -L ttyS0 9600 vt100
  425 ?        SNLs   1:34 ntpd

-----

[Christopher Dreier]

Hello Mark,

Yes, that's what I needed. You are experiencing CSCti03741: mainApp crash on Interface_Scan_Init while doing an SNMP walk

You can track the progress of this bug via the CCO Bug Toolkit: http://tools.cisco.com/Support/BugToolKit/action.do?hdnAction=searchBugs If this bug fix is important to you, I do suggest contacting your Cisco Account Team so that they can convey your sentiment and drive this bug to resolution.

Please let me know if I can help you with anything further within the context of this thread. If your question has been Answered, please mark the thread as such so that it will be helpful to other users. Also, please feel free to Rate this thread to reflect your experience.

Thank you,

Blayne Dreier

Cisco TAC Escalation Team

**Please check out our Podcasts**

TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast

TAC IPS Media Series: https://supportforums.cisco.com/docs/DOC-12758

[Mark^]

Blayne,

Thank you!  As I said, it was driving me nuts.  Now I can be satisfied that it is not just me.

I did read the bug, but am trying to understand it.  Basically, is it only the SNMP walk that causes it to crash?  Meaning; I can still get specific SNMP info as long as I know the OID without crashing the mainApp?

[Christopher Dreier]

Hello Mark,

I've been testing this quite a while and I have not experienced the failure while doing an snmpget (instead of an snmpwalk) for a particular OID. I have a script running now that polls the interface OIDs in a loop, using both v1 and v2c.

The bug was written with the observation that an snmpwalk triggers the issue. However, it does not explicitly exclude snmpgets from causing the issue. I'll keep my script running and if it does crash, I'll update the bug and this thread.

Please let me know if I can help you with anything further within the context of this thread. If your question has been Answered, please mark the thread as such so that it will be helpful to other users. Also, please feel free to Rate this thread to reflect your experience.

Thank you,

Blayne Dreier

Cisco TAC Escalation Team

**Please check out our Podcasts**

TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast

TAC IPS Media Series: https://supportforums.cisco.com/docs/DOC-12758

[Mark^]

Hi Blayne,

Thank you for the additional information.  My question has been thoroughly answered, but I am not sure how to mark it as such.

[Christopher Dreier]

Hello Mark,

I ran individual snmpgets in a loop, getting all of the OIDs up to the IF-MIB::ifSpecific.# OIDs and I did encounter the issue. I'm going to try and narrow the OID scope to reveal what OIDs are the trigger.

On each message post in the thread, you should see a "Correct Answer" box. After clicking on the text in that box, that particular message is marked as an answer to the thread. Below is an example.

Thank you,

Blayne Dreier

Cisco TAC Escalation Team

**Please check out our Podcasts**

TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast

TAC IPS Media Series: https://supportforums.cisco.com/docs/DOC-12758

[Mark^]

I did not mark this thread as a "question" so that must be why I am not seeing the "correct answer" button.

Maybe your research on this will lead to a fix.

Thank you for being so persistent!

[Mark^]

I am just getting back into this and would like to query the COPU usage via SNMP.  Where can I see the list of OID's so I know which one to use?

Thanks.