05-09-2011 12:29 PM - edited 03-10-2019 05:21 AM
Setting up some snmp monitoring and as I am poking around to figure out what exactly I'd like to get, I run the following commands from my Debian based monitoring host:
While running either of these commands it begins to spit out OID information and then each time it ends with these two lines:
IF-MIB::ifSpecific.5 = OID: SNMPv2-SMI::zeroDotZero
Timeout: No Response from AIP_IPADDRESS
And at that moment, the magic happens; the module no longer responds, pings time-out, and the ASA5505 needs to be physically power cycled for everything to come back up normally.
ASA syslog reports:
May 9 10:34:15 ASA_IP May 09 2011 10:34:15 HOSTNAME : %ASA-1-505005: Module in slot 1 is initializing control communication. Please wait...
May 9 10:34:25 ASA_IP May 09 2011 10:34:25 HOSTNAME : %ASA-1-323001: Module in slot 1 experienced a control channel communication failure.
May 9 10:34:28 ASA_IP May 09 2011 10:34:28 HOSTNAME : %ASA-1-505005: Module in slot 1 is initializing control communication. Please wait...
May 9 10:34:38 ASA_IP May 09 2011 10:34:38 HOSTNAME : %ASA-1-323001: Module in slot 1 experienced a control channel communication failure.
(as you can see, these pairs are 10 seconds apart until the ASA is rebooted)
Is this expected behavior or otherwise already documented?
Let me add that this is not happening for the ASA or any other snmp devices.
As one could imagine -- this is driving me nuts.
05-09-2011 02:19 PM
Hello Mark,
I just tested this locally and I believe that I was able to replicate what you are experiencing. This may be due to a known defect. Can you please run a "show tech" on your SSC-5 (after power-cycling the ASA) and post the information from the core.txt output in the show tech? This will confirm whether I am experiencing what you are experiencing and help to correlate this issue to known defects.
Alternatively, you can open a TAC case and we can carry out the investigation there.
Thank you,
Blayne Dreier
Cisco TAC Escalation Team
**Please check out our Podcasts**
TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast
TAC IPS Media Series: https://supportforums.cisco.com/docs/DOC-12758
05-10-2011 05:34 AM
Is this what you are looking for? I hadn't power cycled the ASA since yesterday, so if that is going to make a difference I can re-post this or open a TAC case.
Thanks.
-----
exec: ls -l /usr/cids/idsRoot/core/mainApp/core.txt
-rw-rw-rw- 1 cids cids 2408 May 9 11:16 /usr/cids/idsRoot/core/mainApp/core.txt
exec: cat /usr/cids/idsRoot/core/mainApp/core.txt
Application thread 462 received trap: 11
--------------------------------------------------------------
eax 0xffffffff -1
ebx 0x408be86c 1082910828
edx 0x00000000 0
ecx 0x00000001 1
edi 0x41600010 1096810512
esi 0x00000080 128
eip 0x4080f646 1082193478
ebp 0x43db1cb4 1138433204
esp 0x43db1c8c 1138433164
cs 0x00000023 35
es 0x0000002b 43
ds 0xc010002b -1072693205
gs 0x000000d7 215
fs 0x00000000 0
ss 0x0000002b 43
efl 0x00010202 66050
uesp 0x43db1c8c 1138433164
trapno 0x0000000e 14
err 0x00000004 4
--------------------------------------------------------------
0x0x4080f646 +/lib/libc.so.6(calloc+0xd6) [0x4080f646];
0x0x407c38f0 +/lib/libc.so.6 [0x407c38f0];
0x0x4044e57d +/lib/libnetsnmpmibs.so.9(Interface_Scan_Init+0x4ad) [0x4044e57d];
0x0x4044f161 +/lib/libnetsnmpmibs.so.9(Interface_Index_By_Name+0x1d) [0x4044f161];
0x0x4044d90f +/lib/libnetsnmpmibs.so.9 [0x4044d90f];
0x0x4044d1f5 +/lib/libnetsnmpmibs.so.9(var_atEntry+0x83) [0x4044d1f5];
0x0x4041de42 +/lib/libnetsnmphelpers.so.9(netsnmp_old_api_helper+0x1e2) [0x4041de42];
0x0x40439bfa +/lib/libnetsnmpagent.so.9(netsnmp_call_handler+0xb5) [0x40439bfa];
0x0x40439dad +/lib/libnetsnmpagent.so.9(netsnmp_call_handlers+0x160) [0x40439dad];
0x0x40431922 +/lib/libnetsnmpagent.so.9(handle_var_requests+0xc7) [0x40431922];
0x0x40431fb5 +/lib/libnetsnmpagent.so.9(handle_getnext_loop+0x64) [0x40431fb5];
0x0x404324dd +/lib/libnetsnmpagent.so.9(handle_pdu+0x268) [0x404324dd];
0x0x40432231 +/lib/libnetsnmpagent.so.9(netsnmp_handle_request+0x8b) [0x40432231];
0x0x40430a23 +/lib/libnetsnmpagent.so.9(handle_snmp_packet+0x1d2) [0x40430a23];
0x0x4049b41d +/lib/libnetsnmp.so.9 [0x4049b41d];
0x0x4049bdc1 +/lib/libnetsnmp.so.9(_sess_read+0x882) [0x4049bdc1];
0x0x4049be10 +/lib/libnetsnmp.so.9(snmp_sess_read+0x24) [0x4049be10];
0x0x4049b52f +/lib/libnetsnmp.so.9(snmp_read+0x35) [0x4049b52f];
0x0x4042ea84 +/lib/libnetsnmpagent.so.9(agent_check_and_process+0x12a) [0x4042ea84];
0x0x831ea7c +/usr/cids/idsRoot/bin/mainApp(_ZN3Cid12Notification9SnmpAgent9agentTaskEPNS_2Mt12ThreadedTaskEPv+0x40) [0x831ea7c];
0x0x4008e7db +/usr/cids/idsRoot/lib/libcidcore.002.041.so(_ZN3Cid2Mt12ThreadedTask11threadStartEPv+0x901) [0x4008e7db];
0x0x40024004 +/lib/libpthread.so.0 [0x40024004];
0x0x4087287a +/lib/libc.so.6(clone+0x3a) [0x4087287a];
exec: ps -ew f
PID TTY STAT TIME COMMAND
1 ? S 0:28 init
2 ? S 0:00 [keventd]
3 ? SN 0:00 [ksoftirqd_CPU0]
4 ? S 0:00 [kswapd]
5 ? S 0:00 [bdflush]
6 ? S 0:00 [kupdated]
50 ? S 0:00 [kjournald]
75 ? S 0:00 [kjournald]
107 ? Ss 0:00 /sbin/syslogd -m 0
110 ? Ss 0:00 /sbin/klogd
122 ? Ss 0:00 /usr/sbin/inetd
126 ? Ss 0:00 /sbin/sshd
14738 ? Ss 0:02 \_ sshd: cisco@pts/0
14757 pts/0 Ss+ 0:01 \_ -cidcli
14759 pts/0 S+ 0:00 \_ -cidcli
14760 pts/0 SN+ 0:05 \_ -cidcli
14768 pts/0 SN+ 0:00 \_ -cidcli
15706 pts/0 SN+ 0:00 \_ -cidcli
317 ? S< 0:00 /usr/cids/idsRoot/bin/SSM_control_proc
341 ? Ss 0:02 /usr/cids/idsRoot/bin/mainApp -d -c 0
344 ? S 0:02 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
345 ? SN 0:18 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
461 ? SN 0:01 | \_ /usr/cids/idsRoot/bin/sensorApp -z 345
485 ? SN 0:00 | \_ /usr/cids/idsRoot/bin/sensorApp -z 345
486 ? SN 4:05 | \_ /usr/cids/idsRoot/bin/sensorApp -z 345
504 ? SN 0:00 | \_ /usr/cids/idsRoot/bin/sensorApp -z 345
926 ? S< 45:57 | \_ /usr/cids/idsRoot/bin/sensorApp -z 345
346 ? S 1:09 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
412 ? SN 15:07 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
15707 ? SN 0:00 | \_ /bin/bash /usr/cids/idsRoot/bin/cidDump -text -wxml -nostatus -stdout
15768 ? RN 0:00 | \_ ps -ew f
413 ? S 0:00 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
419 ? S 0:10 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
433 ? SN 0:02 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
434 ? SN 0:00 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
435 ? SN 0:00 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
436 ? SN 0:00 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
437 ? SN 0:00 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
438 ? SN 0:06 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
439 ? SN 0:06 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
440 ? SN 0:06 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
441 ? SN 0:06 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
442 ? SN 0:06 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
443 ? SN 0:07 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
444 ? SN 0:06 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
445 ? SN 0:06 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
446 ? SN 0:08 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
447 ? SN 0:07 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
448 ? SN 0:01 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
451 ? SN 3:35 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
452 ? SN 0:00 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
462 ? SN 0:01 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
463 ? RN 0:11 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
15753 ? SN 0:00 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
15754 ? SN 0:00 \_ /usr/cids/idsRoot/bin/mainApp -d -c 0
383 tty1 Ss+ 0:00 /sbin/getty 38400 tty1
384 tty2 Ss+ 0:00 /sbin/getty 38400 tty2
385 ttyS0 Ss+ 0:00 /sbin/getty -L ttyS0 9600 vt100
425 ? SNLs 1:34 ntpd
-----
05-10-2011 09:31 AM
Hello Mark,
Yes, that's what I needed. You are experiencing CSCti03741: mainApp crash on Interface_Scan_Init while doing an SNMP walk
You can track the progress of this bug via the CCO Bug Toolkit: http://tools.cisco.com/Support/BugToolKit/action.do?hdnAction=searchBugs If this bug fix is important to you, I do suggest contacting your Cisco Account Team so that they can convey your sentiment and drive this bug to resolution.
Please let me know if I can help you with anything further within the context of this thread. If your question has been Answered, please mark the thread as such so that it will be helpful to other users. Also, please feel free to Rate this thread to reflect your experience.
Thank you,
Blayne Dreier
Cisco TAC Escalation Team
**Please check out our Podcasts**
TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast
TAC IPS Media Series: https://supportforums.cisco.com/docs/DOC-12758
05-11-2011 05:51 AM
Blayne,
Thank you! As I said, it was driving me nuts. Now I can be satisfied that it is not just me.
I did read the bug, but am trying to understand it. Basically, is it only the SNMP walk that causes it to crash? Meaning; I can still get specific SNMP info as long as I know the OID without crashing the mainApp?
05-16-2011 05:19 PM
Hello Mark,
I've been testing this quite a while and I have not experienced the failure while doing an snmpget (instead of an snmpwalk) for a particular OID. I have a script running now that polls the interface OIDs in a loop, using both v1 and v2c.
The bug was written with the observation that an snmpwalk triggers the issue. However, it does not explicitly exclude snmpgets from causing the issue. I'll keep my script running and if it does crash, I'll update the bug and this thread.
Please let me know if I can help you with anything further within the context of this thread. If your question has been Answered, please mark the thread as such so that it will be helpful to other users. Also, please feel free to Rate this thread to reflect your experience.
Thank you,
Blayne Dreier
Cisco TAC Escalation Team
**Please check out our Podcasts**
TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast
TAC IPS Media Series: https://supportforums.cisco.com/docs/DOC-12758
05-17-2011 05:13 AM
Hi Blayne,
Thank you for the additional information. My question has been thoroughly answered, but I am not sure how to mark it as such.
05-17-2011 09:18 AM
Hello Mark,
I ran individual snmpgets in a loop, getting all of the OIDs up to the IF-MIB::ifSpecific.# OIDs and I did encounter the issue. I'm going to try and narrow the OID scope to reveal what OIDs are the trigger.
On each message post in the thread, you should see a "Correct Answer" box. After clicking on the text in that box, that particular message is marked as an answer to the thread. Below is an example.
Thank you,
Blayne Dreier
Cisco TAC Escalation Team
**Please check out our Podcasts**
TAC Security Show: http://www.cisco.com/go/tacsecuritypodcast
TAC IPS Media Series: https://supportforums.cisco.com/docs/DOC-12758
05-18-2011 06:43 AM
I did not mark this thread as a "question" so that must be why I am not seeing the "correct answer" button.
Maybe your research on this will lead to a fix.
Thank you for being so persistent!
08-10-2011 07:05 AM
I am just getting back into this and would like to query the COPU usage via SNMP. Where can I see the list of OID's so I know which one to use?
Thanks.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide