Cisco ASA5505 SNMP Polling Fails

[Mirko Jelic]

I am having issues with monitoring our Cisco ASA5505 devices with "SolarWinds Orion NPM 10.2" through the use of SNMPv2. On some devices we see that SNMP polling stops and that the ASA's interfaces would show up as unknown - usually when the link to the device goes down/up or after a random ammount of time. At that point SNMP polling data is no longer updated and all we can rely on is ICMP for device status. I can resolve the issue by restarting the remote ASA OR restarting the SolarWinds server after which polling resumes. We are only seeing this behaviour with our remote ASA's.

Our setup is as follows:

Head End:

Cisco ASA 5520 [ASA 8.3(2)]

Remote:

Cisco ASA 5505 [ASA 8.3(2)]

I have found a SolarWinds article listed below that possibly identifies the issue that we are having but am not sure where to start.

SolarWinds Article:

http://knowledgebase.solarwinds.com/kb/questions/1193/Why+are+my+interfaces+changing+to+%22unknown%22+status%2C+even+when+I+know+they+are+active%3F

[Jennifer Halim]

Are you polling through VPN tunnel? if you are, maybe your VPN tunnel is down, hence you are not able to poll the remote ASA, and once your VPN tunnel is up and running again, then the polling starts to work.

[Mirko Jelic]

Yes it is polling from within the VPN tunnel (inside interface), and we have confirmed that the VPN tunnel remains up. We have approx 10 ASA5505's deployed and all of them exhibit this behaviour. 

[Jennifer Halim]

Base on that article from Solar Winds, it might not work properly through stateful firewall, and ASA is indeed a stateful firewall.

Can you please share the output of "show asp drop" from both ASAs.

I doubt that there is anything to tweak within the ASA with the snmp request id, as ASA does not perform any deep packet inspection for snmp packet. And since SNMP uses UDP if UDP packet violates the standard RFC, ASA will drop the packet.

[Mirko Jelic]

Below are my results from the remote ASA5505 which we have problems monitoring and the head end ASA5520.

REMOTE END:

===========================================================

#sh asp drop

Frame drop:

  Invalid encapsulation (invalid-encap)                                    46917

  No valid adjacency (no-adjacency)                                           17

  Flow is denied by configured rule (acl-drop)                               155

  Invalid SPI (np-sp-invalid-spi)                                             20

  First TCP packet not SYN (tcp-not-syn)                                       1

  TCP RST/FIN out of order (tcp-rstfin-ooo)                                    1

  IPSEC tunnel is down (ipsec-tun-down)                                        6

  Slowpath security checks failed (sp-security-failed)                     56466

  Interface is down (interface-down)                                         182

  Non-IP packet received in routed mode (non-ip-pkt-in-routed-mode)            1

  Dropped pending packets in a closed socket (np-socket-closed)               25

Last clearing: Never

Flow drop:

Last clearing: Never

===========================================================

HEAD END:

===========================================================

# sh asp drop

Frame drop:

  Invalid TCP Length (invalid-tcp-hdr-length)                                 15

  No valid adjacency (no-adjacency)                                            3

  Flow is denied by configured rule (acl-drop)                             75558

  NAT-T keepalive message (natt-keepalive)                                763752

  First TCP packet not SYN (tcp-not-syn)                                    1145

  TCP failed 3 way handshake (tcp-3whs-failed)                               396

  TCP RST/FIN out of order (tcp-rstfin-ooo)                                16251

  IPSEC tunnel is down (ipsec-tun-down)                                      202

  Slowpath security checks failed (sp-security-failed)                  12297540

  ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched)         72

  DNS Inspect id not matched (inspect-dns-id-not-matched)                      4

  FP L2 rule drop (l2_acl)                                                     3

  Interface is down (interface-down)                                           4

  Dropped pending packets in a closed socket (np-socket-closed)               24

Last clearing: Never

Flow drop:

  Inspection failure (inspect-fail)                                          850

  SSL handshake failed (ssl-handshake-failed)                                  3

  SSL received close alert (ssl-received-close-alert)                          1

Last clearing: Never

===========================================================

[alex]

Did you ever resolve this?

I have exactly the same issue with SNMP polling randomly stopping.

Solarwinds tech support pointed me a the stateful firewall but before I break out wire shark I was wondering if there was a known answer!

[Mirko Jelic]

Unfortunately I am still dealing with the issue, I have a case open with SolarWinds and Cisco (multiple depts). It seems like the ASA just stops responding to SNMP requests (not dropping them from what the ASPDROP captures show). The odd part is that after a long ammount of time polling resumes.

[Mirko Jelic]

@Alex - can you please list your environment and what devices you are having issues with... if your scenario matches mine maybe I can get more attention on this to get it resolved - wherever the problem may lie.

[alex]

I've also got a ticket open with Solarwinds so hopefully they can get this moving.

I've got a Solarwinds Orion server at our core site (10.3.1) behind a Cisco ASA5510 cluster. (8.2 I think)

The nodes I have the most problem with are at a site behind a Cisco 5505 (Latest 8.4) and are polled over the VPN tunnel.

I periodically have this issue with other hosts that are at seperate sites behind other brands of routers and not access via the VPN, although not nearly as frequently.

Solarwinds were saying that certain stateful firewalls block SNMP request-id's with high numbers and although starting the job engine v2 seems to fix it (as request-id's restart from 0) my wireshark captures seem to indicate that this isn't exactly the case as I have packets with higher request-id's being passed okay.

I can usually reproduce the fault by rebooting the ASA5505 at the client site but this frequently happens overnight as the sites conenction drops.

The devices i'm trying to poll are a Windows 2008 R2 server and a Cisco 3750G switch.

When the SNMP packets fail I get a TTL exceeded back from the ASA5510 at the core site in Wireshark whcih i think must be something to do with it.

I find that polling never resumes in my case, the only fix is a restart of the job service v2 or a reboot of the monitoring server.

[Gavin Barber]

Have you set the options for the NAT rules for your remote subnets?

I've had this behaviour and on my NAT for my remote IPsec VPN subnets I disable Proxy ARP on egress interface. I dedicate the inside interface for management on the ASA 5505s.

This then resolves the ICMP and SNMP polling issue from Solarwinds to the remote ASA 5505s when using the inside interface on the remote ASAs.