Unexplained ASP Drops in ASA

ben.weber · ‎07-03-2019

I have a strange situation I've never seen before.

I have a firewall with a DMZ with one server in it. It's been set up for years and everything has always worked fine. The server has a static NAT translation it uses when it accesses the internet. And that translation works fine for inbound connections (it's a portal server). However, that server has lost its ability to establish outbound connections to the internet.

-When I run a packet tracer command everything comes back fine. No problems.

-When I run captures I see packets hitting the DMZ interface from the server, so not a gateway issue or anything like that. But those packets do not egress the outside interface. Instead they show up in an asp-drop capture. But there's no explanation of why they are getting dropped.

-There is nothing in the log files indicating that these packets are getting dropped or why.

So perfectly fine tcp/80 packets, ping packets, etc. are all triggering asp drops with no explanation. And until a few days ago this all worked fine.

Thoughts? Anyone ever see this?

Ben

Seb Rupik · ‎07-03-2019

Hi there,

The asp-drop capture should detail the reason why the packets were dropped.

Can you share the relevant output from the capture involving the server.

cheers,

Seb.

ben.weber · ‎07-03-2019

Sure. This is what I see in the asp-drop capture:

507: 10:04:29.768591 802.1Q vlan#12 P0 64.30.34.106.5570 > 216.92.220.36.80: S 778865908:778865908(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
558: 10:04:32.768331 802.1Q vlan#12 P0 64.30.34.106.5570 > 216.92.220.36.80: S 778865908:778865908(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
591: 10:04:36.367382 802.1Q vlan#12 P0 64.30.34.106.5574 > 216.92.220.36.80: S 2250592753:2250592753(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
614: 10:04:38.774480 802.1Q vlan#12 P0 64.30.34.106.5570 > 216.92.220.36.80: S 778865908:778865908(0) win 8192 <mss 1380,nop,nop,sackOK>
627: 10:04:39.382868 802.1Q vlan#12 P0 64.30.34.106.5574 > 216.92.220.36.80: S 2250592753:2250592753(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
730: 10:04:45.388575 802.1Q vlan#12 P0 64.30.34.106.5574 > 216.92.220.36.80: S 2250592753:2250592753(0) win 8192 <mss 1380,nop,nop,sackOK>
935: 10:04:57.400827 802.1Q vlan#12 P0 64.30.34.106.5577 > 216.92.220.36.80: S 414515772:414515772(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
1059: 10:05:00.411187 802.1Q vlan#12 P0 64.30.34.106.5577 > 216.92.220.36.80: S 414515772:414515772(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK>
1133: 10:05:06.417092 802.1Q vlan#12 P0 64.30.34.106.5577 > 216.92.220.36.80: S 414515772:414515772(0) win 8192 <mss 1380,nop,nop,sackOK>

Seb Rupik · ‎07-03-2019

Hmmm, is that really from?:

capture asp-drop type asp-drop all
sh capture asp-drop

ben.weber · ‎07-03-2019

Yep. Here's my syntax:

capture aspcap type asp-drop all

Seb Rupik · ‎07-03-2019

OK, and during one of these occasions of the all the oubound traffic being dropped, what does the output of sh asp drop look like? Any of the counters incrementing?

Also what ASA model and software version are you using?

ben.weber · ‎07-03-2019

It looks like they are incrementing, especially on the no valid adjacency line.

It's an active/standby pair of 5515-x's running 9.6(4)-30

ben.weber · ‎07-03-2019

I cleared the ARP cache per this bug report:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz72137/?rfs=iqvred

But that didn't do anything. Still seeing those packets in the asp-drop capture.

ben.weber · ‎07-03-2019

I know it's weird. I was piping to include for the 216.92.220.36 IP because that's the one we were using for testing. But the public IP that server is NATed to is the 64.30.34.106. I do notice that when I look at that same asp-drop capture for that public IP (which nothing else uses) I'm seeing other packets that say "no valid adjacency":

106: 10:04:09.427727 802.1Q vlan#12 P0 64.30.34.106.5555 > 72.21.81.200.443: S 2531809289:2531809289(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK> Drop-reason: (no-adjacency) No valid adjacency
107: 10:04:09.427803 802.1Q vlan#12 P0 64.30.34.106.5557 > 72.21.81.200.443: S 1324824462:1324824462(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK> Drop-reason: (no-adjacency) No valid adjacency
108: 10:04:09.427864 802.1Q vlan#12 P0 64.30.34.106.5558 > 72.21.81.200.443: S 466982405:466982405(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK> Drop-reason: (no-adjacency) No valid adjacency
125: 10:04:12.426415 802.1Q vlan#12 P0 64.30.34.106.5558 > 72.21.81.200.443: S 466982405:466982405(0) win 8192 <mss 1380,nop,wscale 8,nop,nop,sackOK> Drop-reason: (no-adjacency) No valid adjacency

That could be a code bug. They are running on 9.6(4)-30. They were on -23 until yesterday when I upgraded them, thinking it could be a code bug. -30 is the most recent 9.6(4) code and is supposed to be safe harbor. And I saw a bug report that suggested it might affect this. I wonder if I should go to a whole new train?