ASA5545-X dropping packets on inside and outside interfaces

deyster94 — Tue, 26 Mar 2019 00:54:01 GMT

I worked remotely with one of our clients to get a new ASA5545-X installed for them yesterday because they were having internet connectivity issues and their old 5520 was getting pounded. However, this didn't fix their internet connectivity issues and we are fairly certain it's not this new ASA. The kicker with this issue is, is that it happens when they approach 12,000 connections through the ASA. This was one reason we thought it was the old ASA as the 5520 gets crushed with that many connections. They have an odd setup for how their data gets to the internet. Here is how the traffic flows:

Core Switch-->iPrism (content filter)-->ASA-->FatPipe-->Comcast Cable Modem-->Internet

We took the iPrism out of the mix and still had issues. The client is going to call Comcast and FatPipe as well. If I look at the inside and outside interfaces, both are constantly showing dropped packets:

Interface GigabitEthernet0/0 "OUTSIDE", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
MAC address f44e.059e.f430, MTU 1500
IP address 173.15.X.X, subnet mask 255.255.255.240
78542888 packets input, 77587589850 bytes, 0 no buffer
Received 33737 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
61894274 packets output, 26309754996 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 1 interface resets
0 late collisions, 0 deferred
4 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (465/364)
output queue (blocks free curr/low): hardware (446/120)
Traffic Statistics for "OUTSIDE":
78542884 packets input, 76113344584 bytes
61894274 packets output, 25031942679 bytes
775210 packets dropped
1 minute input rate 3774 pkts/sec, 4081168 bytes/sec
1 minute output rate 2473 pkts/sec, 445102 bytes/sec
1 minute drop rate, 84 pkts/sec
5 minute input rate 3801 pkts/sec, 4044740 bytes/sec
5 minute output rate 2728 pkts/sec, 520275 bytes/sec
5 minute drop rate, 46 pkts/sec

Interface GigabitEthernet0/1 "INSIDE", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
MAC address f44e.059e.f42c, MTU 1500
IP address 172.16.X.X, subnet mask 255.255.255.0
58769406 packets input, 23949262559 bytes, 0 no buffer
Received 958 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
81620575 packets output, 79553299593 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 6 interface resets
0 late collisions, 0 deferred
277 input reset drops, 375 output reset drops
input queue (blocks free curr/low): hardware (509/370)
output queue (blocks free curr/low): hardware (505/307)
Traffic Statistics for "INSIDE":
58768288 packets input, 22710337662 bytes
81621115 packets output, 78021353831 bytes
292132 packets dropped
1 minute input rate 2381 pkts/sec, 440472 bytes/sec
1 minute output rate 3737 pkts/sec, 3985284 bytes/sec
1 minute drop rate, 25 pkts/sec
5 minute input rate 2377 pkts/sec, 395251 bytes/sec
5 minute output rate 3671 pkts/sec, 4001053 bytes/sec
5 minute drop rate, 14 pkts/sec

If I do a 'show asp drop,' this is what I see:

Frame drop:
NAT-T keepalive message (natt-keepalive) 212
IPSEC tunnel is down (ipsec-tun-down) 20
Flow is denied by configured rule (acl-drop) 421814
First TCP packet not SYN (tcp-not-syn) 113789
Bad TCP checksum (bad-tcp-cksum) 1
Bad TCP flags (bad-tcp-flags) 36
TCP data send after FIN (tcp-data-past-fin) 1
TCP failed 3 way handshake (tcp-3whs-failed) 10823
TCP RST/FIN out of order (tcp-rstfin-ooo) 29204
TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 83
TCP SYNACK on established conn (tcp-synack-ooo) 31
TCP packet SEQ past window (tcp-seq-past-win) 6247
TCP invalid ACK (tcp-invalid-ack) 8708
TCP Out-of-Order packet buffer full (tcp-buffer-full) 71898
TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout) 9022
TCP RST/SYN in window (tcp-rst-syn-in-win) 80
TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 12998
TCP packet failed PAWS test (tcp-paws-fail) 822
Slowpath security checks failed (sp-security-failed) 147
Expired flow (flow-expired) 1194
DNS Inspect id not matched (inspect-dns-id-not-matched) 1
Dropped pending packets in a closed socket (np-socket-closed) 39

Last clearing: 09:33:43 EDT Oct 23 2014 by admin

Flow drop:
Inspection failure (inspect-fail) 562

Last clearing: 09:33:43 EDT Oct 23 2014 by admin

I am doing some other looking into what the issue might be, but I figured I would put this out here to have the community look at to see if others have had this issue. We would like them to get rid of the FatPipe and iPrism, but due to other circumstances, they cannot at the moment.

TIA for any help.

Dan

Hi,I don't see anything in

Vibhor Amrodia — Fri, 24 Oct 2014 01:29:47 GMT

Hi,

I don't see anything in the Show Interface output that would point to Over subscription on the ASA device due to traffic.

From the ASP drops , i would say to look into the tcp-not-syn drops as they seem quite high.

This might point to Asymmetric Routing etc in the network which might point to the latency issues as well.

Also , please share the show traffic outputs at the time when the traffic through the ASA device is maximum.

Thanks and Regards,

Vibhor Amrodia

Vibhor,Thanks for the reply.

deyster94 — Fri, 24 Oct 2014 11:00:06 GMT

Vibhor,

Thanks for the reply. I opened a TAC case as well and they said the same thing about the tcp-not-syn drops. This is puzzling to me since they only have one internet connection and all the routing is static. The only thing I could figure is maybe it's something with the fatpipe device since it is routing traffic between the ASA and Comcast cable modem. The client is going to call fatpipe today and I am waiting to see when the issue happens again to gather information for the TAC engineer.

Dan

Hi,Would you be able to

Vibhor Amrodia — Fri, 24 Oct 2014 11:12:28 GMT

Hi,

Would you be able to provide me with the TAC SR ?