Cisco ASA Inside Interface Packet Loss

[Charger1129]

Hey everyone. I have an ASA firewall connected at a site and I'm noticing a lot of packet loss on the inside interface. The duplex and speed are set to auto, so they've negotiated to 100/Full. But even with that I still see heavy packet loss. I'm not too sure what to look at that would cause such a high loss. Any thoughts?

[Philip D'Ath]

I have had an issue like this happen before where the interface buffers simply ran out - packets came in faster than the ASA could process and forward them.

I ended up using a Gigabit connection so that I could enable Gigabit pause frames (needs the device that the ASA plugs into to also support pause frames).

Is it possible that the some of the traffic on other interfaces is greater than 100Mb/s?  Perhaps more load is being generated than a single 100Mb/s link can handle.

Does the 100Mb/s interface perhaps have lots of VLANs on it?  If so, could you use an additional interface and move some of the VLANs off to it?  Extra interfaces means extra interface buffers.

[Charger1129]

Below is a copy of my show interface on my ASA. To add some detail:

1. The ASA is GB but it is going to a 10/100 fast ethernet port on the switch.
2. The inside interface does have multiple VLANs on it currently. Only 3 are in use at the moment, the inside, partner1, and guest-wifi. The inside and guest-wifi are the more heavily used interfaces if anything. The others haven't been put in to production just yet. 

Could I possibly need to go GB to GB from ASA to switch?

show int
Interface GigabitEthernet0/0 "outside", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)
Input flow control is unsupported, output flow control is off
Description: Internet-Fiber
MAC address 84b8.F01E.4fc2, MTU 1500
IP address COMPANY--ASA, subnet mask 255.255.255.248
3926664 packets input, 2535791275 bytes, 0 no buffer
Received 2363 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
3846526 packets output, 2158300376 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 1 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (481/451)
output queue (blocks free curr/low): hardware (451/420)
Traffic Statistics for "outside":
3937625 packets input, 2465246863 bytes
3846526 packets output, 2088302023 bytes
1236 packets dropped
1 minute input rate 34 pkts/sec, 20527 bytes/sec
1 minute output rate 35 pkts/sec, 18336 bytes/sec
<--- More --->

1 minute drop rate, 0 pkts/sec
5 minute input rate 32 pkts/sec, 19047 bytes/sec
5 minute output rate 32 pkts/sec, 17178 bytes/sec
5 minute drop rate, 0 pkts/sec

Interface GigabitEthernet0/5 "", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(100 Mbps)
Input flow control is unsupported, output flow control is off
Available but not configured via nameif
MAC address 84b8.F01E.4fc1, MTU not set
IP address unassigned
3830871 packets input, 2052901389 bytes, 0 no buffer
Received 53903 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
3687298 packets output, 2301695984 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 3 interface resets
0 late collisions, 0 deferred
<--- More --->

0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (508/456)
output queue (blocks free curr/low): hardware (493/438)
Interface GigabitEthernet0/5.10 "voice", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
VLAN identifier 10
Description: **-Voice-VLAN**
MAC address 84b8.F01E.4fc1, MTU 1500
IP address COMPANY--Voice-Gateway, subnet mask 255.255.255.0
Traffic Statistics for "voice":
2717070 packets input, 1511158732 bytes
2509155 packets output, 1484089001 bytes
204284 packets dropped
Interface GigabitEthernet0/5.20 "inside", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
VLAN identifier 20
Description: **COMPANY-End-User-Data-VLAN**
MAC address 84b8.F01E.4fc1, MTU 1500
IP address 10.100.20.1, subnet mask 255.255.255.0
Traffic Statistics for "inside":
1102620 packets input, 454243555 bytes
1164140 packets output, 719318151 bytes
27944 packets dropped
Interface GigabitEthernet0/5.30 "PARTNER1", is up, line protocol is up
<--- More --->

Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
VLAN identifier 30
Description: **PARTNER1-Data-VLAN**
MAC address 84b8.F01E.4fc1, MTU 1500
IP address 10.100.30.1, subnet mask 255.255.255.0
Traffic Statistics for "PARTNER1":
4608 packets input, 949674 bytes
1 packets output, 28 bytes
4608 packets dropped
Interface GigabitEthernet0/5.40 "PARTNER2", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
VLAN identifier 40
Description: **PARTNER2-Data-VLAN**
MAC address 84b8.F01E.4fc1, MTU 1500
IP address 10.100.40.1, subnet mask 255.255.255.0
Traffic Statistics for "PARTNER2":
0 packets input, 0 bytes
1 packets output, 28 bytes
0 packets dropped
Interface GigabitEthernet0/5.50 "general", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
VLAN identifier 50
Description: **General-Data-VLAN**
MAC address 84b8.F01E.4fc1, MTU 1500
<--- More --->

IP address 10.100.50.1, subnet mask 255.255.255.0
Traffic Statistics for "general":
0 packets input, 0 bytes
1 packets output, 28 bytes
0 packets dropped
Interface GigabitEthernet0/5.60 "guest-wifi", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
VLAN identifier 60
Description: **Guest-Wireless-Data-VLAN**
MAC address 84b8.F01E.4fc1, MTU 1500
IP address 10.100.60.1, subnet mask 255.255.255.0
Traffic Statistics for "guest-wifi":
6563 packets input, 675344 bytes
14023 packets output, 15925180 bytes
60 packets dropped
Interface Management0/0 "management", is down, line protocol is down
Hardware is en_vtun rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex, Auto-Speed
Input flow control is unsupported, output flow control is off
MAC address 84b8.F01E.4fbe, MTU 1500
IP address 192.168.1.1, subnet mask 255.255.255.0
29742 packets input, 1249164 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
<--- More --->

0 pause input, 0 resume input
0 L2 decode drops
1 packets output, 42 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 1 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (0/0)
output queue (blocks free curr/low): hardware (0/0)
Traffic Statistics for "management":
0 packets input, 0 bytes
1 packets output, 28 bytes
0 packets dropped
1 minute input rate 0 pkts/sec, 0 bytes/sec
1 minute output rate 0 pkts/sec, 0 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 0 pkts/sec, 0 bytes/sec
5 minute output rate 0 pkts/sec, 0 bytes/sec
5 minute drop rate, 0 pkts/sec
Management-only interface. Blocked 0 through-the-device packets

[Philip D'Ath]

You aren't running out of NIC buffers (like the problem I had).

I see your voice vlan has dropped quite a few packets (27944 packets dropped).

I'm not convinced you have an actual problem.  Is there any problem observable by the users?

If you watch the log in ASDM does it mention anything about dropping packets?

[Charger1129]

Well this particular site happens to be having internet speed issues, so this was something I was looking at as a possible cause. My network monitoring is not alerting high packet loss though so this may be normal.

I do however have another ASA that is alerting high packet loss and shows a similar result in the output. 

[Philip D'Ath]

Try using the "show asp drop" command to get more detailed reasons as to why the interface is showing so many drops.  It should give a big hint.

[Charger1129]

Here's the results.

Frame drop:
IPSEC tunnel is down (ipsec-tun-down) 74
SVC Module does not have a session (mp-svc-no-session) 3
Invalid encapsulation (invalid-encap) 377
No valid adjacency (no-adjacency) 18
No route to host (no-route) 613
Flow is denied by configured rule (acl-drop) 256104
First TCP packet not SYN (tcp-not-syn) 4152
Bad TCP flags (bad-tcp-flags) 31
TCP failed 3 way handshake (tcp-3whs-failed) 164
TCP RST/FIN out of order (tcp-rstfin-ooo) 3127
TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 15
TCP SYNACK on established conn (tcp-synack-ooo) 7
TCP packet SEQ past window (tcp-seq-past-win) 58
TCP RST/SYN in window (tcp-rst-syn-in-win) 20
Early security checks failed (security-failed) 1
Slowpath security checks failed (sp-security-failed) 84119
IP option drop (invalid-ip-option) 1
ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched) 1
ICMP Error Inspect no existing conn (inspect-icmp-error-no-existing-conn) 1
DNS Inspect invalid packet (inspect-dns-invalid-pak) 5
DNS Inspect id not matched (inspect-dns-id-not-matched) 55
FP L2 rule drop (l2_acl) 345
Interface is down (interface-down) 54830
Dropped pending packets in a closed socket (np-socket-closed) 97
IKE new SA limit exceeded (ike-sa-rate-limit) 206052

Last clearing: Never

Flow drop:
Need to start IKE negotiation (need-ike) 427792
Inspection failure (inspect-fail) 394
SSL handshake failed (ssl-handshake-failed) 30
DTLS hello processed and closed (dtls-hello-close) 3

Last clearing: Never

[Philip D'Ath]

I don't think you have any problems.

A huge number of the drops are because of "deny" rules:

Flow is denied by configured rule (acl-drop) 256104

[Charger1129]

What's your taking on this one? This one is the ASA alerting of dropped packets on the inside interface.

Frame drop:
IPSEC tunnel is down (ipsec-tun-down) 198
VPN reclassify failed (vpn-reclassify-failed) 14
Invalid IP header (invalid-ip-header) 10
Invalid IP length (invalid-ip-length) 2
Invalid UDP Length (invalid-udp-length) 3
No valid adjacency (no-adjacency) 236
No route to host (no-route) 206
Reverse-path verify failed (rpf-violated) 427
Flow is denied by configured rule (acl-drop) 756704
Invalid SPI (np-sp-invalid-spi) 14
First TCP packet not SYN (tcp-not-syn) 16301
Bad TCP flags (bad-tcp-flags) 13
TCP data send after FIN (tcp-data-past-fin) 34
TCP failed 3 way handshake (tcp-3whs-failed) 910
TCP RST/FIN out of order (tcp-rstfin-ooo) 55107
TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff) 25
TCP SYNACK on established conn (tcp-synack-ooo) 86
TCP packet SEQ past window (tcp-seq-past-win) 107
TCP RST/SYN in window (tcp-rst-syn-in-win) 288
Slowpath security checks failed (sp-security-failed) 468064
ICMP Inspect bad icmp code (inspect-icmp-bad-code) 1
ICMP Inspect seq num not matched (inspect-icmp-seq-num-not-matched) 66
DNS Inspect id not matched (inspect-dns-id-not-matched) 76
FP L2 rule drop (l2_acl) 9953949
Interface is down (interface-down) 24
IKE new SA limit exceeded (ike-sa-rate-limit) 4505

Last clearing: Never

Flow drop:
Tunnel has been torn down (tunnel-torn-down) 28
Need to start IKE negotiation (need-ike) 32098
VPN handle not found (vpn-handle-not-found) 2
Expired VPN context (vpn-context-expired) 4
Inspection failure (inspect-fail) 76988

Last clearing: Never

[Philip D'Ath]

You are getting a lot of:

Slowpath security checks failed (sp-security-failed) 468064

Check out this article describing causes:

https://supportforums.cisco.com/discussion/11168351/sp-security-failed

Are you are getting lots and lots of these:

FP L2 rule drop (l2_acl) 9953949

It tends to suggest that a lot of packets are being dropped by a configured rule.

[Shivapramod M]

Hi,

You mentioned that the packets are getting dropped on ASA interface. could you tell what kind of traffic? is it for a specific IP or random packet drop? 

Is the traffic drop is for VPN traffic or traffic to internet? Are you seeing drop for TCP or UDP?

If it is a specific traffic getting dropped by the ASA then you check the packet tracer for the source and destination IP address.

When you take the "show asp drop" please take it multiple times to check the increment in the value of the counters.

Thanks,
Shivapramod M
Please remember to select a correct answer and rate helpful posts

[Philip D'Ath]

Can you paste the "show interface" output?  Exactly which type of packet loss is happening?  Is it intermittent or happening all the time?