ASA 5508 - Dropping Packets - Input, CRC, Frame and Output Reset Drops

[rboklewski]

We have a tunnel from our main office to our DR site.  We have Veeam replication jobs to this DR site and during a job I noticed we are losing packets.  Our data center provider saw that we were hitting the 40mbps Policer set on their equipment.  We increased the bandwidth to a 100mbps and throttled our replication traffic to 90mbps so we wouldn't hit the Policer limit (The Policer limit drops packets and doesn't queue).  I started digging into our outside interface on our 5508 ASA to check for errors.  I am seeing Input errors, CRC, frame errors as well as some output reset drops.  The data center provider is seeing a bunch of Output Drops on the interface pointed to us and they saw a big reduction in drops on their policer after we throttled the traffic, they still did see some drops though (May have to drop to 80mbps).  I checked and they are hard-coded to 100mb full and so are we.  I did notice this firewall we have does have 'Shun' enabled for threat detection.  After running show shun/show threat-detection shun there isn't anything being blocked.  So I think I ruled that out as an issue.  I was seeing some rate-1 (scanning) drops in the log, which is what led me to look into the threat detection settings.  I am really wondering if this could be a cabling issue though. Between yesterday and today I saw the input drops increase by 6, CRC errors increased by 4, and frame increase by 2.  Anyway, here is the interface detail output.  I did notice that my interface is gigabit and theirs is fast ethernet, maybe an issue there even though they both show 100mbps full? Any help is much appreciated!

DR-Firewall# sh int gigabitethernet1/1 detail
Interface GigabitEthernet1/1 "outside", is up, line protocol is up
Hardware is Accelerator rev01, BW 1000 Mbps, DLY 10 usec
Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)
Input flow control is unsupported, output flow control is off
MAC address 7070.8b67.d797, MTU 1500
IP address 55.34.22.230, subnet mask 255.255.255.248
10652650677 packets input, 15705781274067 bytes, 0 no buffer
Received 11403465 broadcasts, 0 runts, 0 giants
7730835 input errors, 2255304 CRC, 5475531 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
7074336024 packets output, 1012413818419 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 0 interface resets
0 late collisions, 0 deferred
0 input reset drops, 173 output reset drops
input queue (blocks free curr/low): hardware (2019/1847)
output queue (blocks free curr/low): hardware (2047/1928)
Traffic Statistics for "outside":
10643524215 packets input, 15514135203400 bytes
7074336024 packets output, 884987598447 bytes
7540434 packets dropped
1 minute input rate 10 pkts/sec, 1696 bytes/sec
1 minute output rate 11 pkts/sec, 4404 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 21 pkts/sec, 3213 bytes/sec
5 minute output rate 31 pkts/sec, 8672 bytes/sec
5 minute drop rate, 0 pkts/sec
Control Point Interface States:
Interface number is 2
Interface config status is active
Interface state is active

[cmarva]

classic sign of a duplex mismatch, if you check the other side of this connection you will see it at half duplex.

[rboklewski]

That is what I thought.  However, here was their output.  Shows 100 full duplex. 

SW01.xK#show int fa1/0/11

FastEthernet1/0/11 is up, line protocol is up (connected)

  Hardware is Fast Ethernet, address is 001c.0f0b.f70d (bia 001c.0f0b.f70d)

  Description: xxxx

  MTU 1998 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 110/255, rxload 7/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, media type is 10/100BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output 00:00:03, output hang never

  Last clearing of "show interface" counters 1w4d

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 25694960

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 3004000 bits/sec, 2604 packets/sec

  5 minute output rate 43408000 bits/sec, 3665 packets/sec

     550991396 packets input, 78796075985 bytes, 0 no buffer

     Received 0 broadcasts (0 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     0 input packets with dribble condition detected

     804829727 packets output, 1158381187477 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

[cmarva]

ok, fair enough. It could be cabling. I also see there is an MTU mismatch, the switchport is 1998 and the ASA is 1500.

another question, how are your interfaces set, are they locked or are they running auto?

[rboklewski]

Hmmm...what do you mean by locked or auto?  We are both hard coded for 100 full. Good catch on the MTU.  However, I have been pinging to a machine in the data center with a buffer size of 2000 and I didn't experience any packet loss.  So, doubt that is an issue.

[cmarva]

locked = hard coded

now, as for the MTU, the most likely place you would see a problem is with TCP based traffic that traverses the firewall. Or at least that's where I usually see a mismatch manifest. I'm not saying it is or is not the cause of the errors/drops, but it is a difference that you may want to consider making equal. It's probably nothing, but you never know. At L2, probably nothing. But if the switchport is in, say, vlan 10, and the SVI for vlan 10 is also set for 1998, then that could cause your problem.

[rboklewski]

Yeah, I am thinking not the issue.  So it is their switch interface (10/100) to our gigabitethernet interface on our ASA.  I would think to see some packet loss there after increasing the buffer size when I was using ping.  One of the engineers mentioned a potential issue with having a fastethernet port connected to gigabitethernet port even though they are both set to 10/100.  Ever heard of any of any issues there?  I literally can't hold down an SSH connection to my ASA as the connection keeps breaking because of drops I believe.  

I am also seeing the below and we do have 'Shun' enabled under threat protection.  Though again, running the below shows nothing blocked.

DR-Firewall# sh threat-detection shun
Shunned Host List:
DR-Firewall#

Below is strange, not sure what to make of this, the latest attacker is the outside interface of my firewall and the latest target host is the source for my replication traffic at my main site.  I changed ip's for security purposes.  Nothing shows as being blocked above so I am assuming nothing here to look at.

DR-Firewall# show threat-detection scanning-threat
Latest Target Host & Subnet List:
218.21.34.216 (outside)
Latest Attacker Host & Subnet List:
90.24.115.140 (NP Identity Ifc)

I am also wondering if I should have the data center engineer run a cable test on the Cisco switch.  I have used it in the past but not sure how well it will work on their FastEthernet interface.  I read somewhere only two pairs may only get tested when it is FastEthernet. 

https://community.cisco.com/t5/networking-blogs/switch-how-to-test-a-cable-status/ba-p/3107543

I also see these in my log, but again, I don't think anything is being blocked.  I think these are just warning.

%ASA-4-733100: [ Scanning] drop rate-1 exceeded. Current burst rate is 44 per second, max configured rate is 10; Current average rate is 2 per second, max configured rate is 5; Cumulative total count is 1637
%ASA-4-733100: [ Scanning] drop rate-2 exceeded. Current burst rate is 8 per second, max configured rate is 8; Current average rate is 1 per second, max configured rate is 4; Cumulative total count is 5119

[Muhammad Awais Khan]

Hi,

If there was no drop and issue was with replication only then i would have suggest to fine tune the MTU. I beleive it can be either issue with cable or issue due to fragmentation of large bulk of traffic of database replication which is traversing the tunnel.

My experience with Data replication over Tunnels require fine tunning MTU. With default MTU of 1500, when IPSEC Tunnel overhead added, it will cause packet to fragmented which means on the reaceiving side Firewall packet need to be re-assembled which sometimes create issues with Database replication type of traffic session.

I would suggest two things:

1) reduce to MTU to  1300

sysopt connection tcpmss 1300

2) test the cable which i beleive you are already doing

[rboklewski]

Yeah, so it seems the issue isn't with replication only.  I am even getting disconnects in my putty session to the ASA in our DR site.  There is something really wrong.  One thing to note as I kind of mentioned above.  The Data Center vendor must be using an old Cisco switch as it only supports fast ethernet (10/100).  Now, I have for a long time always hard-coded speed and duplex.  However, with this switch only supporting fast ethernet, I am wondering if we may need to set to both sides to auto-negotiate.  I found this article, please read the part posted by Jneiberger in the link below as to why you should set to auto-negotiate for fast ethernet standard. I know both sides do show 100 full, but maybe there still is an issue here.  What do you think?

https://learningnetwork.cisco.com/thread/24824

[rboklewski]

Any other input here?

[Marius Gunnerud]

I have seen a similar issue at one of my sites recently where we were hard coded to full duplex and speed 100 while the neighbor router was auto auto.  This resulted in a duplex mismatch. Once I changed my side to auto auto aswell we were up at full duplex and speed 1000.

I would suggest giving it a go at setting auto duplex and auto speed.

Depending on your environment,  If you are worried about losing remote connectivity (which could very  well happen), I always set the reload in 6 command before making changes that could lock me out.  You'll have some down time during the reload but at least you wont have to go onsite to console in to it.

[rboklewski]

Gotcha, can you disable the reload command if I don't lose connection?  Assuming that means reload in 6 minutes. 

[Marius Gunnerud]

Yes. Reload cancel
And yes, it is 6 minutes