ASA5550 drops packets (sh asp drop) under load

[Amir Asfandyarov]

Hi!

We have a HP network load replicator/generator which simulated live network traffic. We are running the ASA’s through a test where about 85Mb of “live” traffic is being pushed through it (plain HTTP). The initial tests resulted in a large number of packet drops once the data rate hit about 35megs.

By issuing the “ show asp drop” command we were able to determine that the dropped packets were due to tcp session being closed prematurely.

sh asp drop
Frame drop:
  First TCP packet not SYN (tcp-not-syn)  25894
  TCP failed 3 way handshake (tcp-3whs-failed) 3703

The “sysopt connection time” helped slightly but the device is still dropping a lot of packets. According to the customer the true purpose of the HP network replicator was to simulate their customer's, “online check out sessions” their old PIX525 does about 600 sessions per minute, and  the ASA5550 hovers at 380 sessions per minute.

So, again, PIX525 is working OK on this environment.

And

-we've tried to switch from 7.2 to 8.0 and 8.2.1

-we turned off http inspects and threat-detection

-no duplex, speed mismatches, interface errors,

-memory and CPU loads are OK

- no info on the forum or in the bug toolkit

Any thoughts (please see sanitized config attached, the direction of traffic - from outside to inside via static NAT)?

Regards, Amir

[Kureli Sankar]

What code is the PIX-525 running?

PIX is processing production traffic but, they are stress testing the ASA to replace the PIX?

or

Are they testing the old PIX as well?

You have found the reasons for ASP drop. What seems to be the question? Why it is dropping for the two reasons or what can be done to reduce the asp drops? or Why the ASA seems to drop after 300+ connections?

If you issue "clear asp drop" then start the traffic after about how many connections do you start seeing these errors?

You have pretty much verified what we would ask you to verify.

ASA5550 data sheet is here: http://www.cisco.com/en/US/prod/collateral/vpndevc/ps6032/ps6094/ps6120/product_data_sheet0900aecd80601de5.html

-KS

[Amir Asfandyarov]

kusankar, thank you for your quick reply.

PIX is processing production traffic but, they are stress testing the ASA to replace the PIX? - Yes, they are stress teting a new ASA to replace the PIX.

You have found the reasons for ASP drop. What seems to be the question? Why it is dropping for the two reasons or what can be done to reduce the asp drops? or Why the ASA seems to drop after 300+ connections? - The question is - "Why it is dropping for the two reasons or what can be done to reduce the asp drops" and also "Why an older PIX doesn't behave the same way here?"

If you issue "clear asp drop" then start the traffic after about how many connections do you start seeing these errors? - I'll check again when I will be on-site but it is about 2500 of conns when we are starting to see these drops (more connections/sec = more drops, of course).

PIX runs 7.2x (they also tried the 8.0 when they replaced an ASA on their test-bench just to check).

Regards, Amir

[Kureli Sankar]

What code is the old PIX running?

If it is running 6.x then asp drop wasn't there. It may still have dropped these packets.

May be when they run the stress test for example if the SYN ACK packet arrives first, before the firewall saw the SYN packet -  it will obviously drop it right? The first packet as to be the SYN. That is the first reason.

The second one is 3-whs not completed. SYN, SYN-ACK and ACK - three way hand shake has to be completed for any TCP flow and according to these asp drops it wasn't completed. The firewall is behaving as desinged. When you collect packet captures on the interface when they run the stress test, you can see exactly what is happening.

Pls. let me know if you find out more on this issue when you go onsite.

-KS

[Amir Asfandyarov]

May be it will help smbody....

First of all - kusankar, thank you for your help.

The customer has found a workaround to the “session” issue he was having.

Hard coding the gig interface on the ASA to 100base-full on both sides of the link resolved the problem. Prior to this change there was no

Speed or duplex mismatch, the interface’s had been running at 1000base-full. In the past we have run into situations where it’s necessary to hardcode an interface, to get the link to come up, but not an instance where a interface works fine at one speed and not the other.

We've opened up a TAC case with Cisco

Regards, Amir..