Significant upload bandwidth decrease with ASA

[fregeus.ca]

I am at a customers site and there is an issue I've haven't been able to find a cause for.

They have an ASA-5510 with version 8.2(5).  They just upgraded their Internet bandwidth to 30 Mb both ways.

If we do a speed test in front of the ASA, we get 28 Mb/s upload and download, with a ping of about 5 to 10 ms.

If we go behind the ASA, the download is about the same, the upload is decreased to about 12 Mb/s and the ping goes to 260 ms

The license is base, there are no additionnal function added to the firewall (no IPS).  I've check the speed and duplex and everything is fine.

There are no drops on the interfaces or rules of the firewall, no drops on the Interface of the ISP router either. All interfaces are configured at 100Mb full duplex.

I saw a couple of discussions on this in the forums, but they don't seem to come up with anything and they look like they end in the middle of the whole story, like once the problem is solved, they don't update their discussion.

Has anyone seen this before and found a fix?

[jocamare]

How many devices are between the ASA and the computer from where you are running the tests?

Would it be possible to have it DIRECTLY connected to the firewall?

[fregeus.ca]

Hello jocamare

We did perform the tests directly connected to the internal interface of the ASA with the same results.

There is one bluecoat proxy, then two switches between the ASA and the final station.  The tests were done at all level (to find out at what point the "congestion", for the lack of a better word, was happening) and we found that the ASA was the culprit.

I've also attached two images that represent the results of the tests we did.  Image01 is in front of the ASA and image02 is behind the firewall.

Thx.

[jocamare]

Can you share the output of the "show interface" command from the ASA?

Please specify which are the involved addresses.

If you notice some CRC errors, try to replace the cables used on both interfaces.

[fregeus.ca]

asa-XXXXXXXXX# sh int

Interface Ethernet0/0 "outside", is up, line protocol is up

  Hardware is i82546GB rev03, BW 100 Mbps, DLY 100 usec

        Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)

        Input flow control is unsupported, output flow control is off

        Description: Internet

        MAC address c84c.75ea.40c2, MTU 1500

        IP address XXX.XXX.XXX.XXX, subnet mask 255.255.255.248

        18192833 packets input, 18921635428 bytes, 0 no buffer

        Received 3675 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        11610268 packets output, 2447944341 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 0 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops, 0 tx hangs

        input queue (blocks free curr/low): hardware (255/230)

        output queue (blocks free curr/low): hardware (255/229)

  Traffic Statistics for "outside":

        18192833 packets input, 18589279794 bytes

        11610268 packets output, 2209284865 bytes

        93465 packets dropped

      1 minute input rate 179 pkts/sec,  195524 bytes/sec

      1 minute output rate 112 pkts/sec,  13274 bytes/sec

      1 minute drop rate, 1 pkts/sec

      5 minute input rate 123 pkts/sec,  129193 bytes/sec

      5 minute output rate 76 pkts/sec,  8017 bytes/sec

      5 minute drop rate, 0 pkts/sec

Interface Ethernet0/1 "inside", is up, line protocol is up

  Hardware is i82546GB rev03, BW 100 Mbps, DLY 100 usec

        Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)

        Input flow control is unsupported, output flow control is off

        MAC address c84c.75ea.40c3, MTU 1500

        IP address XXX.XXX.XXX.XXX, subnet mask 255.255.255.0

        143338748 packets input, 37203243080 bytes, 0 no buffer

        Received 90422 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        155322564 packets output, 116373061143 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 0 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops, 0 tx hangs

        input queue (blocks free curr/low): hardware (255/230)

        output queue (blocks free curr/low): hardware (255/202)

  Traffic Statistics for "inside":

        143338650 packets input, 34437817408 bytes

        155322564 packets output, 113434641511 bytes

        1782255 packets dropped

      1 minute input rate 338 pkts/sec,  118160 bytes/sec

      1 minute output rate 380 pkts/sec,  226499 bytes/sec

      1 minute drop rate, 2 pkts/sec

      5 minute input rate 369 pkts/sec,  222775 bytes/sec

      5 minute output rate 367 pkts/sec,  155148 bytes/sec

      5 minute drop rate, 3 pkts/sec

Interface Ethernet0/2 "dmz", is up, line protocol is up

  Hardware is i82546GB rev03, BW 100 Mbps, DLY 100 usec

        Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)

        Input flow control is unsupported, output flow control is off

        MAC address c84c.75ea.40c4, MTU 1500

        IP address XXX.XXX.XXX.XXX, subnet mask 255.255.255.0

        15533019 packets input, 9151896702 bytes, 0 no buffer

        Received 36 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        20167502 packets output, 5485385321 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 0 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops, 0 tx hangs

        input queue (blocks free curr/low): hardware (255/230)

        output queue (blocks free curr/low): hardware (255/7)

  Traffic Statistics for "dmz":

        15532965 packets input, 8855151966 bytes

        20167502 packets output, 5108663011 bytes

        128535 packets dropped

      1 minute input rate 200 pkts/sec,  32097 bytes/sec

      1 minute output rate 257 pkts/sec,  110022 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 243 pkts/sec,  25988 bytes/sec

      5 minute output rate 322 pkts/sec,  219609 bytes/sec

      5 minute drop rate, 0 pkts/sec

Interface Ethernet0/3 "", is administratively down, line protocol is down

  Hardware is i82546GB rev03, BW 100 Mbps, DLY 100 usec

        Auto-Duplex, Auto-Speed

        Input flow control is unsupported, output flow control is off

        Available but not configured via nameif

        MAC address c84c.75ea.40c5, MTU not set

        IP address unassigned

        0 packets input, 0 bytes, 0 no buffer

        Received 0 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        0 packets output, 0 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 1 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops, 0 tx hangs

        input queue (blocks free curr/low): hardware (255/255)

        output queue (blocks free curr/low): hardware (255/255)

Interface Management0/0 "", is administratively down, line protocol is down

  Hardware is i82557, BW 100 Mbps, DLY 100 usec

        Auto-Duplex, Auto-Speed

        Input flow control is unsupported, output flow control is unsupported

        Available but not configured via nameif

        MAC address c84c.75ea.40c6, MTU not set

        IP address unassigned

        0 packets input, 0 bytes, 0 no buffer

        Received 0 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        0 packets output, 0 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 0 interface resets

        0 babbles, 0 late collisions, 0 deferred

        0 lost carrier, 0 no carrier

        0 input reset drops, 0 output reset drops

        input queue (curr/max packets): hardware (0/0) software (0/0)

        output queue (curr/max packets): hardware (1/0) software (0/0)

The slowness is on the upload to the internet, so it's from the inside to the outside.  We made sure that the speeds and duplexes where ok, since then, no errors appear.  Since the last posting, we have found out that uploads done with FTP are at a good speeds.  So the problem seems to be limited to HTTP uploads. 

[jocamare]

There are no erros on the involved interfaces, but there are dropped packets, and a lot of them on the inside interface.

I would clear the interface counters using the "clear interfaces" command [won't cause problems] and get the "show interfaces" output 5 minutes after that and while running the test.

The problem might not be related to the protocol per se, but to the way we test the speed.

Speedtests are known for sending a bunch of small packets when running the tests, these small packets will consume the same resources a normal-sized packet [1.5 K] will use, these will affect the final results.

It doesn't all the time in all directions, but it happens.

Since you ran the tests using FTP and you got the right speed, there are no erros on the interfaces and the host is directly connected to the ASA, i would try to use a different protocol to upload the files and test the speed.

I would try to upload a file to a storage server. [fileshare,mega,etc]

[fregeus.ca]

I will try to upload with the services you mentionned.  Unfortunately, time is scarce and I may not get to it for a while.  So thanks in advance for your assistance.

Marty