Slow upload speeds to Internet via ASA 5520.

[whiteford]

Hello,

I'm pretty sure this is a firewall issue, let me explain the issue I have.

Setup:

We have 2 ASA 5520's in Active/standby mode. The 'outside' port it connected to a VLAN on a 3750 switch where our 50mb lease line is (ISP Cisco router). The 'inside' of the ASA goes into another VLAN on the 3750 switch where our internal LAN switches are. On this 3750 switch there are various other VLANs that are sub-interfaces on the ASA via the trunk from the 3750 to ASA.

I've been running some speed test for our Internet lease line out of hours. It is a 50mb line and download speeds are around the 47mb mark which is fine.

I'm using http://speed.redstonemanaged.co.uk/ and http://www.speedtest.net/

Now the issue, the upload speeds are only ever 8-11mb and I have tried it on variuos different location on the internal LAN and get the same results.

If I go onto a server in a VLAN on the 3750 switch again I get the same issue, as the servers travel via the trunk to the ASA and out to the 'outside' interface to the VLAN where the Internet router is.

Now if I put a laptop directly into this outside VLAN on the 3750 where our 'outside' interface of the ASA is and ISP router then I get an upload of 47mb! I had to give the laptop a public IP and the gateway of the ISP router.

It just seems anything that has to pass through the firewall it has an slow issue transmitting/uploading data outbound to the Internet.

Our ASA also have the IPS module, I turned this off and it made little difference. To turn the module off (only way I know) is to use Cisco IPS Manager Express > confgiuration > Event Action Rules > Rules0 > disable event action. Also on the ASA usign the ASDM I went to Service Policy Rules and unticked the interfaces to monitor.

Can you thing of any other steps I can do? Is it a NAT/PAT issue? I am lost for ideas.

Thanks

[paulmmacd]

kusankar,

Of the three interfaces in use - inside, outside and failover, here are the results for inside and outside:

Interface Ethernet0/0 "outside", is up, line protocol is up
  Hardware is i82546GB rev03, BW 100 Mbps
Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)
MAC address 001b.d5e8.cbac, MTU 1500
IP address x.x.x.x, subnet mask 255.255.255.0
396474903 packets input, 79066447227 bytes, 0 no buffer
Received 2211277 broadcasts, 0 runts, 0 giants
0 input errors, 26224 CRC, 0 frame, 0 overrun, 26224 ignored, 0 abort
0 L2 decode drops
530985772 packets output, 550403493621 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (0/0) software (0/0)
output queue (curr/max blocks): hardware (0/250) software (0/0)
  Traffic Statistics for "outside":
404210359 packets input, 72160280929 bytes
530985772 packets output, 540544337897 bytes
2809681 packets dropped
      1 minute input rate 86 pkts/sec,  27165 bytes/sec
      1 minute output rate 72 pkts/sec,  12729 bytes/sec
      1 minute drop rate, 1 pkts/sec
      5 minute input rate 149 pkts/sec,  48188 bytes/sec
      5 minute output rate 186 pkts/sec,  160347 bytes/sec
      5 minute drop rate, 3 pkts/sec

Result of the command: "sh int e0/1"

Interface Ethernet0/1 "inside", is up, line protocol is up
  Hardware is i82546GB rev03, BW 100 Mbps
Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)
MAC address 001b.d5e8.cbad, MTU 1500
IP address 192.168.48.1, subnet mask 255.255.255.0
548573967 packets input, 557297292042 bytes, 0 no buffer
Received 17195192 broadcasts, 0 runts, 0 giants
0 input errors, 9691710 CRC, 0 frame, 0 overrun, 9691710 ignored, 0 abort
0 L2 decode drops
392477718 packets output, 77793299669 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (0/0) software (0/0)
output queue (curr/max blocks): hardware (0/51) software (0/0)
  Traffic Statistics for "inside":
548573167 packets input, 547121964625 bytes
392477718 packets output, 69454135108 bytes
15357922 packets dropped
      1 minute input rate 68 pkts/sec,  17981 bytes/sec
      1 minute output rate 69 pkts/sec,  40291 bytes/sec
      1 minute drop rate, 5 pkts/sec
      5 minute input rate 190 pkts/sec,  163545 bytes/sec
      5 minute output rate 149 pkts/sec,  49707 bytes/sec
      5 minute drop rate, 5 pkts/sec

It appears the internal interface is reporting the majority of the CRC errors, but statistically the rate seems low.  The ASA in question has been up for just over 21 days and was only rebooted to troubleshoot this problem.

I will investigate the rest of your suggestions, though I'm certain the problem is in the ASA itself because we've isolated the ASA from the internal network as a matter of course while troubleshooting.  Indeed, we wiped the configuration from one ASA to make sure it wasn't the failover capability causing the problem.  In essence we ended up with a laptop on one side, our Internet connection on the other with only the ASA in the middle and still saw the problem.  Bypassing the ASA - connecting directly to the cable on the upstream side - eliminates the problem, but I'll do due diligence.

Thanks again,

Paul

[Kureli Sankar]

Post the output for

sh run int e0/0

sh run int e0/1

If the speed and duplex is specified in the above output make sure it matches on the switch side and set the speed and duplex manually on the switch ports.  If the switch side is set to auto, change in the above to auto as well. This should clear the CRC errors.  Once these errors are gone run your tests again.

I not sure what else everyone suggested that you change on this ASA so, it would be better you attach the current config if you have any further questions on this issue.

-KS

[paulmmacd]

kusankar,

Heeding your advice, I've adjusted the switch settings for both the internal and external interfaces so they are optimal and identical between switch and ASA.  Running the "clear interface" command, then monitoring the interfaces with the 'sh int e0/0" and "sh int e0/1" commands shows no more CRC errors on either interface.

Re-running the speakeasy.net speed tests shows more symetrical, although much more varied results, including - for the first time since starting this process - an upload speed that exceeds the download speed.  While my upload speed still isn't as consistently high as my download speed, there's a definite improvement and I may be seeing the best I can get.

Here's the output of the "sh int e0/0" command:

Interface Ethernet0/0 "outside", is up, line protocol is up
  Hardware is i82546GB rev03, BW 100 Mbps
Auto-Duplex(Full-duplex), Auto-Speed(100 Mbps)
MAC address 001b.d5e8.cbac, MTU 1500
IP address x.x.x.x, subnet mask 255.255.255.0
2485674 packets input, 1620330786 bytes, 0 no buffer
Received 18893 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 L2 decode drops
2872078 packets output, 2336538828 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (0/0) software (0/0)
output queue (curr/max blocks): hardware (0/250) software (0/0)
  Traffic Statistics for "outside":
2485644 packets input, 1569397011 bytes
2872078 packets output, 2279435040 bytes
70611 packets dropped
      1 minute input rate 268 pkts/sec,  201723 bytes/sec
      1 minute output rate 294 pkts/sec,  237071 bytes/sec
      1 minute drop rate, 8 pkts/sec
      5 minute input rate 350 pkts/sec,  232823 bytes/sec
      5 minute output rate 400 pkts/sec,  307069 bytes/sec
      5 minute drop rate, 12 pkts/sec

Here's the output of the "sh int e0/1" command:

Interface Ethernet0/1 "inside", is up, line protocol is up
  Hardware is i82546GB rev03, BW 100 Mbps
Full-Duplex(Full-duplex), 100 Mbps(100 Mbps)
MAC address 001b.d5e8.cbad, MTU 1500
IP address 192.168.48.1, subnet mask 255.255.255.0
2868844 packets input, 2364562970 bytes, 0 no buffer
Received 98667 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 L2 decode drops
2427814 packets output, 1573194314 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
input queue (curr/max blocks): hardware (1/0) software (0/0)
output queue (curr/max blocks): hardware (0/51) software (0/0)
  Traffic Statistics for "inside":
2868780 packets input, 2308033896 bytes
2427814 packets output, 1523371562 bytes
81712 packets dropped
      1 minute input rate 373 pkts/sec,  238278 bytes/sec
      1 minute output rate 377 pkts/sec,  334810 bytes/sec
      1 minute drop rate, 6 pkts/sec
      5 minute input rate 385 pkts/sec,  309956 bytes/sec
      5 minute output rate 337 pkts/sec,  217388 bytes/sec
      5 minute drop rate, 7 pkts/sec

Unless you see something in the information above, or want me to post some other bit of information, I presume this is the best we can do.

Again, my thanks to you and Andy for your respective help,

Paul

[Kureli Sankar]

Excellent. These look clean.

I believe with the better results that you are seeing, the issue is resolved. You probably will notice other websites loading a lot faster than before.

If there is a big diff. between the results outside the firewall and from behind the firewall pls. look at inspections and QoS if you have them configured on the ASA as well as the client NIC drivers and speed/duplex on the NIC and switch end that you are using to do the test.

Nice job.

-KS

[chris-horne]

Andy,

I'm getting the same problem with upload speed maxing out at about 10Mb, while my download speed utilises almost the full 100Mb we have.  We have an ASA5510 with IPS. I've change the setting on the IPS to bypass it and this imediately resulted in an upload speed that matched the download, so it would appear that it is down to the IPS.  You mention about changing the RegexDepth setting for upload.  Can you tell me what commands I need to do this.

Thanks

Chris