Slow Traffic on Cisco IPSec VPN Tunnels - Page 2

Jake Pratt · ‎05-03-2013

We have many VPN tunnels back to our corporate office. All of these tunnels are very slow (same with our client VPN's). Our main firewall device at the corporate office is an ASA5510. We have a 100 Mb/sec Metro Ethernet internet connection here. We do not allow split-tunneling.

Our remote sites vary. We have DSL connections, cable internet connections, and other types of broadband that vary in speeds from 5 to 100 Mb/sec (up and down). The remote sites mostly have PIX 501's, but we have an ASA 5505 in one of the locations.

To take an example. On one of our remote sites that has a 100 Mb/sec connection, if I ping device to device, I'm getting ping times of about 50ms. And I'm pinging back through another 100 Mb/sec connection. If I get on a computer down there and run a speed test, I'm showing down speeds of about 1.5 Mb/sec... nowhere near 100. Some of that could be due to the lack of split tunneling, but I also suspect this could be an MTU issue. If anyone could help me figure it out, that would be great.

Right now, all my MTU's are just set to the default 1500. Perhaps this is too high. I used this site to check my max:

http://www.dslreports.com/faq/695

I did a few tests from behind several of my firewalls. I pinged from a machine on one side of the tunnel to the firewall on the other end. I'm assuming the max MTU I come up with is the max MTU for the firewall I'm behind while pinging, right? The max amounts I came up with for some of my devices were as follows:

Corporate ASA 5510 > 1272 (if you add the 28 byte packet header that would make it 1300)

Remote PIX 501 > 1416 (if you add the 28 byte packet header that would make it 1444)

Remote ASA 5505 > 1418 (if you add the 28 byte packet header that would make it 1446)

So, do I just need to set my MTU values to the appropriate amounts? I have tried changing the value, but I don't see any change in speed/performance. But I also don't know if I need to reboot the firewalls after changing the MTU. I know with Catalyst switches, you have to reload. But I didn't see any messages about needing to reboot on the ASA's/PIX's.

If anyone has some more info on it, I would greatly appreciate it. Or maybe this has nothing to do with MTU, and I'm barking up the wrong tree. I will be happy to post sanitized configs if anyone needs to see them.

Thanks

William Reed · ‎11-14-2013

Did you ever get this fixed? I am having same issues, and have similar speeds and using ASA5510(HQ) and a 5505(Remote)

Jake Pratt · ‎11-21-2013

No, I haven't really figured this out. We are still struggling with our slow speeds. Though, in all fairness, I have not yet tried the suggestions made by Eduardo Rodriguez:

i´m using ip tcp adjust-mss 1400 and have nice responses, less incorrect checksums, why don´t you give it a try?

Cisco says that for ipsec in transport mode is 1420 as mtu and for ipsec over gre 1440. but in both cases they recommend 1400.

I hope to be able to try them soon. I just haven't gotten around to it, since I'll likely need to do it after hours.

Peter Koltl · ‎11-21-2013

Have you checked CPU and other graphs in ASDM? Or

show perf

output?

What is the traffic bandwidth reported on the interfaces?

Hankvern73 · ‎02-11-2014

Maybe a stupid question, but could it be a liscense issue?

marklampi · ‎01-25-2014

sorry to keep reviving this thread but I have a very similar layout with the exact same problem. 100m internet syncronous connection at main site with ASA 5550. multiple lan to lan vpn tunnels back some with 100m as well and 2901 routers. all vpn traffic gets squeezed to 1.5 megs/sec. cpu is low on both ends. I DO however allow split tunneling so its easy for me to verify the interenet speeds at both sites are in fact 100m (I typically see speed tests around 80m). but ping tests over vpn with 1400 mtu's only hit 1.5 m/s. download speeds from internal servers, websites, ftp, tftp etc. same slow speed.

ANYONE have ANY ideas on this?

William Reed · ‎01-25-2014

Is your ISP Cogent? We have narrowed our problems down to our ISP Cogent admitting they have very high congestion at some of their peering points around the net. I have a data center about 10 miles away running on Above.NET for Internet and I can copy files at 95mbps over the VPN, but from my home Comcast connection (which Cogent admitted to having peering issues with) I get HORRIBLE sub 1mpbs speeds.

marklampi · ‎01-25-2014

no, actually we have quite a mix of ISPs. between transbeam, verizon, optimum lightpath, comcast, and even various person vpn clients on PCs that I've tested. ALL tunnels seem to get squeezed to 1.5m/s, I dont think on any combination of connections i've seen anything over 1.5m/s when riding the tunnel but in EVERY case going to the internet on these same links is significantly faster (like i said before, up to 80m/s). since all of these terminate in the ASA I would really think the problem would be there. it really seems like its being shaped somewhere but I never set up any shaping in the ASA. unless there's some type of default that I have to disable...

I wish I had enough spare equipment to set up a lab with just ethernet cables between things to rule out ISPs and really wireshark the hell out of it but since there's such a varied amount of ISP's i'm using I doubt its them. plus the internet speeds are fast so again, probably not the ISP.

thanks though...

alatechpro · ‎02-04-2014

I'm in the same boat here....We have ASA5505's out our construction trailers on remote jobsites, full tunnel mode to home office with ASA5510. We want full tunneling so that everyone is protected with our Ironport web filter. The remote sites bandwidth test way below what they should be. For example, a site with a 30mb connection may show only 1.5mb over the tunnel when running a speedtest. If I enable split-tunneling, then the speeds are reported correctly again.

I've played around with the MTU and command "sysopt connection tcpmss (1300)", but no change. With split tunneling, testing pings to the Internet does reply when using 1472 bytes, so MTU of 1500 is correct in split tunnelling mode. However, when full tunnel mode is set, the calculated MTU seems to be 1414.

ping -f -l 1388 8.8.8.8

Pinging 8.8.8.8 with 1388 bytes of data:
Request timed out.
Request timed out.

ping -f -l 1386 8.8.8.8

Pinging 8.8.8.8 with 1386 bytes of data:
Reply from 8.8.8.8: bytes=64 (sent 1386) time=52ms TTL=48
Reply from 8.8.8.8: bytes=64 (sent 1386) time=52ms TTL=48

alatechpro · ‎02-05-2014

Well, I found the fix for the speedtest results...I disabled IPSEC over TCP on the client device after reading this snippet:

However, it remains to be seen if this will work for some of our remote users. Some of our construction teams are behind hospital's infrastructure, and using IPSEC over TCP 443 was our workaround to get throught their firewalls.

http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/ezvpn505.html#wp1017851

Configuring IPSec Over TCP

By default, the Easy VPN hardware client and server encapsulate IPSec in User Datagram Protocol (UDP) packets. Some environments, such as those with certain firewall rules, or NAT and PAT devices, prohibit UDP. To use standard Encapsulating Security Protocol (ESP, Protocol 50) or Internet Key Exchange (IKE, UDP 500) in such environments, you must configure the client and the server to encapsulate IPSec within TCP packets to enable secure tunneling. If your environment allows UDP, however, configuring IPSec over TCP adds unnecessary overhead.

William Reed · ‎02-05-2014

I allow UDP so that is not my issue. We have pretty much narrowed it down to a peering point between my ISP and Comcast.

Mark Mattix · ‎05-05-2014

I also have this problem. I'm using a 5520 ASA with 8.2(2) software. I upgraded to 8.2(5) and added the command, "crypto engine large-mod-accel" but this didn't have any affect. I set up a spare 5520 in a lab and I've been able to replicate this problem. I've also tried adjusting the tcp mss adjust on LAN interfaces which didn't make a difference.

I'm working with a TAC on this issue and will update here if we find the solution.

Mark Mattix · ‎05-05-2014

Well the TAC was able to fix the problem. They changed the duplex setting on the outside interface from Full to Auto. Someone mentioned here that it could be a duplex mismatch but I thought nothing of it as my interface was set to Full. I don't understand why a duplex configured to Full when it's running at Full could cause a problem? I doubt the ISP is running half-duplex... Regardless, auto-duplex made a huge difference, I have yet to roll it out into production but I expect the same results.

Mark Mattix · ‎10-14-2014

Unfortunately, rolling out the same fix to production did not have the same effect... I've recently upgraded to a 5515X firewall and (I think 8.4 code) still not seeing improvements... I will report back if I ever resolve it...

William Reed · ‎05-05-2014

I finally fixed mine, it was my ISP Cogent. When they signed the deal with Netflix and the like, my speeds have improved dramatically.

Mark Mattix · ‎05-05-2014

Glad you got it straightened out but I think myself and a few others are having a different problem. My ISP isn't involved and I have a lab set up, I'm getting terrible speeds through the firewall. I did a packet capture on the ASA but everything seems ok. If there was a problem with the ASA configuration I'm not sure what I should be looking for in the packet capture.