Slow Traffic on Cisco IPSec VPN Tunnels

[Jake Pratt]

We have many VPN tunnels back to our corporate office.  All of these tunnels are very slow (same with our client VPN's).  Our main firewall device at the corporate office is an ASA5510.  We have a 100 Mb/sec Metro Ethernet internet connection here.  We do not allow split-tunneling.

Our remote sites vary.  We have DSL connections, cable internet connections, and other types of broadband that vary in speeds from 5 to 100 Mb/sec (up and down).  The remote sites mostly have PIX 501's, but we have an ASA 5505 in one of the locations.

To take an example.  On one of our remote sites that has a 100 Mb/sec connection, if I ping device to device, I'm getting ping times of about 50ms.  And I'm pinging back through another 100 Mb/sec connection.  If I get on a computer down there and run a speed test, I'm showing down speeds of about 1.5 Mb/sec... nowhere near 100.  Some of that could be due to the lack of split tunneling, but I also suspect this could be an MTU issue.  If anyone could help me figure it out, that would be great.

Right now, all my MTU's are just set to the default 1500.  Perhaps this is too high.  I used this site to check my max:

http://www.dslreports.com/faq/695

I did a few tests from behind several of my firewalls.  I pinged from a machine on one side of the tunnel to the firewall on the other end.  I'm assuming the max MTU I come up with is the max MTU for the firewall I'm behind while pinging, right?  The max amounts I came up with for some of my devices were as follows:

Corporate ASA 5510 > 1272 (if you add the 28 byte packet header that would make it 1300)

Remote PIX 501 > 1416 (if you add the 28 byte packet header that would make it 1444)

Remote ASA 5505 > 1418 (if you add the 28 byte packet header that would make it 1446)

So, do I just need to set my MTU values to the appropriate amounts?  I have tried changing the value, but I don't see any change in speed/performance.  But I also don't know if I need to reboot the firewalls after changing the MTU.  I know with Catalyst switches, you have to reload.  But I didn't see any messages about needing to reboot on the ASA's/PIX's.

If anyone has some more info on it, I would greatly appreciate it.  Or maybe this has nothing to do with MTU, and I'm barking up the wrong tree.  I will be happy to post sanitized configs if anyone needs to see them.

Thanks

[Jeremy Schreiner]

Did you ever get this resolved.  I'm having the exact same issues and have tried tcpmss, mtu, and its set to auto duplex.

Thanks,

[Jake Pratt]

No, unfortunately not.  I've been struggling with this for years, and haven't found a good way to maximize the speeds on my tunnels.  I'm sure things would go faster if I was allowing split-tunneling, but our compliance policies do not allow that.  I still would LOVE to find a good solution.

[Mark Mattix]

Are you still using the 8.2 software Axionfinancial? If you upgraded to anything newer I'd be interested knowing if it made a difference.

[Jake Pratt]

Yes, my core ASA 5510 is running 8.2(1), and I have a few different versions on my remote devices.  I think my most up-to-date ASA 5505 is running 8.2(4).

[Mark Mattix]

I haven't gotten my problem resolved yet, jeremyschreiner. I recently configured my remote sites with GRE encrypted in IPSec and it didn't make a difference. I used a TCP MSS Adjust value of 1380. Do you have the same 8.2 ASA code jeremyschreiner? I think my next step of troubleshooting is to completely replace the firewall with a 5515X series device. That won't be for a few weeks but I'll let you know how it goes.

[Jeremy Schreiner]

I still have not gotten this resolved either.  I'm working on it again currently.  I'll let the forum know if I make any progress. 

I've tried df set to clear with pre-fragment handling on.  I've tried mtu size adjustments, etc.

I'm using 3des in phase one, but aes for phase 2.

[David_Che]

Hi,

As mtu and tcp mss have been tried, and the issue is still there. I think this issue was caused by packets disorder when delivered. packet disorder was always caused by multiple links load balance in ISP, in this way some latter packet arrived before the previous packet did.

TCP 'Selective ACK (SACK)' option need to be enabled on both server and client. this can overcome packet disorder and improve the performance.

Good luck.

David

[Mark Mattix]

Do you know how to configure SACK on 2 Cisco routers? I googled it but can't find any documentation on how to configure it. Thanks a lot

[David_Che]

SACK should only be configured on pc host and server, there is no way and no effect  to configure on router. 

[arturo.ayala]

Hello all,

My links won't go over 3.5Mbps. I've been doing some testing with iperf and I can get much better speeds when I run the test with multiple threads running. I've seen up to 50Mbps over an IPSEC tunnel between the US and Europe.

From what I've read this has to do with TCP window size, and using iperf you get to change the window size for testing and see the results. One suggestion I found was to use a software or appliance that will keep a set of multiple streams/threads going between the two sides to maximize the throughput. Riverbed is a name I recognized, I'll be looking at their solution next.

[Glenn Gaines]

Has anyone found the resolution for this issue? I am having similar issues using IKEv2 and SSL tunnels. I even used a different ISP and received the same issues. It is very frustrating. Thanks...

[Raul Ricano]

Have any of you guys been able to find a good resolution to this?  I am on the same Boat.  Site to Site and Remote VPNs run Slow maybe 5/6 Mbps tops.  I've observed the Links and are barely utilizing any Bandwidth from the ISP at each end.  I've tested from TWC to TWC and still have the same issue.  This is with 100 Mbps at the main site and 50 Mbps at the remote site.  I can test downloading from a web server at the Main site from the remote site and can then max out the link, but not over the VPN tunnel.

[SamirD]

This is crazy.  I can't believe you guys are running into these type of problem with real Cisco 'big-iron' gear.  I found this thread because I'm trying to get all I can out of a 5Mbps pipe, and I get more bandwidth than you guys with your 100Mbps pipes.  I'm also running on outdated small business routers, not the real deal like you guys are!

I think the place to start in diagnosing is to get some cheap off-the-shelf $200 smb vpn routers like the rv series from Cisco.  When you can, unplug the big cisco iron and plug in these on each end, set up a tunnel and do some speed tests.  This should tell you a couple of things.

One, if you get faster speeds, it's definitely not the bandwidth, route, or isp.

Two, if you get slower or the same speeds, it could very well be the bandwidth, router, or isp.

Three, if you get significantly slower speeds, then maybe the ASAs are doing their job the best they can due to some external cause, whatever that may be.

Four, if you get significantly faster speeds, it definitely is something on the ASAs.

Good luck guys, and please post your results.

[Mark Mattix]

My IPSec tunnel problems have finally been resolved! I told my tunnels to use a maximum tunnel MTU size of 1400 and I configured the TCP adjust-mss to 1300 per the recommendation of a Cisco TAC. My fragmentation has greatly reduced which results in much faster speeds.

For anyone still having issues with their IPSec tunnel speeds, perform a wire capture and look for any fragmentation occurring. If you see any fragmentation this is most likely your issue and you have to resolve this problem. For myself Netmotion was also causing a problem by using 1428 byte packets, I had to adjust it down to 1250.

Hope everyone gets their IPSec issues sorted out!

[SamirD]

Thank you very much for the reply!  Good to know you found a solution.  What type of bandwidth are you getting through the tunnel now?