Slow Throughput on L2L IPsec Tunnel

[laclairp4]

Sorry for the long post but here goes …

I am experiencing slow throughput on a L2L IPsec tunnel that we have between one of our offices on the west coast (WC) US and another on the east coast (EC) US. The tunnel endpoint on the WC resides on a 5510 and a 5545x on the EC. The DIA circuit speed on the WC is 45 Mbps and 200 Mbps on the EC. The throughput of this IPsec tunnel is maxing out at approx. 4 – 5 Mbps. The utilization of the DIA circuits at both offices is under 5% when running various FTP test transfers. Both devices have low memory and CPU utilization.

We have a 2^nd office on the EC (45 Mbps DIA) which I built a tunnel on a 5510 with the WC office and it is experiencing the same slow throughput. In covering all my bases we have a colocation facility on the WC and in building a tunnel between the 2 WC offices I WAS seeing close to full line rate speeds over the tunnel. Additionally, I built a tunnel between the 2 EC offices and I saw full line rate speeds. With the physical distance between the WC & EC offices I would expect some loss in throughput speeds but I would not expect it to drop as low as 4 – 5 Mbps. In thinking something may be up with the 5510 in our WC office we shipped a 5505 to the WC office and we built the same IPsec tunnels on it and it is experiencing the same.

In working with our support vendor to try and solve the WC <-> EC throughput issue they had me change the MTU, TCP mss, DF-bit, types of encryption/hash on the IPsec tunnel but nothing has resolved it.  We are not showing fragmentation or PMTU issues on the tunnel. In contacting the ISP of our WC office they mentioned that they do not have any type or rate limiting in place. Our WC ISP had a CCIE review our configurations but nothing was found.

Any ideas?

Thanks!

[david.tran]

Phil LaClair wrote:
Our WC ISP had a CCIE review our configurations but nothing was found.

A CCIE reviewed your configuration.  That is a really funny line .  What does that mean?  I too have a ccie but it doesn't mean anything.

Joking aside,  what is the latency between the EC and WC offices between the two VPN peers?  it should not be more than 80ms.  If it is more than 80ms, you need to talk to the provider to get it down to around 80ms.

You're able to verify the speed of the WC, that's good.  How about the speed on the EC.  The best way to do this is to put a Linux host behind the ASA on the EC and run either multiple wget to download large ISO files over the Internet to validate that you're actually getting 200Mbps on the EC connection.  That can be easily done with wget.

Now using ftp test is not ideal over the IPSEC tunnel is not ideal.  I would advise using something like Iperf or hping to test your throughput.  With Iperf or hping, you can specify the MTU to whatever you want with relative ease and you can validate this very quickly. 

My guess that you have high latency between the EC and WC office.

[nickthesun]

Hi.

Could you please give any details on this latency issue.
We are experiencing exactly the same problem.

We got 100Mbps fibre connection at our main office with 5510 on it and several branches with 5505 and channels differing from 2 to 50 Mbps.

Most of our branches are in the same city and have close to 100% of line bandwidth through tunnel to main office.

At the same time we have an branch in another country with a 30 Mbps connection.

But tunnel bandwidth doesn't get above 5 Mbps, dropping to 100 kbps when sending a bunch of small files.

The thing is we have 5-10 ms latency between branches that are in the same city with main office and that separate one has 90-100 ms.

Doubtfully we can fix latency for international communications so is there nothing else we can do?