Slow Traffic on Cisco IPSec VPN Tunnels

Jake Pratt · ‎05-03-2013

We have many VPN tunnels back to our corporate office. All of these tunnels are very slow (same with our client VPN's). Our main firewall device at the corporate office is an ASA5510. We have a 100 Mb/sec Metro Ethernet internet connection here. We do not allow split-tunneling.

Our remote sites vary. We have DSL connections, cable internet connections, and other types of broadband that vary in speeds from 5 to 100 Mb/sec (up and down). The remote sites mostly have PIX 501's, but we have an ASA 5505 in one of the locations.

To take an example. On one of our remote sites that has a 100 Mb/sec connection, if I ping device to device, I'm getting ping times of about 50ms. And I'm pinging back through another 100 Mb/sec connection. If I get on a computer down there and run a speed test, I'm showing down speeds of about 1.5 Mb/sec... nowhere near 100. Some of that could be due to the lack of split tunneling, but I also suspect this could be an MTU issue. If anyone could help me figure it out, that would be great.

Right now, all my MTU's are just set to the default 1500. Perhaps this is too high. I used this site to check my max:

http://www.dslreports.com/faq/695

I did a few tests from behind several of my firewalls. I pinged from a machine on one side of the tunnel to the firewall on the other end. I'm assuming the max MTU I come up with is the max MTU for the firewall I'm behind while pinging, right? The max amounts I came up with for some of my devices were as follows:

Corporate ASA 5510 > 1272 (if you add the 28 byte packet header that would make it 1300)

Remote PIX 501 > 1416 (if you add the 28 byte packet header that would make it 1444)

Remote ASA 5505 > 1418 (if you add the 28 byte packet header that would make it 1446)

So, do I just need to set my MTU values to the appropriate amounts? I have tried changing the value, but I don't see any change in speed/performance. But I also don't know if I need to reboot the firewalls after changing the MTU. I know with Catalyst switches, you have to reload. But I didn't see any messages about needing to reboot on the ASA's/PIX's.

If anyone has some more info on it, I would greatly appreciate it. Or maybe this has nothing to do with MTU, and I'm barking up the wrong tree. I will be happy to post sanitized configs if anyone needs to see them.

Thanks

rpadwal · ‎05-04-2013

Hi Jake,

Please post sanitized configs

Please mention the upload and the download speeds of the terminating endpoints.

ASA5510

PIX501

ASA5505

Thanks and Regards,

ROHAN

Thanks and Regards, ROHAN :)

Jake Pratt · ‎05-07-2013

Ok, for this example, I'll use 3 offices: Corporate and 2 remotes.

Corporate: ASA 5510. 100M down/100M up (Metro Ethernet)

Remote1: Cisco PIX 501 100M down/100M up (local broadband provider, not sure type)

Remote2: ASA 5505. 14M down/3M up (local broadband provider, not sure type)

And here are the configs. I heavily sanitized them, and pulled out items that shouldn't be relative. Let me know if some of these need to be included.

Corporate ASA 5510:

#####################################################

ASA Version 8.2(1)

!

hostname

domain-name

!

interface Ethernet0/0

speed 100

duplex full

nameif Outside

security-level 0

ip address standby

!

interface Ethernet0/1

speed 100

duplex full

nameif Inside

security-level 100

ip address standby

!

interface Ethernet0/2

shutdown

no nameif

no security-level

no ip address

!

interface Ethernet0/3

description LAN/STATE Failover Interface

!

interface Management0/0

nameif management

security-level 100

ip address

management-only

!

ftp mode passive

clock timezone MST -7

clock summer-time MDT recurring

dns domain-lookup Outside

dns server-group DefaultDNS

name-server

domain-name

same-security-traffic permit intra-interface

pager lines 24

logging enable

logging buffer-size 1000000

logging monitor debugging

logging buffered debugging

logging trap notifications

logging asdm informational

logging host Inside

mtu Outside 1500

mtu Inside 1500

mtu management 1500

ip local pool

failover

failover lan unit primary

failover lan interface failover Ethernet0/3

failover key *****

failover replication http

failover link failover Ethernet0/3

failover interface ip failover standby

icmp unreachable rate-limit 1 burst-size 1

asdm image disk0:/asdm-649-103.bin

no asdm history enable

arp timeout 14400

global (Outside) 1 interface

nat (Outside) 0 access-list NONAT

nat (Outside) 1 255.255.0.0

nat (Inside) 0 access-list NONAT

nat (Inside) 1 255.255.255.0

nat (Inside) 1 255.255.0.0

access-group outside-in in interface Outside

access-group inside-out in interface Inside

route Outside 0.0.0.0 0.0.0.0 1

route Inside 255.255.0.0 1

route Outside 255.255.255.0 1

timeout xlate 3:00:00

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02

timeout sunrpc 0:10:00 h323 0:00:00 h225 0:00:00 mgcp 0:05:00 mgcp-pat 0:05:00

timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00

timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute

timeout tcp-proxy-reassembly 0:01:00

dynamic-access-policy-record DfltAccessPolicy

aaa-server ADLDAP protocol ldap

aaa authentication ssh console LOCAL

http server enable

http 0.0.0.0 0.0.0.0 Inside

no snmp-server location

no snmp-server contact

snmp-server community *****

snmp-server enable traps snmp authentication linkup linkdown coldstart

service resetoutside

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac

crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac

crypto ipsec transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac

crypto ipsec transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac

crypto ipsec transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac

crypto ipsec transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac

crypto ipsec transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac

crypto ipsec transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac

crypto ipsec transform-set ESP-DES-SHA esp-des esp-sha-hmac

crypto ipsec transform-set ESP-DES-MD5 esp-des esp-md5-hmac

crypto ipsec transform-set my-set esp-3des esp-md5-hmac

crypto ipsec security-association lifetime seconds 28800

crypto ipsec security-association lifetime kilobytes 4608000

crypto dynamic-map companyvpn 10 set pfs group1

crypto dynamic-map companyvpn 10 set transform-set my-set

crypto dynamic-map companyvpn 10 set reverse-route

crypto map VPN 10 match address

crypto map VPN 10 set peer

crypto map VPN 10 set transform-set my-set

crypto map VPN 10 set security-association lifetime seconds 28800

crypto map VPN 10 set security-association lifetime kilobytes 4608000

crypto map VPN 30 match address

crypto map VPN 30 set peer

crypto map VPN 30 set transform-set my-set

crypto map VPN 30 set security-association lifetime seconds 28800

crypto map VPN 30 set security-association lifetime kilobytes 4608000

crypto map VPN 65535 ipsec-isakmp dynamic companyvpn

crypto map VPN interface Outside

crypto isakmp identity address

crypto isakmp enable Outside

crypto isakmp enable management

crypto isakmp policy 10

authentication pre-share

encryption 3des

hash md5

group 2

lifetime 86400

crypto isakmp policy 30

authentication pre-share

encryption 3des

hash md5

group 5

lifetime 86400

crypto isakmp ipsec-over-tcp port 10000

vpn-addr-assign local reuse-delay 120

telnet timeout 5

ssh 0.0.0.0 0.0.0.0 Inside

ssh timeout 15

console timeout 0

management-access Inside

dhcpd address management

dhcpd enable management

!

threat-detection basic-threat

threat-detection statistics host

threat-detection statistics access-list

no threat-detection statistics tcp-intercept

dynamic-filter use-database

dynamic-filter enable interface Outside classify-list botnet-exclude

ntp server source Outside

ntp server source Outside prefer

webvpn

group-policy DefaultRAGroup internal

group-policy DefaultRAGroup attributes

dns-server value

vpn-tunnel-protocol IPSec

default-domain value

group-policy companyvpn internal

group-policy companyvpn attributes

banner value

dns-server value

vpn-tunnel-protocol IPSec

group-lock value companyvpn

default-domain value

username encrypted privilege 15

tunnel-group type ipsec-l2l

tunnel-group ipsec-attributes

pre-shared-key *

tunnel-group companyvpn type remote-access

tunnel-group companyvpn general-attributes

address-pool

authentication-server-group ADLDAP LOCAL

authentication-server-group (Inside) ADLDAP

default-group-policy companyvpn

tunnel-group companyvpn ipsec-attributes

pre-shared-key *

tunnel-group type ipsec-l2l

tunnel-group ipsec-attributes

pre-shared-key *

!

class-map inspection_default

match default-inspection-traffic

class-map botnet-DNS

match port udp eq domain

class-map ips_class_map

match access-list traffic_for_ips

!

policy-map type inspect dns preset_dns_map

parameters

message-length maximum 512

policy-map global_policy

class inspection_default

inspect ftp

inspect h323 ras

inspect rsh

inspect rtsp

inspect sqlnet

inspect skinny

inspect sunrpc

inspect xdmcp

inspect sip

inspect netbios

inspect tftp

inspect dns preset_dns_map

class ips_class_map

ips inline fail-open

policy-map botnet-policy

class botnet-DNS

inspect dns dynamic-filter-snoop

!

service-policy global_policy global

service-policy botnet-policy interface Outside

prompt hostname context

#####################################################

Remote1 PIX 501 Config:

#####################################################

PIX Version 6.3(5)

interface ethernet0 auto

interface ethernet1 100full

nameif ethernet0 outside security0

nameif ethernet1 inside security100

hostname

domain-name

fixup protocol dns maximum-length 512

fixup protocol ftp 21

fixup protocol h323 h225 1720

fixup protocol h323 ras 1718-1719

fixup protocol http 80

fixup protocol rsh 514

fixup protocol rtsp 554

fixup protocol sip 5060

fixup protocol sip udp 5060

fixup protocol skinny 2000

fixup protocol smtp 25

fixup protocol sqlnet 1521

fixup protocol tftp 69

names

access-list vpn-acl permit ip 255.255.255.0 any

access-list nonat-acl permit ip 255.255.255.0 any

pager lines 24

logging on

logging trap debugging

logging host inside

mtu outside 1472

mtu inside 1500

ip address outside 255.255.255.0

ip address inside 255.255.255.0

ip audit info action alarm

ip audit attack action alarm

pdm logging informational 100

pdm history enable

arp timeout 14400

nat (inside) 0 access-list nonat-acl

route outside 0.0.0.0 0.0.0.0

timeout xlate 0:05:00

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 rpc 0:10:00 h225 1:00:00

timeout h323 0:05:00 mgcp 0:05:00 sip 0:30:00 sip_media 0:02:00

timeout sip-disconnect 0:02:00 sip-invite 0:03:00

timeout uauth 0:05:00 absolute

aaa-server TACACS+ protocol tacacs+

aaa-server TACACS+ max-failed-attempts 3

aaa-server TACACS+ deadtime 10

aaa-server RADIUS protocol radius

aaa-server RADIUS max-failed-attempts 3

aaa-server RADIUS deadtime 10

aaa-server LOCAL protocol local

http server enable

http 255.255.255.0 inside

no snmp-server location

no snmp-server contact

snmp-server community public

no snmp-server enable traps

floodguard enable

sysopt connection permit-ipsec

crypto ipsec transform-set vpnset esp-3des esp-md5-hmac

crypto map vpnmap 10 ipsec-isakmp

crypto map vpnmap 10 match address vpn-acl

crypto map vpnmap 10 set peer

crypto map vpnmap 10 set transform-set vpnset

crypto map vpnmap interface outside

isakmp enable outside

isakmp key ******** address netmask 255.255.255.255

isakmp policy 10 authentication pre-share

isakmp policy 10 encryption 3des

isakmp policy 10 hash md5

isakmp policy 10 group 2

isakmp policy 10 lifetime 86400

telnet 255.255.255.0 inside

telnet timeout 5

ssh 0.0.0.0 0.0.0.0 outside

ssh 0.0.0.0 0.0.0.0 inside

ssh timeout 5

console timeout 0

terminal width 80

######################################

Remote 2 ASA 5505 Config:

######################################

ASA Version 7.2(3)

!

hostname

domain-name

enable

names

!

interface Vlan1

nameif inside

security-level 100

ip address 255.255.255.0

!

interface Vlan2

nameif outside

security-level 0

ip address 255.255.255.248

!

interface Ethernet0/0

switchport access vlan 2

!

interface Ethernet0/1

speed 100

duplex full

!

interface Ethernet0/2

!

interface Ethernet0/3

!

interface Ethernet0/4

!

interface Ethernet0/5

!

interface Ethernet0/6

!

interface Ethernet0/7

!

passwd encrypted

ftp mode passive

dns server-group

domain-name

access-list nonat extended permit ip 255.255.255.0 any

access-list 101 extended permit ip 255.255.255.0 any

access-list from-internet extended permit icmp any any

pager lines 24

logging enable

logging monitor debugging

logging trap debugging

logging asdm informational

logging host inside

mtu inside 1500

mtu outside 1500

icmp unreachable rate-limit 1 burst-size 1

asdm image disk0:/asdm-523.bin

no asdm history enable

arp timeout 14400

global (outside) 1 interface

nat (inside) 0 access-list nonat

nat (inside) 1 0.0.0.0 0.0.0.0

access-group from-internet in interface outside

route outside 0.0.0.0 0.0.0.0

timeout xlate 3:00:00

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02

timeout sunrpc 0:10:00 h323 0:00:00 h225 0:00:00 mgcp 0:05:00 mgcp-pat 0:05:00

timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00

timeout uauth 0:05:00 absolute

http server enable

http 0.0.0.0 0.0.0.0 inside

no snmp-server location

no snmp-server contact

snmp-server enable traps snmp authentication linkup linkdown coldstart

crypto ipsec transform-set my-set esp-3des esp-md5-hmac

crypto map mymap 20 match address 101

crypto map mymap 20 set peer

crypto map mymap 20 set transform-set my-set

crypto map mymap interface outside

crypto isakmp enable outside

crypto isakmp policy 10

authentication pre-share

encryption 3des

hash md5

group 2

lifetime 86400

crypto isakmp nat-traversal 20

telnet 0.0.0.0 0.0.0.0 inside

telnet timeout 5

ssh 0.0.0.0 0.0.0.0 outside

ssh timeout 5

console timeout 0

management-access inside

!

class-map inspection_default

match default-inspection-traffic

!

policy-map type inspect dns preset_dns_map

parameters

message-length maximum 512

policy-map global_policy

class inspection_default

inspect dns preset_dns_map

inspect ftp

inspect h323 ras

inspect rsh

inspect rtsp

inspect esmtp

inspect sqlnet

inspect skinny

inspect sunrpc

inspect xdmcp

inspect sip

inspect netbios

inspect tftp

!

service-policy global_policy global

username encrypted privilege 15

tunnel-group type ipsec-l2l

tunnel-group ipsec-attributes

pre-shared-key *

prompt hostname context

Muhammad Zahid · ‎01-03-2016

@Jake Did you find the fix? I'm facing similar issues.

Please Rate Posts (by clicking on Star) and/or Mark Solutions as Accepted, when applies

Raul Ricano · ‎01-14-2016

I used iPerf for bandwidth testing through the VPN tunnels. Actually found out my application was using TCP and it was getting Max speeds topping out at 5/6 Mbps due to TCP Window size. I recommend installing a FTP server on one end and running multiple streams from the other side to really test the full tunnel's throughput. I was able to increase the TCP window size of the systems and push more Data through the VPN tunnels.

Speed Test tool. Now has TCP & UDP testing.

https://iperf.fr/

Good Article about TCP & Bandwidth Calculation.

http://bradhedlund.com/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/

and a calculator:

https://www.switch.ch/network/tools/tcp_throughput/

Example is 100Mbps with 80 msec latency.

Bandwidth-delay Product and buffer size

BDP (100 Mbit/sec, 80.0 ms) = 1.00 MByte
required tcp buffer to reach 100 Mbps with RTT of 80.0 ms >= 976.6 KByte
maximum throughput with a TCP window of 64 KByte and RTT of 80.0 ms <= 6.55 Mbit/sec.

Kangalala · ‎03-28-2016

hi, Im facing same issue with gre tunnel connecting two sites, has anybody found the fix yet?

Kenneth Sharp · ‎05-07-2013

Jake,

I'm not an expert with the ASA, but earlier this year I was troubleshooting performance on our ASA, and found this helpful document:

http://www.cisco.com/en/US/products/ps6120/products_tech_note09186a0080ba9521.shtml

We found a duplex mismatch between our ASA and the provider's equipment. A rookie mistake, but what a difference it made once it was fixed.

I hope this helps you, or gives you some other things to try.

- Ken

Jake Pratt · ‎05-07-2013

Thanks for the advise. I have double-checked, and it's not something that simple. We are running 100 Full on all of these devices. No duplex mismatches.

rpadwal · ‎05-07-2013

Hi Jake,

Please follow the below doc

http://www.cisco.com/en/US/products/ps6120/products_configuration_example09186a008081e621.shtml

follow the ipsec tunnel mode section in the belwo doc.

http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml#t15

http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml#more

Please try and adjust the tcp mss value you can check the tcp mss by

sh run all sysopt

Eg: for testing you can use a value of 1300

please attach a wireshark capture from a host trying to communiacte across the VPN capture should be take on both the ends of the tunnel simultaneously.

use the below link for packet analysis as a reference for the measuring throughput,

http://www.dslreports.com/faq/15888

Thanks and Regards,

ROHAN

Thanks and Regards, ROHAN :)

Jake Pratt · ‎05-08-2013

I have set my tcp mss to 1300 on my corporate ASA, and on my remote PIX and ASA. I am not seeing any change in speed at all. I ran a wireshark packet capture (before making the change), while transferring a bunch of FTP traffic accross the tunnel. I haven't posted the captures yet, because I'm a little weary about posting that online. Are we just looking for packet fragmentation? When looking through my capture. The largest packets I could see were 1260 bytes of FTP data, and the packet size was 1314.

In the first article you sent, it mentions making changes on both the ASA and the "router". Is this referring to the ISP's router? That is the only device I see in the diagram. If so, I am not able to modify that device.

I'm not sure if I need to set my tcp mss as well as manually setting a lower MTU. Or if I also need to try setting PMTUD.

Let me know what you want to see from wireshark. And thanks for your help!

Jake Pratt · ‎05-08-2013

I have been focusing on my remote site that has 100mbps up/down, since I have a 100mbps up/down pipe on both ends. I set the tcpmss and the mtu to 1300 on both ends. I am seeing no difference. My speed tests from the remote site are about 1.3 mbps down and 1.7 mbps up. Based on speeds in the corporate office, that is about an 80+% loss. I'll post wireshark screenshots if there is something specific you are looking for.

I am seeing a lot of bad checksum headers. Many of those have a length much higher than 1300 bytes. Here is just a sample of the scan from my remote machine. You can see the DF flag is set, and there are lots of bad header checksum errors.

Marvin Rhoads · ‎05-08-2013

Don't worry about the bad header checksum errors. Almost any modern adapater and operating system will throw those errors in Wireshark because (as the higlighted line in the screenshot suggests) IP checksum offload often causes this. There's even a setting in Wireshark to turn off that particular check since it's almost always seen in raw captures. The problem is that the captures get the traffic after the NIC has already done the checksum and thus it looks "wrong".

I turn it off on any Wireshark installation I use:

http://packetlife.net/blog/2008/aug/23/disabling-checksum-validation-wireshark/

Jake Pratt · ‎05-09-2013

Thanks for the tip. I already had TCP and UDP checksums turned off, but I had to disable the IP4 check as well.

I just ran another test from the remote side. This time, I was doing a speed test online (which still comes accross the tunnel, since I don't allow split-tunnelling). So this was all HTTP traffic. I get a whole bunch of "[TCP segment of a reassembled PDU]" packets. Don't know if that's normal or not:

Jake Pratt · ‎05-09-2013

After setting MTU and tcpmss to 1300 on both ends, my speeds were horrible. Way worse than before. My ping times were a little slower, but the amount of time it took to do anything was ridiculous. Everyone was complaining that they could not function, so I set everything back to the default 1500. I'm not sure what to do next. This is pretty frustrating.

Edgardo Rodriguez · ‎07-18-2013

i´m using ip tcp adjust-mss 1400 and have nice responses, less incorrect checksums, why don´t you give it a try?

Cisco says that for ipsec in transport mode is 1420 as mtu and for ipsec over gre 1440. but in both cases they recommend 1400.

Good luck.