Help with VPN speed issues using ASA 5512

[acfnetworks]

Hi Everyone,

I've been working with Cisco TAC but even they cannot resolve this issue

Infrastructure:

• I have a client that has an ASA 5510 and an ASA 5512
• Their Internet is a 50/50

Issue:

• When using the Cisco VPN client with IPSEC/UDP on the 5510, the throughput while connected remotely is fair, not the full 50/50 as expected but usable
• When connecting to the 5512 using either the Cisco VPN Client or AnyConnect (tested both ways) the speed max is about 500KB/s
• When the 5512 was first implemented everything was fine, but then one day the speeds just dropped out and wouldn't return.
• The (5512) device has a very very basic configuration on it as it's primary purpose is VPN traffic only, only consisting of what is required to connect to the internet and VPN setup using the wizard
• The 5512 was even RMA recently, the speeds improved over the previous issues but again, no where near what they should be given the client internet speeds

I'm at a loss and the client is very frustrated as they spent a lot of money on this and cannot use it properly

Thanks in advance!! Please let me know if there's any other details I need to provide

SCRUBBED SHOW RUN BELOW

ASA Version 8.6(1)2
!
hostname *************

names
!
interface GigabitEthernet0/0
nameif Outside
security-level 0
ip address *************
!
interface GigabitEthernet0/1
nameif Inside
security-level 100
ip address *************
!
interface GigabitEthernet0/2
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet0/3
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet0/4
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet0/5
shutdown
no nameif
no security-level
no ip address
!
interface Management0/0
nameif management
security-level 100
ip address 192.168.1.1 255.255.255.0
management-only
!
boot system disk0:/asa861-2-smp-k8.bin
ftp mode passive
object network NETWORK_OBJ_192.168.90.0_24
subnet 192.168.90.0 255.255.255.0
access-list INTERNAL standard permit *************
access-list INTERNAL standard permit *************
pager lines 24
logging asdm informational
mtu Outside 1500
mtu Inside 1500
mtu management 1500
ip local pool SSL_VPN_POOL ************* mask 255.255.255.0
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-66114.bin
no asdm history enable
arp timeout 14400
nat (Inside,Outside) source static any any destination static NETWORK_OBJ_*************_24 NETWORK_OBJ_*************_24 no-proxy-arp route-lookup
route Outside 0.0.0.0 0.0.0.0 ************* 1
route Inside ************* 255.255.255.0 ************* 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
user-identity default-domain LOCAL
http server enable
http 192.168.1.0 255.255.255.0 management
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart warmstart
crypto ipsec ikev1 transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-DES-SHA esp-des esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-DES-MD5 esp-des esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac
crypto ipsec ikev1 transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac
crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 65535 set ikev1 transform-set ESP-AES-128-SHA ESP-AES-128-MD5 ESP-AES-192-SHA ESP-AES-192-MD5 ESP-AES-256-SHA ESP-AES-256-MD5 ESP-3DES-SHA ESP-3DES-MD5 ESP-DES-SHA ESP-DES-MD5
crypto map Outside_map 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP
crypto map Outside_map interface Outside
crypto ikev1 enable Outside
crypto ikev1 policy 10
authentication crack
encryption aes-256
hash sha
group 2
lifetime 86400
crypto ikev1 policy 20
authentication rsa-sig
encryption aes-256
hash sha
group 2
lifetime 86400
crypto ikev1 policy 30
authentication pre-share
encryption aes-256
hash sha
group 2
lifetime 86400
crypto ikev1 policy 40
authentication crack
encryption aes-192
hash sha
group 2
lifetime 86400
crypto ikev1 policy 50
authentication rsa-sig
encryption aes-192
hash sha
group 2
lifetime 86400
crypto ikev1 policy 60
authentication pre-share
encryption aes-192
hash sha
group 2
lifetime 86400
crypto ikev1 policy 70
authentication crack
encryption aes
hash sha
group 2
lifetime 86400
crypto ikev1 policy 80
authentication rsa-sig
encryption aes
hash sha
group 2
lifetime 86400
crypto ikev1 policy 90
authentication pre-share
encryption aes
hash sha
group 2
lifetime 86400
crypto ikev1 policy 100
authentication crack
encryption 3des
hash sha
group 2
lifetime 86400
crypto ikev1 policy 110
authentication rsa-sig
encryption 3des
hash sha
group 2
lifetime 86400
crypto ikev1 policy 120
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
crypto ikev1 policy 130
authentication crack
encryption des
hash sha
group 2
lifetime 86400
crypto ikev1 policy 140
authentication rsa-sig
encryption des
hash sha
group 2
lifetime 86400
crypto ikev1 policy 150
authentication pre-share
encryption des
hash sha
group 2
lifetime 86400
telnet timeout 5
ssh timeout 5
console timeout 0
dhcpd address 192.168.1.2-192.168.1.254 management
dhcpd enable management
!
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
ssl encryption aes128-sha1
webvpn
enable Outside
anyconnect-essentials
anyconnect image disk0:/anyconnect-win-4.2.01035-k9.pkg 1
anyconnect enable
tunnel-group-list enable
group-policy GroupPolicy_************* internal
group-policy GroupPolicy_************* attributes
wins-server none
dns-server value *************
vpn-tunnel-protocol ssl-client
split-tunnel-policy tunnelspecified
split-tunnel-network-list value INTERNAL
default-domain value *************
group-policy GroupPolicy1 internal
group-policy GroupPolicy1 attributes
vpn-tunnel-protocol ikev1 ssl-client
group-policy ************* internal
group-policy ************* attributes
dns-server value *************
vpn-tunnel-protocol ikev1
password-storage disable
split-tunnel-policy tunnelspecified
split-tunnel-network-list value INTERNAL
default-domain value *************

tunnel-group ************* type remote-access
tunnel-group ************* general-attributes
address-pool SSL_VPN_POOL
default-group-policy GroupPolicy_*************
tunnel-group ************* webvpn-attributes
group-alias ************* enable
tunnel-group ************* type remote-access
tunnel-group ************* general-attributes
address-pool SSL_VPN_POOL
default-group-policy *************
tunnel-group ************* ipsec-attributes
ikev1 pre-shared-key *****
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum client auto
message-length maximum 512
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect rsh
inspect rtsp
inspect esmtp
inspect sqlnet
inspect skinny
inspect sunrpc
inspect xdmcp
inspect sip
inspect netbios
inspect tftp
inspect ip-options
!
service-policy global_policy global
prompt hostname context
no call-home reporting anonymous
Cryptochecksum:dde8b11d25191e8de37cce99bccd68ca

[Karsten Iwen]

Don't think that it's the cause of the problem, but you are using one of the first releases of the ASA. Whenever you face a problem that can't be directly solved, I would upgrade the device to a recent release and look if it get's any better.

[acfnetworks]

I just finished updating the ASA and ASDM images to the latest available after it was pointed out that they are old. This was the unit I received from Cisco and didn't think it would have come with the original OS so I didn't think to update it. Either way, after the updates, the issue is still present

[mattjones03]

Hi,

When connecting to your ISP/Internet from inside the network, are you observing the full circuit performance of 50Mb/s?

Just some other questions;

1. What are the firewalls connecting back into from an ISP/external connectivity perspective?
   
   
2. If you disconnect one of the firewalls from the network (maybe the 5510, during a maintenance window, does the performance improve)?
   
   
3. Can you provide the output of the following commands on both the 5510 and 5512;

"sh int outside"

"sh int inside"

[acfnetworks]

When I complete a speedtest for example, I get the full or close to full 50/50 depending on network utilization

1. The 5510/5512 connect into a managed switch which then connects to the ISP equipment, have also rulled this out by using the same port that the 5510 was connected to and moving the 5510 to a different port. The 5510 performance remains consistanlty good while the 5512 stays bad

2. Completed this, performance does not improve

3. 

sh int outside
Interface GigabitEthernet0/0 "Outside", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Full-Duplex(Full-duplex), 1000 Mbps(1000 Mbps)
Input flow control is unsupported, output flow control is off
MAC address 88f0.*************, MTU 1500
IP address *************, subnet mask 255.255.255.240
118341 packets input, 103446198 bytes, 0 no buffer
Received 2961 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
129356 packets output, 104836617 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 2 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (496/459)
output queue (blocks free curr/low): hardware (488/442)
Traffic Statistics for "Outside":
118341 packets input, 101312310 bytes
129356 packets output, 102507817 bytes
280 packets dropped
1 minute input rate 0 pkts/sec, 39 bytes/sec
1 minute output rate 0 pkts/sec, 46 bytes/sec
1 minute drop rate, 0 pkts/sec
5 minute input rate 1 pkts/sec, 61 bytes/sec
5 minute output rate 0 pkts/sec, 41 bytes/sec
5 minute drop rate, 0 pkts/sec

sh int inside
Interface GigabitEthernet0/1 "Inside", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
MAC address 88f0.*************, MTU 1500
IP address *************, subnet mask 255.255.254.0
178860 packets input, 100486678 bytes, 0 no buffer
Received 40990 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
129111 packets output, 99069721 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 1 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (503/453)
output queue (blocks free curr/low): hardware (503/432)
Traffic Statistics for "Inside":
178860 packets input, 97038329 bytes
129111 packets output, 96537533 bytes
18837 packets dropped
1 minute input rate 13 pkts/sec, 1088 bytes/sec
1 minute output rate 1 pkts/sec, 619 bytes/sec
1 minute drop rate, 4 pkts/sec
5 minute input rate 13 pkts/sec, 1150 bytes/sec
5 minute output rate 1 pkts/sec, 608 bytes/sec
5 minute drop rate, 5 pkts/sec

[mattjones03]

From the outputs provided, all interfaces are looking good, and your speed test back this up.

The managed switch that you have in place, are all interfaces associated with connectivity to the ISP hardware looking good (from an interface statistics perspective)?

Does the managed switch, or ISP, have any protocol specific rate limiting in place potentially?

[acfnetworks]

The switch is dedicated for the ISP connectivity since we run multiple firewalls and the ISP only provides one port on their device. 

There is nothing to limit the rate of anything that I'm aware of, the VPN throughput is the only thing that is having an issues, on the other firewalls I'm getting ok speeds. When downloading or uploading files for example there are no issues with throughput. 
If I create a site to site VPN, the throughput is perfect

[mattjones03]

Have Cisco TAC advised that you reduce the TCP MSS value on your ASA?

[acfnetworks]

They didn't no, I tried changing the MTU size on the Outside interface to 1300

I've also tried removing the Force maximum segment size for TCP under Firewall > Advanced > TCP Options and also changing it to 1300 but no dice

In my googling, I tried running ip tcp adjust-mss but it doesn't appear to be a valid command so I'm not sure if it's something different now

Edit - Seeing how the VPN connects via UDP, this command shouldn't really effect anything as it's TCP related only?

[mattjones03]

You mentioned Anyconnect as well. The mss will have an impact on the overall packet size, and if they are exceeding the defined mtu size (1500) then you will be experiencing fragmentation of packets.

Next steps will be to setup a SPAN on the managed switch to complete some packet analysis, or complete a packet capture via the ASA.