Re: ASAv on AWS slow internet bandwidth

piwo030 · ‎03-05-2021

Hi Folks,

I’m a little frustrated right now and really hope you can help me fixing one major performance issue we have.

Short description: We are running a Cisco ASAv in AWS to connect into our cloud infrastructure over there. All traffic is routed through the SSL VPN connection. That means we use the internet outbreak from AWS while we are connected with AnyConnect.

Unfortunately the bandwidth performance is horrible and I can’t find the issues.

Attached are some screenshots where you can see the differences while I am connected with VPN.

Let me provide as much and detailed information as possible:

Home office details:

Internet line with 250 Mbps in Download and 15 Mbps in Upload

AWS ASAv Instance:

c5.large instance with 10Gbps network interfaces
3x network interfaces: Management (172.20.19.250), OUTSIDE (172.20.10.250) and INSIDE (172.20.1.250)

AWS Environment

2 public subnets (1 for Management, 1 for OUTSIDE)
1 private subnet (for INSIDE)
Elastic IPs attached on Management ENI and OUTSIDE ENI
The internet connection for INSIDE ENI is going through a NAT Gateway and Internet Gateway

When I do a speedtest on my servers in my public and private subnets, I got around 500Mbits in up and download.

I have also attached a network plan for you.

Hope you can find the issue.

Here comes my ASAv configuration:

: Saved


:

: Serial Number: XXXXXXXXXXX

: Hardware:   ASAv, 4096 MB RAM, CPU Xeon 4100/6100/8100 series 3599 MHz, 1 CPU (2 cores)

:

ASA Version 9.15(1)1

!

hostname XXXXASA01

domain-name XXX.XXXXXX.XX

enable password ***** pbkdf2

service-module 0 keepalive-timeout 4

service-module 0 keepalive-counter 6

names

no mac-address auto

ip local pool VPN-Pool 10.0.250.1-10.0.250.254 mask 255.255.255.0


!

interface Management0/0

 management-only

 nameif management

 security-level 100

 ip address dhcp setroute

!

interface TenGigabitEthernet0/0

 nameif OUTSIDE

 security-level 0

 ip address 172.20.10.250 255.255.255.0

!

interface TenGigabitEthernet0/1

 nameif INSIDE

 security-level 100

 ip address 172.20.1.250 255.255.255.0

!

ftp mode passive

clock timezone CEST 1

clock summer-time CEDT recurring last Sun Mar 2:00 last Sun Oct 3:00

dns domain-lookup management

dns server-group DefaultDNS

 name-server 172.20.1.10 management

 domain-name XXX.XXXXXX.XX

same-security-traffic permit inter-interface

same-security-traffic permit intra-interface

pager lines 23

mtu management 1500

mtu OUTSIDE 1500

mtu INSIDE 1500

no failover

no failover wait-disable

no monitor-interface service-module

icmp unreachable rate-limit 1 burst-size 1

no asdm history enable

arp timeout 14400

no arp permit-nonconnected

arp rate-limit 8192

nat (OUTSIDE,INSIDE) source dynamic any interface

route OUTSIDE 0.0.0.0 0.0.0.0 172.20.10.1 1

route INSIDE 0.0.0.0 0.0.0.0 172.20.1.1 tunneled

timeout xlate 3:00:00

timeout pat-xlate 0:00:30

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02

timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00

timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00

timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute

timeout tcp-proxy-reassembly 0:01:00

timeout floating-conn 0:00:00

timeout conn-holddown 0:00:15

timeout igp stale-route 0:01:10

aaa-server LDAP-Servers protocol ldap

aaa-server LDAP-Servers (INSIDE) host 172.20.1.10

 ldap-base-dn DC=XXX,DC=XXXXXX,DC=XX

 ldap-group-base-dn OU=XXXX - Groups,DC=XXX,DC=XXXXXX,DC=XX

 ldap-scope subtree

 ldap-naming-attribute sAMAccountName

 ldap-login-password *****

 ldap-login-dn CN=Cisco ASAv LDAP Connect,OU=XXXX - Service Accounts,DC=XXX,DC=XXXXXX,DC=XX

 server-type microsoft

user-identity default-domain LOCAL

aaa authentication login-history

http server enable

http 0.0.0.0 0.0.0.0 management

http 0.0.0.0 0.0.0.0 OUTSIDE

http redirect management 80

http redirect OUTSIDE 80

http redirect INSIDE 80

no snmp-server location

no snmp-server contact

crypto ipsec ikev2 ipsec-proposal AES256

 protocol esp encryption aes-256

 protocol esp integrity sha-1

crypto ipsec ikev2 ipsec-proposal AES192

 protocol esp encryption aes-192

 protocol esp integrity sha-1

crypto ipsec ikev2 ipsec-proposal AES

 protocol esp encryption aes

 protocol esp integrity sha-1

crypto ipsec ikev2 ipsec-proposal 3DES

 protocol esp encryption aes

 protocol esp integrity sha-1

crypto ipsec ikev2 ipsec-proposal DES

 protocol esp encryption aes

 protocol esp integrity sha-1

crypto ipsec security-association pmtu-aging infinite

crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 65535 set ikev2 ipsec-proposal AES256 AES192 AES 3DES DES

crypto map OUTSIDE_map 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP

crypto map OUTSIDE_map interface OUTSIDE

crypto ca trustpoint _SmartCallHome_ServerCA

 no validation-usage

 crl configure

crypto ca trustpoint ComodoRSA

 enrollment terminal

 subject-name CN=XXXXX.XXXXXX.XX,OU=XX XXXXX,O=XXXX,C=XX,St=XXXXX,EA=XXXXXXXX

 crl configure

crypto ca trustpoint COMODORSA

 enrollment terminal

 crl configure

crypto ca trustpool policy

 auto-import

crypto ca certificate chain _SmartCallHome_ServerCA

 certificate ca 0509

 ...

  quit

crypto ca certificate chain ComodoRSA

 certificate XXXX

 ...

  quit

crypto ca certificate chain COMODORSA

 certificate ca XXXX

 ...

  quit

crypto ikev2 policy 1

 encryption aes-256

 integrity sha256

 group 5

 prf sha256

 lifetime seconds 86400

crypto ikev2 policy 10

 encryption aes-192

 integrity sha256

 group 5

 prf sha256

 lifetime seconds 86400

crypto ikev2 policy 20

 encryption aes

 integrity sha256

 group 5

 prf sha256

 lifetime seconds 86400

crypto ikev2 policy 40

 encryption aes

 integrity sha256

 group 5

 prf sha256

 lifetime seconds 86400

crypto ikev2 enable OUTSIDE client-services port 443

crypto ikev2 remote-access trustpoint ComodoRSA

telnet timeout 5

ssh stricthostkeycheck

ssh timeout 30

ssh version 1 2

ssh key-exchange group dh-group14-sha256

ssh 0.0.0.0 0.0.0.0 management

console timeout 0

threat-detection basic-threat

threat-detection statistics access-list

no threat-detection statistics tcp-intercept

ssl trust-point ComodoRSA management

ssl trust-point ComodoRSA OUTSIDE

ssl trust-point ComodoRSA INSIDE

webvpn

 enable OUTSIDE

 http-headers

  hsts-server

   enable

   max-age 31536000

   include-sub-domains

   no preload

  hsts-client

   enable

  x-content-type-options

  x-xss-protection

  content-security-policy

 anyconnect image disk0:/anyconnect-win-4.9.06037-webdeploy-k9.pkg 1

 anyconnect image disk0:/anyconnect-win-arm64-4.9.06037-webdeploy-k9.pkg 2

 anyconnect image disk0:/anyconnect-macos-4.9.06037-webdeploy-k9.pkg 3

 anyconnect image disk0:/anyconnect-linux64-4.9.06037-webdeploy-k9.pkg 4

 anyconnect enable

 tunnel-group-list enable

 cache

  disable

 error-recovery disable

group-policy GroupPolicy_XXXXXX internal

group-policy GroupPolicy_XXXXXX attributes

 wins-server none

 dns-server value 172.20.1.10

 vpn-tunnel-protocol ssl-client

 default-domain value XXX.XXXXXX.XX

dynamic-access-policy-record DfltAccessPolicy

dynamic-access-policy-record VPN-Users-Group

 action terminate

username admin password ***** pbkdf2 privilege 15

username admin attributes

 service-type admin

 ssh authentication publickey XXXX hashed

tunnel-group XXXXXX type remote-access

tunnel-group XXXXXX general-attributes

 address-pool VPN-Pool

 authentication-server-group LDAP-Servers LOCAL

 default-group-policy GroupPolicy_XXXXXX

tunnel-group XXXXXX webvpn-attributes

 group-alias XXXXXX enable

!

class-map inspection_default

 match default-inspection-traffic

!

!

policy-map type inspect dns preset_dns_map

 parameters

  message-length maximum client auto

  message-length maximum 512

  no tcp-inspection

policy-map global_policy

 class inspection_default

  inspect ip-options

  inspect netbios

  inspect rtsp

  inspect sunrpc

  inspect tftp

  inspect dns preset_dns_map

  inspect ftp

  inspect h323 h225

  inspect h323 ras

  inspect rsh

  inspect esmtp

  inspect sqlnet

  inspect sip 

  inspect skinny 

  inspect snmp

policy-map type inspect dns migrated_dns_map_2

 parameters

  message-length maximum client auto

  message-length maximum 512

  no tcp-inspection

policy-map type inspect dns migrated_dns_map_1

 parameters

  message-length maximum client auto

  message-length maximum 512

  no tcp-inspection

!

service-policy global_policy global

prompt hostname context

call-home reporting anonymous

call-home

 profile CiscoTAC-1

  no active

  destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService

  destination address email callhome@cisco.com

  destination transport-method http

  subscribe-to-alert-group diagnostic

  subscribe-to-alert-group environment

  subscribe-to-alert-group inventory periodic monthly

  subscribe-to-alert-group configuration periodic monthly

  subscribe-to-alert-group telemetry periodic daily

 profile License

  destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService

  destination transport-method http

Cryptochecksum:XXXXXXX

: end

no asdm history enable

Sheraz.Salim · ‎03-06-2021

could you show the output of show route on your vasa. Also you missing nat rules for anyconnect if you using anyconnect for full tunnel

here and here is the configuration of the anyconnect will guide you. to setup a best practice configuration.

please do not forget to rate.

piwo030 · ‎03-06-2021

Hi there. Many thanks for your reply. There are two routes:

route OUTSIDE 0.0.0.0 0.0.0.0 172.20.10.1
route INSIDE 0.0.0.0 0.0.0.0 172.20.1.1 tunneled

And thats the NAT Rule I have:

# sh nat

Manual NAT Policies (Section 1)
1 (outside) to (inside) source dynamic any interface 
    translate_hits = 0, untranslate_hits = 0

Auto NAT Policies (Section 2)
1 (nlp_int_tap) to (inside) source dynamic nlp_client_0_intf2 interface 
    translate_hits = 0, untranslate_hits = 0
2 (nlp_int_tap) to (inside) source dynamic nlp_client_0_ipv6_intf2 interface ipv6 
    translate_hits = 0, untranslate_hits = 0

Peter Koltl · ‎03-09-2021

Does it have similar performance when DTLS is disabled?

Sheraz.Salim · ‎03-09-2021

yes peter noted the same issue with DTLS.

please do not forget to rate.

timetrade · ‎03-11-2022

I came searching today on the same topic. I've been running an ASAv10 on a c4.large in AWS for years (us-east-1) and bandwidth has been less that what I thought, but it was acceptable for most of the work we did.

Then COVID hits and the users on AnyConnect goes from avg of 10 to 80 in a day. Plus, we hire people "remote first" in locations far from us-east-1, and the complaints of off-shore developers joins the chorus of unhappy users.

When I do some tests, the overall speed of a download from the ASA's Web Portal is abysmal compared to an apache host in the same subnet. When going through the AnyConnect, it's even worse. When doing either of the above from any physical (thus network hop) distance from Virginia US, it get to a point of dial-up speeds.

I tried stack exchange and I have a TAC case open, but each email response take a week (!) and still not even close to a real investigation.

I've setup another instance in us-west-2 on a modern c5.large and it's sorta better, but still a huge drop-off from what a c5 linux instance would deliver.

Brings me to the serious question, does ANYONE have an ASAv that performs well?

If I wget a file from the unauthenticated side of the web portal from a host local at AWS, it's 36 MB/s. If a user in Argentina on a 1Gbps connection does the same it's 163 KB/s.

piwo030 · ‎03-11-2022

Hey there!

To be honest I still did not find a solution on my end. What I can say is that this behavior not only shows up with Cisco ASA. I also performed some tests with a virtual SonicWall SSL VPN appliance. Same issues with that.

AWS tells me that there is no bandwidth restriction on ssl or any other protocols but it feels kind of strange that I can reproduce the same issue on different applicances.

Since now I weren't able to try my ASAv configuration on an instance that would be totally exaggerated. But if you have the time it would be interesting how it works on c5.xlarge (Recommended by Cisco) or even better on c5.9xlarge for a short test because this last one have a guaranteed 10Gbps bandwidth and not just "up to".

Would be lovely if you could keep us posted.

achavez@cccis.com · ‎03-07-2023

make sure to have BOTH tcp and udp 443 traffic allowed inbound to the outside interface of your ASAv in AWS; this is for DTLS traffic. same may apply to other vendors and in Azure, but literally ran into this just now and updating the AWS outside interface security group to add UDP 443 instantally changed my speed test results. running c5.2xlarge with ASAv50 (10G) license. from 5mb to 50mb download after change. hope this helps!

masees85 · ‎04-11-2023

Thanks, that was helpful

networking@proactiv.com · ‎09-13-2023

I've been struggling with this problem for the past year and as soon as I allowed udp 443 on the outside interface, I got 10x the speed that I was getting. Thanks alot !

Richard Tapp · ‎09-16-2023

I am also facing the same issue, It is quite frustrating, that from ASA firewall internet is working fine but when the user goes to the internet via anyconnect he hardly browse, i did allow udp 443 on ASA outside the security group but there no help still facing the issue.

Any suggestion