Slow CRS 1000v Throughput in AWS - any help appreciated

[bryuen001]

We have two VPCs, one in US East and one in US West Region. There is one CSR 1000v in each VPC. They are running AX 2.5G evaluation license. A simple site-to-site IPSec tunnel has been configured between them so traffic between the two VPCs will route through the tunnel. We are experiencing slow throughput between two Windows instances, one in US East VPC and one in US West VPC. Through the CSR VPN tunnel, iperf test shows about 1.5Mbit/s while through elastic IP and AWS NAT (i.e. outside the CSR path) the iperf throughput shows 15Mbit/s so there is a significant reduction.

The windows instances are t2.medium and the CSR instances are c3.large.

Platform hardware throughput level show the correct level:

The current throughput level is 2500000 kb/s

1.5 Mbit/s is not acceptable and not sure why the CSR reduces the throughput significantly.

Any one had similar experience? Any help is appreciated

[Umanath S]

I am also facing the exact problem with the CSR1000v, just getting 1.5 Mbps throughput, and to my surprise there is not even a single response for this query for the past 10 months..!

[raccoon]

Exactly the same issue, same licenses installed.

Doing Iperf via Public IP, between 2 linux instances on same region, i got 800 Mbps

But if i do the same test using private IP passing by the CSRs (vpc transit), it drop to 160 Mbps (80% of loss). I asked amazon premium support and they don't have any clue either ...

[johnnylingo]

I'm seeing this too, and have observed it's specific to Internet access.  If I download a file via Internet using the CSR1000v as default gateway, speeds are horrible (100-200 Kbps at best).  Downloading the same file via IPSec VPN tunnel terminated on the CSR1000v is 100 times faster.  I see this both with the 2.5 Gbps trial license and after purchasing and installing a 100 Mbps permanent one and rebooting.  I doubt it's a problem with AWS because using their NAT gateway, speeds are 50 Mbps.  

We deployed the CSR1000v on AWS t2.medium instances and it wasn't until upgrading to 16.10.1b that they were stable.  I don't have a root cause or explanation from Cisco yet as to why this is, but really seems to be an issue with the low CPU credits on the t2.mediums and/or incompatibility related to Hypervisor or vNIC.   I'd be curious if I re-launch on c4.large or c5.large how things go.  

[johnnylingo]

I re-launched the CSR1000v on t2.medium, c4.large and c5.large and configured them to only do routing and NAT with a 100 Mbps throughput license. These were the speed results:

# Test download via CSR1000v on t2.medium
ubuntu@ip-10-13-22-161:~$ curl ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/i386/ISO-IMAGES/11.2/FreeBSD-11.2-RELEASE-i386-bootonly.iso > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 261M 100 261M 0 0 30733 0 2:28:55 2:28:55 --:--:-- 92708

# Test download via CSR1000v on c4.largeubuntu@ip-10-13-22-161:~$ curl ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/i386/ISO-IMAGES/11.2/FreeBSD-11.2-RELEASE-i386-bootonly.iso > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 261M 100 261M 0 0 9993k 0 0:00:26 0:00:26 --:--:-- 11.2M

# Test download via CSR1000v on c5.large
ubuntu@ip-10-13-22-161:~$ curl ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/i386/ISO-IMAGES/11.2/FreeBSD-11.2-RELEASE-i386-bootonly.iso > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 261M 100 261M 0 0 10.4M 0 0:00:25 0:00:25 --:--:-- 11.2M

[jellybeanshiba]

I'm having the exact same problem... My CSR has a pure mGRE interface without Tunnel Protection and is acting as a DMVPN hub. When I try to transfer some data over the tunnel, I get no more than 2 Mbps of throughput. Just horrible.

ip-100-67-102-254#show platform hardware throughput level 
The current throughput level is 200000000 kb/s

ip-100-67-102-254#show version 
Cisco IOS XE Software, Version 16.10.01b
Cisco IOS Software [Gibraltar], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.10.1b, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Wed 12-Dec-18 02:52 by mcpre

[johnnylingo]

I was running 16.10.1b too, and hitting this bug:

CSCvn52259 - CSR1Kv:Throughput/BW level not set correctly when upgrade to 16.10.1 .bin image with AWS/Azure PAYG

Upgrading to 16.10.2 or 16.11 fixed it

I opened a TAC case and they were completely unhelpful.  This is something I had to discover on my own.

[jellybeanshiba]

Thank you for the tip! I'll replace the code and see if it helps.

[jellybeanshiba]

I can confirm that upgrading to 16.10.02 did help and now I'm seeing the full throughput :))) Thank you very much @johnnylingo !